EUROPEAN COMMISSION Brussels, 17.12.2021 SWD(2021) 965 final COMMISSION STAFF WORKING DOCUMENT Final evaluation of the ISA² programme Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Results of the final evaluation of the ISA² programme {COM(2021) 965}



Table of contents 1.Introduction 2.Background on the intervention logic of the ISA2 programme 3.Implementation / State of Play 3.1. How is the ISA2 programme implemented? 3.2. Important developments in the policy field 4.Methodology 4.1. Fact-finding 4.2. Limitations and robustness of findings 5.Analysis and answers to the evaluation questions 5.1. Relevance 5.2. Effectiveness 5.3. Efficiency 5.4. Coherence 5.5. EU added value 5.6. Utility 5.7. Sustainability 6.Conclusions 6.1. Successful elements of the intervention

6.2. Less successful elements of the intervention 6.3. Will issues be resolved over time or do they need to be addressed with specific measures? 6.4. Lessons learnt Glossary Term or acronym Meaning or definition ADMS Asset description metadata schema AI Artificial intelligence CEF Connecting Europe Facility CEPS Centre for European Policy Studies CPSV-AP Core public service vocabulary application profile DCAT-AP Data catalogue vocabulary application profile for data portals in Europe DG Directorate-General DG DIGIT Directorate-General for Informatics DIGIT.D2 Directorate-General for Informatics, Directorate D Digital Services, Unit D2 Interoperability DSM Digital single market EC European Commission EEA European Economic Area EIA European interoperability architecture EIC/EICart European interoperability cartography eIDAS Electronic identification and trust services EIF European interoperability framework EIRA European interoperability reference architecture EIS European interoperability strategy ELISE European location interoperability solutions for e-government EQ Evaluation question ERDF European Regional Development Fund ESPD European Single Procurement Document EU European Union EVM Earned value management GDPR General Data Protection Regulation Horizon 2020 EU funding programme for research and innovation IAP Interoperability action plan (Annex I to the Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions: European Interoperability Framework — Implementation Strategy. Interoperability action plan, Brussels, 23.3.2017, COM(2017) 134 final) ICT Information and communication technology IDA Programme on interchange of data between administrations IDABC Programme on interoperable delivery of pan-European eGovernment services to public administrations, businesses and citizens IMAPS Interoperability maturity assessment of a public service Interoperability As explained in Article 2(1) of the ISA2 Decision , ‘interoperability’ means the ability of diverse organisations to interact towards mutually beneficial and agreed common goals. It involves the sharing of information and knowledge between the organisations, through their business processes and by means of the exchange of data between their respective ICT systems. ISA Programme on interoperability solutions for European public administrations ISA2 Programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens ISA2 actions webpage https://ec.europa.eu/isa2/actions_en ISA2 dashboard https://ec.europa.eu/isa2/dashboard/ ISA2 decision L 318/1 Decision (EU) 2015/2240 of the European Parliament and of the Council of 25 November 2015 establishing a programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens (ISA2 programme) as a means for modernising the public sector, Brussels 4.12.2015. ISA2 proposal European Commission (2014), Proposal for a Decision of the European Parliament and of the Council establishing a programme on interoperability solutions for European public administrations, businesses and citizens (ISA²) – Interoperability as a means for modernising the public sector, COM(2014) 357 final. ISA2 solutions webpage https://ec.europa.eu/isa2/solutions_en ISA2 website https://ec.europa.eu/isa2/isa2_en ISSG Inter-Service Steering Group IT Information technology Joinup Collaborative platform facilitating the sharing and reuse of IT solutions developed for public administrations NGOs Non-governmental organisations NIFs National interoperability frameworks NIFO National interoperability framework observatory OECD Organisation for Economic Cooperation and Development PMKI Public multilingual knowledge management infrastructure for the digital single market REFIT The European Commission's regulatory fitness and performance programme RegDel Inter-institutional Register of Delegated Acts SDG Single digital gateway SCM Standard cost model SEMIC The ISA2 programme action that promotes semantic interoperability among EU countries SPI Schedule performance index SRSP Structural reform support programme sTESTA Secure trans-European services for telematics between administrations TSI Technical support instrument 1.Introduction The ISA2 programme – interoperability solutions for public administration, businesses and citizens – aimed to support the digital transformation of the public sector by providing digital solutions that enable public administrations, citizens, and businesses across the EU to reap the benefits of interoperable cross-border and cross-sector public services. ISA2’s primary beneficiaries are EU, national and regional public administrations: by reusing the solutions offered by ISA2 they can provide better – more interoperable, user-centric, and digital – public services. However, the programme also helps a broader group of stakeholders, namely EU businesses and citizens. Established by the ISA2 Decision the programme was operational from 1 January 2016 to 31 December 2020 with a total budget of € 131 million distributed over the five-year period. It was open to EU countries, other EEA countries and candidate countries. In line with Article 13(3) of the ISA2 Decision, the Commission must carry out the final evaluation of the ISA2 programme by 31 December 2021. Besides fulfilling this legal obligation, the evaluation aims to improve the implementation of the Digital Europe Programme – which is funding public sector interoperability from 2021 onwards – and contribute to the development of a new interoperability policy for the EU’s public sector. The evaluation covers ISA2 activities from the programme’s start until October 2020 in all participating countries. The evaluation was based on an evaluation framework composed of seven evaluation criteria and 10 evaluation questions (see Annex 4). Five evaluation criteria stem from the Commission’s better regulation requirements and two additional criteria from the ISA2 Decision (see Box 3 ).



2.Background on the intervention logic of the ISA2 programme The ISA2 programme’s ultimate objective was to promote the ICT-based modernisation of the public sector in Europe and to help addressing the needs of businesses and citizens, via improved interoperability of European public administrations. End-users, citizens and businesses should benefit from common, re-usable and interoperable front-office services resulting from better integration of processes and exchange of data through the back offices of European public administrations. More specifically the programme aimed to do the following: Facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens. Contribute to the development of a more effective, simplified, and user-friendly e-administration at the national, regional, and local administration levels. Promote a holistic approach to interoperability in the EU, by identifying, creating and operating interoperability solutions and facilitating their reuse by European public administrations. This will support the implementation of various EU policies and activities. By working towards achieving the above objectives, the programme intended to address the problem of existing or emerging ‘electronic barriers that impede the proper functioning of the internal market’. It is important to note that ISA2 was designed to be part of a wider policy framework related to the digitalisation of public administrations in the EU. In cooperation with the EU countries and the Commission, it promoted and supported the European Interoperability Framework (EIF) , which has been in place since 2010 and was revised in 2017. Since the start ISA2 was designed to be synergetic especially with the Connecting Europe Facility (CEF) . Operating from 2014 to 2020 CEF has been a key programme that supported the cross-border interaction between the digital services infrastructures of EU countries. Chapter 5 supplies further details of the links between ISA2 and the above-mentioned programmes and initiatives, under the coherence evaluation criterion (section 5.4 ). The intervention logic presented in Annex 3 aims to clarify the reasoning followed by EU decision makers when setting up the ISA2 programme. It includes a detailed description of:

·the needs, problems and drivers that the programme intended to address, ·the objectives set out for the programme (following a three-level hierarchy: global objective, specific objectives and operational objectives), ·the inputs or activities of the programme, ·the programme's expected results, and ·the logical links between these components. The intervention logic elements and links supply a benchmark against which the final evaluation can assess the programme's achievements. The final evaluation of ISA2 follows the same intervention logic discussed in the interim evaluation of the programme . This approach ensures full comparability between the findings of the final evaluation and the interim evaluation of ISA. ISA2 is the fifth in a series of European Commission programmes providing and promoting interoperability solutions for public administrations in the EU ISA2 succeeded the ISA programme. The actions contributed to different kinds of interoperability solutions, among them open-source software, open specifications (e.g.: data models and frameworks) and managed services (e.g.: Joinup and EU Survey) based on the decision of the ISA2 committee. The results of the ISA programme represent the main baseline used for the purpose of this evaluation. More precisely, the below recommendations stemming from the final evaluation of the ISA programme, serve as points for comparison: Box 1 Conclusions and recommendations from the ISA programme evaluation Strategic recommendations 1.The ISA² programme, which serves an EU policy, should continue to align itself with other relevant EU policies. 2.Support the revision and implementation of the EIS (European Interoperability Strategy) 3.Continue to focus on the current ISA activities but more emphasis on legal and organisational interoperability – stronger focus on ICT impacts assessments of EU policies was an important need expressed by EU countries. 4.Update and implement a communication strategy for the programme, with a focus on targeted engagement including sector-specific stakeholders to ensure that interoperability features as part of EU-level solutions and programmes of other DGs. 5.Develop a more systematic business-case approach for the annual selection of new actions and to identify different financing options for ongoing actions based on its current work in the area of cost-benefit assessment of interoperability. Operational recommendations 6.Respect the targets of the programme’s envisaged staff levels including development of more knowledge on legal interoperability issues. 7.Build on the improvements in coordination of activities related to interoperability and eGovernment across the Commission 8.Continue to document ISA solutions, and their building blocks in EUCart and Joinup 9.Develop a more systematic approach to support the use of common services and generic tools, but also the application and implementation of common frameworks The interim evaluation of the ISA2 programme noted some progress concerning the above-listed areas and complemented them with recommendations to: ·go beyond national administrations when it comes to communication and awareness-raising activities thus target regional and local administrations directly. ·improve the quality of the interoperability solutions by better considering user needs. ·preserve and build on the ISA2 programme’s achievements.



3.Implementation / State of Play How is the ISA2 programme implemented? Up to the end of 2020, the ISA² programme supported a total of 54 actions, grouped in nine packages, which were defined on a yearly basis through an annual rolling work programme. The nine packages include the following: 1.key and generic interoperability enablers; 2.semantic interoperability;

3.access to data / data sharing / open data; 4.geospatial solutions; 5.eProcurement / eInvoicing; 6.decision-making and legislation; 7.EU policies — supporting instruments; 8.supporting instruments for public administrations; 9.accompanying measures. ISA2 managed 39 actions in its first year of operation, 43 actions in 2017, 53 actions in 2018, and 54 actions under both the 2019 and 2020 rolling work programmes. Table 1 and Table 2 provide a breakdown of the allocated budget and the number of actions per package and per year, respectively. Table 1: Overview of allocated budget per package from 2016 to 2020 (in thousands of euros) Package 2016 2017 2018 2019 2020 Total 1. Key and generic interoperability enablers 4,900 5,407 4,250 4,634 4,688 23,879 2. Semantic interoperability 2,008 1,831 1,503 1,989 1,928 9,259 3. Access to data / data sharing / open data 2,800 3,548 3,925 1,710 1,150 13,133 4. Geospatial solutions 983 2,240 1,900 2,200 2,300 9,623 5. eProcurement / eInvoicing 2,400 1,445 1,063 807 1,000 6,715 6. Decision-making and legislation 2,260 2,608 2,235 2,735 3,090 12,928 7. EU policies — supporting instruments 2,935 3,580 4,820 4,990 4,100 20,425 8. Supporting instruments for public administrations 4,425 3,533 5,315 6,030 7,585 26,888 9. Accompanying measures 1,280 730 1,290 1,370 1,530 6,200 Total 23,991 24,922 26,301 26,465 27,371 129,050 Note: This table presents the allocated budget per package as presented in the ISA2 Rolling Work Programmes. The Rolling Work Programmes also list non-allocated budget, which includes reserves. Source: ISA2 rolling work programmes (2016, 2017, 2018, 2019 and 2020), indicative planning and financial overview. Table 2: Overview of the number of actions per package from 2016 to 2020 Package 2016 2017 2018 2019 2020 1. Key and generic interoperability enablers 6 6 7 7 7 2. Semantic interoperability 3 3 4 4 4 3. Access to data / data sharing / open data 5 5 7 7 7 4. Geospatial solutions 1 1 1 1 1 5. eProcurement / eInvoicing 1 1 1 1 1 6. Decision-making and legislation 6 9 10 10 10 7. EU policies — supporting instruments 3 3 5 5 5 8. Supporting instruments for public administrations 12 13 16 17 17 9. Accompanying measures 2 2 2 2 2 Total 39 43 53 54 54 Source: ISA2 rolling work programmes (2016, 2017, 2018, 2019 and 2020), indicative planning and financial overview. The programme in its entirety was overseen by the Interoperability Unit of DG DIGIT (DIGIT.D2), while individual actions were implemented by different Commission services, depending on the thematic scope. The EU countries were also involved in programme governance through two channels: the ISA² Committee, the programme's high-level governing body, and the ISA² Coordination Group, a technical body mandated to ensure coherence between the programme’s actions. All EU countries participated in the programme. Beyond the EU, ISA2 has four additional members — Iceland, Montenegro, North Macedonia and Norway — and an agreement of cooperation had been in place with Uruguay since March 2018 and a cooperation with Ukraine is in place since 2019. ISA2 actions were selected and implemented through a process consisting of four steps taken each year: 1.Submission: Commission services, EU countries, and other countries participating in ISA2 were invited to submit proposals for actions to be included in the rolling work programme via a call for proposals. 2.Evaluation: The submitted proposals were analysed by DIGIT.D2, which compiled a list of proposals that qualified to be included in the rolling work programme, based on the selection and prioritisation criteria as defined in the ISA2 Decision. At this stage, the ISA2 Committee gave an opinion on the proposed action list. 3.Adoption: The rolling work programme was adopted by the Commission and the budget got released.

4.Implementation: The actions in the adopted rolling work programme were implemented by the Commission services in charge. The programme's overall performance and the progress of each action were frequently recorded through the quarterly and annual monitoring and evaluation reports, which fed into the ISA2 dashboard . The dashboard is an online interactive tool that facilitates the dissemination of information about action activities and achievements, and their efficiency, effectiveness and coherence. The efficiency of actions is measured using the earned value management (EVM) analysis (see section 5.3 Efficiency ), effectiveness is presented in terms of performance indicators (see section 5.2 Effectiveness ), and the coherence of actions is mapped using network analysis (see section 5.4 Coherence ). Figure 1 illustrates the performance until Q1 2021, as captured by the ISA2 dashboard based on the earned value management (EVM) mechanism. Figure 1 Earned value analysis at programme level Note: The cumulated planned value is the sum of the planned values of the programme's different actions for which the EVM is used. The cumulated earned value is the sum of the earned values of the programme's different actions for which the EVM is used. As the figure shows, the implementation of some actions continued in 2021 based on contracts signed in the previous year(s). Source: Monitoring team of the ISA2 programme (see also the Efficiency view of the ISA2 Programme ). Important developments in the policy field During the evaluation period from 1 January 2016 until 31 December 2020 digitalisation and more precisely interoperability have become more important at all levels of society and became a priority of the new elected Commission. The following developments can be highlighted: Adoption of a new EIF In 2017 EIF has been updated and extended as planned in the Communication on a Digital Single Market Strategy for Europe and the eGovernment Action Plan 2016-2020 . The new EIF was issued through a Communication and accompanied by an Interoperability Action Plan. ISA2 remained the main resource to ensure: ·the implementation of EIF in the EU countries and sectors; ·the implementation of the Interoperability Action Plan; ·the monitoring of EIF’s implementation and of the Interoperability Action Plan; ·the “governance of these actions” through the ISA2. The new EIF gave the ISA2 programme a new policy framework with coherent objectives. The evaluation contributed to the assessment of the achievements of the EIF and more specifically its annex, the Interoperability Action Plan (IAP) ( Box 2 ).

Box 2: ISA2 contribution to the implementation of the IAP The new EIF adopted in 2017 was accompanied by an Interoperability Action Plan (IAP) (annexed to the 2017 EIF Communication) listing key actions to be undertaken between 2017 and 2020. The EIF Communication acknowledged the ISA2 programme as one of the main implementing instruments of the IAP and the EIF in general. Against this background, the evaluation of ISA2 also considered how the programme contributed to the implementation of the IAP and thus to the implementation of the EIF in broader terms. There is a clear direct relationship between several ISA2 actions and the actions listed in the IAP, while other ISA2 actions provide broader contributions across several areas of the IAP, as described in Annex 6.k. Commitment of EU countries and the European Commission In the evaluation period the political commitment from EU countries and European Commission has been steadily growing: ·The Tallinn Declaration on eGovernment , a declaration made by ministers in charge of eGovernment policy across the EU to spell out their commitment to a number of principles, including ‘interoperability by default’. ·The Berlin Declaration on Digital Society and Value-Based Digital Government , which shows the commitment of the EU countries, alongside EU institutions, to tap into the potential of digital public services and take a value-based approach – incorporating digital sovereignty and interoperability - to the digital transformation of the public sector. ·In February 2020, the European Commission adopted the Communication ‘ Shaping Europe's digital future ’ under the headline ambition ‘Europe fit for the digital age’. The Communication sets out as a key action the development of a ‘reinforced EU governments interoperability strategy’, aiming to foster coordination and the adoption of common standards for public services and data flows. A growing EU regulatory environment While the ISA2 Decision is already referring to a wide range of sectorial and cross-cutting interoperability activities on EU level, important initiatives have been brought in place after the programme’s start and more are to come in the coming months: ·2016 brought e.g. the GDPR with high impacts on processing of personal data for public services and the Web Accessibility Directive ; ·The Single Digital Gateway , which facilitates online access to information, procedures and assistance regarding EU and national rules for citizens and businesses was introduced in 2018. ·In 2019 the EU cybersecurity act and other legislation in the field as well as the Open Data Directive . ·In 2020 the European Commission proposed the Digital Services Act . Digitalisation investments Public administrations across the EU are investing massively in the digitalisation of their services. With the experience of the COVID crisis, the need for digital solutions has been more prominent than ever. Besides ISA2 other EU programmes have contributed to these efforts in the evaluation period:

·The Structural Reform Support Programme (SRSP) , implemented from 2017 to 2020, provided technical support to the EU countries for designing and implementing reforms in various areas, including the modernisation of public administrations. Support continues in the next multi-annual financial framework through the Technical Support Instrument (TSI) for the period 2021 to 2027 . ·Several other EU funding programmes, like Horizon 2020 (dedicated to research and innovation), the European Social Fund (ESF) and the European Regional Development Fund (ERDF) .

4.Methodology Fact-finding To start the Commission outlined and agreed on the draft evaluation design (including the intervention logic and the evaluation questions). Then it reached out to external experts and asked them to support the evaluation process (see Annex 1). The consultant refined the evaluation design and — with the help and under the close monitoring of the Commission — moved to action. First, it collected data, then validated and analysed them, as explained in the following sections. At the end, the consultant summarised its findings in an independent evaluation study (referred to as the CEPS final study in this report). Box 3 Evaluation framework The evaluation was based on an evaluation framework composed of seven evaluation criteria and 10 evaluation questions (see Annex 4). The evaluation criteria are summarised below: Relevance refers to the alignment between the programme's objectives and the evolving needs and problems experienced by stakeholders. Effectiveness focuses on the extent to which the ISA² programme has met the objectives it intended to achieve, and generated the results it intended to produce. Efficiency concerns the minimisation of costs borne by various stakeholders in achieving the objectives/results identified under the ‘effectiveness’ criterion. Coherence is a measure of the degree to which the actions supported by the ISA² programme are consistent with each other (internal coherence) and with the EU policy framework at large and relevant global initiatives (external coherence). EU added value captures the programme's impacts additional to those that would be achieved if the issues addressed by ISA2 were left solely in the hands of national and sub-national authorities. Utility refers to the extent to which the results generated by ISA² satisfy stakeholders' needs and the differing levels of satisfaction among different stakeholder groups (e.g. public administrations, businesses, citizens). Sustainability measures the likelihood of the ISA² programme's results lasting beyond its completion.



Data collection The data collection phase involved a mix of quantitative and qualitative data collection methods and aimed to gather the evidence base to respond to the evaluation questions (EQs) specified in the evaluation framework (Annex 4). More specifically, data were drawn from two main sources: consultation activities (primary data) and desk research (secondary data). The data collection was complemented by an expert assessment, performed by five technical interoperability experts who were part of the consultant’s evaluation team. Primary data Primary data were collected between December 2020 and April 2021 via the following consultation activities: targeted in-depth interviews; targeted online surveys; public consultation; feedback on the ISA2 evaluation roadmap. ·two workshops, which aimed to raise awareness of the ongoing evaluation and discuss preliminary findings with stakeholders. These activities resulted in a total of 102 responses and reached out to different types of stakeholders, ranging from actors involved in the programme’s governance to indirect beneficiaries (i.e., citizens and businesses). Overall, the respondents were characterised by a high level of expertise both generally in the field of digital public services and interoperability, as well as more specifically when it comes to the ISA2 programme, which contributed to the quality and reliability of primary data. On average, consulted stakeholders across stakeholder groups have a very good knowledge of the field of digital public services and interoperability (with an overall average of 3.9 out of 5 for the 100 respondents who completed this question). On average the respondents involved in the governance of ISA2 are most familiar with the programme (with an average score of 4.11 out of 5, based on 9 responses), followed by ISA2 action owners (average score of 4.07 out of 5 based on 15 responses). At the opposite end of the scale, the wider public is least knowledgeable compared to the other stakeholder groups (average score of 2.94 out of 5, based on 36 responses). Figure 2 Consulted stakeholders’ familiarity with digital public services and interoperability and the ISA2 programme

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: 1) Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 2) Total number of respondents: 100 (Knowledge of digital public services and interoperability) and 102 (Knowledge of ISA2). Source: CEPS final study. Primary data provided inputs for the assessment of all evaluation criteria set out in this evaluation. In particular, the in-depth interviews as well as the targeted online surveys, which were tailored to targeted stakeholders, allowed for the collection of in-depth information for the evaluation process. For more details, please consult Annex 2: Synopsis report of the consultation activities, which presents a breakdown of responses by consultation activity and stakeholder category along with respondent characteristics. Chapter 5 of this report presents the aggregate results of the consultation activities using mainly bar charts showing the average scores of responses from each stakeholder group. Secondary data The desk research reviewed the following resources: publicly available documents and data sources, including the ISA2 annual rolling work programmes, the ISA2 dashboard, and the webpages dedicated to actions and solutions; additional operational documents including monitoring and evaluation reports, overviews of communication activities, and lists of participants in the ISA2 Committee and Coordination Group; ·policy documents, studies and reports relevant for public sector interoperability. Annex 6 provides an overview of the supporting evidence collected from desk research. Note that, in order to better guide the data collection activities, a sample of 21 actions was selected from the total of 54 actions included in the 2020 Rolling Work Programme . The criteria used for sampling and the sampled actions are presented in Annex 5: Sample of actions. Data validation The collected data were validated via triangulation in order to ensure the robustness of evidence. Tool #4 of the Commission’s Better Regulation Toolbox defines triangulation as ‘the application and combination of several research methodologies in the study of the same phenomenon’ . In fact, for all evaluation criteria and questions, data were collected from multiple sources and via at least two different data collection methods (e.g.: interviews, targeted questionnaire, public consultation, desk research). Quantitative methods of data analysis Besides the statistical analysis of data collected via consultation activities and the qualitative analysis of open responses provided by consulted stakeholders and information contained in documentary evidence, three specific quantitative methods were used in order to evaluate the programme's efficiency: The standard cost model (SCM) is a method of assessing administrative costs imposed by rules or policies inter alia on businesses and public administrations. It is based on the identification of the basic components of a rule, the information obligations, whose costs for the addressees can be measured and quantified. An information obligation is a specific duty to gather, process or submit information to the public authority or a third party. The SCM was used to measure the costs borne by action owners in preparing and submitting proposals for ISA2 actions. Cost-effectiveness analysis (CEA) is a method of assessing the merits of a policy in an interim and ex post evaluation setting. In a nutshell, CEA measures the value-for-money of past policies, i.e. the amount of benefit generated by unitary costs. Costs are measured in monetary terms, whereas effectiveness is measured in ‘natural units’, and the unit of account varies depending on the nature of the problem addressed (e.g. the number of users of key and generic interoperability enablers). The heterogeneity of performance indicators available for ISA2 actions makes it difficult to draw conclusions about the programme's overall cost-effectiveness. Thus using CEA was only possible for certain action packages and indicators, as described in Chapter 5.3 . ·In line with the PM2 methodology developed by the Commission, the earned value management (EVM) and earned schedule (ES) methods are currently used to monitor and assess the programme’s efficiency. EVM is a project management technique that helps determine work progress against a given baseline, so that costs, time, and scope of a certain activity are constantly tracked. In ISA2’s context, efficiency is assessed at action and programme levels. The implementation of EVM requires managers to calculate the earned value, i.e. a quantification of the ‘worth’ of the work done to date, and the actual costs, i.e. the executed budget for achieving the work, and to compare them with the planned value of such activity. This allows for a better understanding of the programme's performance. The ES is an extension of the EVM method which deepens the level of analysis to a ‘units of time’ layer. In the framework of the ISA2 programme, tailored versions of the EVM and ES approaches are adopted.

Limitations and robustness of findings The availability of a mix of primary and secondary data, gained through the consultation activities and the data collection (Annex 4) allowed the evaluation team to draw robust conclusions for each evaluation question. However, it is necessary to point out that existing caveats may have affected the main findings: ·One constraint consisted in the lack of direct contact with solution users. Only action owners had access to the contact details of their solution users due to confidentiality and data protection reasons. Therefore, action owners were requested to: i) invite users to complete the targeted online survey; and ii) share with DIGIT.D2 the details of those users who gave their consent to participate in interviews. This two-step approach may have limited the number of answers received from solution users. ·Another limitation may be the potential 'consultation fatigue' of respondents and complexity of consultations. With the programme having come to an end, stakeholders may have been confronted with multiple requests for feedback. Given the technical nature of the field of interoperability, the fact that several consultations with various degrees of complexity occurred at similar times might have resulted in some stakeholders being more selective about the consultations they wished to engage in. Compared to the interim evaluation, however, a higher number of respondents contributed to the public consultation (43 respondents in the final evaluation and 14 respondents in the interim evaluation). This result may be due to the fact that the consultation activities were run jointly with the consultations contributing to the evaluation of the EIF and the impact assessment for a future interoperability policy for the EU public sector. The joint consultations may have attracted more respondents to the ISA2 consultation. This outcome, in turn, counterbalances the low rate of responses from solution users (5 respondents in the final evaluation compared to 43 solution users in the interim evaluation).

5.Analysis and answers to the evaluation questions In line with the evaluation framework (see Annex 4), the final evaluation focused on seven evaluation criteria: relevance, effectiveness, efficiency, coherence, EU added value, utility and sustainability. The overall analysis is based on evidence from both the external evaluation report and the Commission’s own sources. This chapter presents the analysis and provides the answers to the general evaluation questions. Relevance In the assessment of the criterion “relevance” the relationship between the needs and problems of stakeholders and the programme’s objectives and actions is analysed as well as the question of the continued suitability of the programmes objectives. The consulted stakeholders confirmed that the original needs and problems as set out in the intervention logic (Annex 3) are still an issue for interoperability in the EU’s public sector. Almost all respondents across the stakeholder groups indicated that the needs are still present to a great extent or completely (based on average scores of 4.2 out of 5 for the need for public administrations to cooperate to enable more secure and efficient public services, and 4.1 out of 5 for the other two needs). Figure 3: Extent to which needs and problems originally addressed by ISA2 are currently experienced by European public administrations, businesses and/or citizens (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Note: 1) Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). 2) Total number of respondents for each need and problem from top to bottom: 101, 101, 101, 98.

Source: CEPS final study. The majority of consulted stakeholders agree that achieving the objectives of the ISA2 programme can contribute to addressing the identified needs and problems (see Figure 4 ). Figure 4: Extent to which achieving ISA2 objectives contributes to addressing the needs and problems originally addressed by the programme (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: 1) Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). 2) Total number of respondents for each need and problem from top to bottom: 101, 96, 100, 95. Source: CEPS final study. It is particularly interesting to note that respondents among experts and academia, as well as the wider public tend to have a more positive view of the programme compared to the other stakeholder groups. By contrast, programme governance stakeholders point to a more limited alignment. With an overarching perspective of the programme, this stakeholder group emphasised in particular the fact that the needs cannot be tackled only through ISA2 and thus other initiatives are important in the field to fully address the needs and problems. Recent academic sources and reports further substantiate the current needs and problems in the field of public sector interoperability (see Annex 6.a). The need for coordination and cooperation, inter alia, is considered as an important requirement for efficient and secure public services. A 2013 study on the need for the cross-border digital public services emphasised that limited cooperation among the public administrations is the most crucial barrier for improving the delivery digital public services. The importance of the cooperation and governance of interoperability initiatives among EU countries is reiterated as a key element by De Abreu. The paper acknowledged that cooperation among EU countries improves the efficiency of public services, contributing to cost savings. The limited sharing of information and re-use of data is another recurring problem. The research by Kalvet et al. on the once-only principle shed more light on the main barriers to the development of cross-border services. The barriers include existing heterogeneity of technological infrastructures, the limited legal interoperability and low awareness of the benefits stemming from the implementation of the once-only principle.

The last year of the functioning of the ISA2 programme was also marked by the COVID-19 pandemic. The pandemic has exacerbated existing needs, including the need for coordination when implementing digital solutions, the need for interoperability in particular fields such as healthcare and mobility and the need to create joint solutions to support contact-tracing efforts and the vaccination campaigns (e.g.: vaccine certification schemes.). In this context, the ISA2 programme reacted among others by facilitating the exchange of best practices and the re-use of solutions through a common repository, the “ Digital Response to COVID-19 ” collection on Joinup. Complementing the findings of the literature review, 38 respondents (out of the 59 stakeholders who participated in the targeted consultations) indicated that there are additional needs and problems related to the interoperability of public services. These needs, which are cross-cutting, cross-border and cross-sectoral, and they are experienced by public administrations at all levels, include: ·The need for digital literacy and skills to ensure that the tools developed can also be used effectively and thus improve take-up. ·Extended diffusion of digital identity. ·The need to exchange best practices between EU countries and public administrations at all levels in a structured and proactive way. ·The need to ensure a feedback loop with citizens to improve the functioning of digital tools and solutions. ·The need for consistent governance of the different initiatives in the field of interoperability at the EU level. ·The need to target regional and local administrations directly. When it comes to the new needs and problems identified by consulted stakeholders, Figure 5 shows that a majority of the 38 respondents who indicated that there were additional needs also emphasised that achieving the ISA² objectives can contribute to addressing these needs at least to some extent. Nevertheless, some of the needs and problems go beyond the scope of what the ISA2 programme is meant to achieve. Figure 5: Extent to which achieving ISA2 objectives contributes to addressing additional (current) needs and problems identified by consulted stakeholders (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: 1) Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). 2) Total number of respondents during targeted consultation: 38. Source: CEPS final study. In conclusion, the original needs and problems that the programme intended to address remain highly relevant and the objectives of ISA2 are pertinent to addressing them. The COVID-19 pandemic has had an important impact on the programme’s relevance and that of interoperability initiatives more generally. In addition, the evaluation has identified a number of new needs and problems related to the interoperability of digital public services that go beyond the scope of what the programme was meant to achieve.



Effectiveness The criterion “effectiveness” focuses on the extent to which the programme has achieved its objectives and generated the expected results. Achievement of general and specific objectives At the time of its adoption, the ISA2 programme, through its solutions, aimed to contribute to a set of general, specific, and operational objectives. After five years, the programme and its solutions have contributed to the achievement of these objectives at least to some extent, according to the consulted stakeholders (see Figure 6 ) and the desk review of secondary data (see Box 4 ). At the level of the general and specific objectives, the majority of respondents confirm that the following objectives were achieved to some extent or to a great extent:

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: 1) Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 2) The average score is based on the answers received through targeted consultations. Source: CEPS final study. Expected and actual results of the programme By comparing the expected and actual results of the ISA2 programme (see Figure 8 ) a similar picture emerges to the alignment between the achievements of ISA2 in relation to its objectives. To some extent, the actual results reflect the expected results, but more work remains to be done to fully achieve the expected results in the field. Figure 8: Extent to which ISA2 solutions achieved the programme's expected results (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: 1) Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 2) The average score is based on the answers received through targeted consultations. Total number of respondents for each result from top to bottom: 54, 54, 55, 50, 55, 54, 52, 53. Source: CEPS final study. As shown by Figure 8 , ISA2 has contributed particularly to “a coherent interoperability landscape in the Union based on a holistic approach to interoperability” and to the “advancement of Union policies and activities by supporting their implementation” Conversely, according to the majority of respondents across stakeholder groups, results have been achieved only to a limited extent or to some extent when it comes to reducing the cost and administrative burden of cross-border interaction, removing the administrative e-barriers and contributing to the swift implementation of ICT systems supporting EU legislation. Many ISA2 solutions focused on the back-office interoperability across domains and sectors. Even though this is a needed step towards seamless public service delivery to citizens and businesses, the results of the back-office integration may not be visible in the short term, as shown by the public consultation replies, due to the longer journey towards a full implementation. Similarly, only limited results are observed by the stakeholders when it comes to contributing to more effective, simplified and user-friendly e-administrations in European public administrations. The limitations identified reflect the feedback on the achievement of the objectives by emphasising that more could be done to develop methodologies for assessing cost-savings derived from the implementation of interoperability solutions. The take-up of the solutions (see Annex 6.b) – brought to maturity – and the relations with standardisation bodies could be improved too (see Box 5 ). Box 5: Desk research findings – areas to improve effectiveness Take-up of the ISA2 solutions The use and re-use of ISA2 solutions varies between countries (see Figure 9 ). Based on the sample of 21 actions covered by the evaluation, it is possible to gain relative overview of the use and re-use of ISA2 solutions by public administrations across the EU (see interactive map on the ISA2 webpage). Nevertheless, while it is clear that solutions are being re-used across the EU countries, more could be done to enhance their take-up and re-use, especially at regional and local levels. For this, it seems crucial to raise awareness on interoperability beyond the technical audience. Two recently started actions in the field are the Interoperability Academy that aims to provide learning resources on the new EU academy platform . Another example in this sense is the Better legislation for smoother implementation community on Joinup, which promotes the idea of digital-ready policymaking and the importance of considering digital aspects, including the use and reuse of existing solutions, already during the policy design phase. The take-up of solutions may be increased in the future by ensuring clear dissemination to public administration and providing one-stop-shop solutions that allow users to clearly see and access available resources. In addition, channelling the efforts to fewer actions and solutions and bringing these to maturity would also help increase the take-up of the programme’s outputs, according to consulted stakeholders. Figure 9: Overview of the use and re-use of solutions by public administrations in the EU countries, based on a sample of solutions Note: The colour gradient denotes the number of individual ISA2 solutions used or re-used by public administrations in the EU countries. The analysis counts only overall usage, not individual instances of solution re-use. This assessment is based on 17 solutions that are part of the sampled actions for this evaluation, building on available data on the ISA2 solutions webpage. Source: CEPS final study. Relations with standardisation bodies Finally, the independent expert assessments contributing to this evaluation emphasised the role of standards developing organisations. A stronger and systematic link with standard developing organisations could help improve the programme’s achievements in several areas such as: further enhancing the coherence of the interoperability initiatives in the EU, supporting the development and updating of common standards and specifications, and mapping the landscape of interoperability solutions and specifications. A 2020 report prepared by the High-Level Expert Group on Business-to-Government Data Sharing emphasised the need to invest in mature solutions and common standards . In addition, the report of the High-Level Expert Group called for the Digital Europe Programme to invest in common standards to facilitate data sharing, taking into consideration the results already achieved as part of the initiatives including the ISA2 programme. Importantly, “the expert group recommends prioritising those standards that are most generally used over creating new ones” and working together with European standardisation bodies. Moving forward, it will be crucial to agree upon and promote the use of a common set of standards to facilitate data exchanges in the public sector. The role of standardisation is further discussed in chapter 5.4 on ISA2’s coherence. Influence of external factors Respondents from all stakeholder groups confirmed that several external factors contribute to the programme’s performance ( Figure 10 ). Particularly important is the need for public administrations to increase their efficiency due to budget constraints, as was emphasised by public authorities who consider that this factor has contributed to a great extent to the performance of ISA2. In fact, the solutions provided respond to a need for common interoperable tools among public administrations and help mitigate budgetary concerns to a certain extent. Besides, ISA2 action owners emphasised that national initiatives aimed at the ICT modernisation of the public sector are also contributing to a great extent to the performance of ISA2. Figure 10: Extent to which the following external factors are contributing to the performance of ISA2 (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely.

Note: 1) Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 2) The average score is based on the answers received through targeted consultations. Total number of respondents for each external factor from top to bottom: 51, 50, 53. Source: CEPS final study. When it comes to standardisation, experts emphasised its importance based on several trends. Addressing needs such as improved security and the introduction of new technologies could lead to wider calls for standardisation and common frameworks. Ensuring information security is likely to be one of the most important areas for the application of standards. In a similar vein, the use of blockchain for public service execution will need to rely on further developments in standardisation. A further relevant trend is the increasing role of open source in standardisation, which is likely to drive even more developments in the field of public sector interoperable solutions. During the consultation activities, respondents to the targeted consultations identified further external factors having an impact on the performance of the programme. The COVID-19 pandemic stands out as a unique factor, a shock that has prompted discussions about the urgent need for digital and interoperable solutions to manage the ensuing public health crisis. The pandemic has thus been an important factor in bringing the issues of interoperability into the spotlight, underlining the importance of the ISA2 programme. As part of ISA2, the ‘ Digital Response to COVID-19 ’ collection was set up on Joinup to bring together a variety of resources including tools and data (such as open source software, open data, websites, platforms and events) that could be useful for public administrations, citizens and businesses in managing the pandemic. Other external factors that influenced the programmes performance are the new ministerial declarations that reinforce the political support for interoperability initiatives – like the Tallinn Declaration in 2017 and the Berlin Declaration in December 2020 –, as well as technological advancements and the general digital transformation of the society. Nevertheless, there are also factors that had a negative effect on the programme’s performance ( Figure 11 ). Legal and institutional complexity stand out as factors having a negative impact to a great extent on how the programme delivers its results (across all stakeholder groups, the legal complexity scored an average of 3.73 out of 5 and the institutional complexities scored an average of 3.64 out of 5). Experts and academia have been the most vocal in expressing their concerns about the impact of these factors. On the one hand, legal complexities arise from different rules and legal requirements that may limit the flexibility of public administrations to adopt flexible and interoperable solutions for the delivery of (digital) public services. EU countries have called for action in the field of digital-ready policymaking including the exchange of best practice in the Berlin Declaration . The Danish approach to foster digital-ready legislation has been very successful. Institutional complexity, on the other hand, derives from the different levels of governance in the EU and the different organisational setups of public administrations at the national, regional, and local levels. Among the consulted stakeholder groups, experts and academia in particular consider that institutional complexity can jeopardise the programme’s performance to a great extent (with average scores of 4.23 out of 5 based on 13 respondents). Institutional complexity is exacerbated by the presence of silos in institutions. Figure 11: Extent to which the following external factors are jeopardising the performance of ISA2 (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: 1) Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 2) The average score is based on the answers received through targeted consultations. Total number of respondents for each external factor from top to bottom: 57, 56, 56, 56. Source: CEPS final study. Stakeholders consulted via the targeted consultation activities also indicate that technical challenges have a negative impact on the performance of the programme. The independent expert assessments reinforce this point. For example, the update of existing legacy solutions does not only imply costs and efforts, but also the strategic decisions to be made. In particular, the question of which provider to choose becomes central – whether this means private companies, EU-based or not, or public sector solutions. The feedback from stakeholders pointed out additional negative factors for the programme’s performance and needs that should be addressed: ·Missing interest and understanding of interoperability from the non-technical audience: As the subject is fairly technical in nature but needs action beyond the technical level to become effective the communication needs to be as clear as possible and fit for a wider audience with non-technical background. Important interoperability decisions cannot be delegated to a technical level but need commitment at policy level. ·Another factor linked to organisational interoperability is the need for stronger connections between ISA2 actions and the interoperability related activities of other services (DGs) of the Commission. ·Finally, there continues to be a certain “natural” resistance of public organisations to the disclosure of their data, as documented in an analysis by Ruijer, Détienne, Baker, Groff, Meijer (2020) . ISA2 outputs Data show that the 21 sampled actions have developed 72 outputs including 34 solutions in the form of common tools/services, frameworks, specifications or standards as well as 38 publications so far, out of which 9 emerged under ISA2. As shown in Table 3 , the solutions fall into one of the following output categories: common tools/services, common frameworks and common specifications or standards. In addition, the sampled actions also created 38 publications, with the majority having been published in the programme’s second half (see Annex 6.d). Table 3: Overview of solutions developed under the sampled actions, by type

Type of solution Number of solutions New solutions developed by ISA2 Common tools/services 22 6 Common frameworks 9 2 Common specifications/standards 3 1 Publications 38 TOTAL 72 47 Source: CEPS final study. Aside from the above listed four solution types, for the action Raising Interoperability Awareness – Communication Activities, which is part of the Accompanying measures package, the action outputs are represented by events organised as part of the ISA2 programme or events to which ISA2 representatives actively contributed (see Box 6 and Annex 6.e). Box 6: Communication activities Through the events organised as part of the ISA2 programme, including conferences, workshops, and webinars, the programme reached diverse stakeholders from different countries, helping to promote interoperability among them. Throughout the duration of the programme, a total of 196 ISA2 funded events have been organised across the EU, both online and in several countries: Belgium, Bulgaria, Finland, Greece, Italy, Luxembourg, Malta and Romania (see Annex 6.e). With the outbreak of the COVID-19 pandemic, most of the events organised in 2020 were exclusively online. In addition, ISA2 representatives actively participated in events that took place in 20 EU countries and two non-EU countries, namely Montenegro and Serbia. Performance indicators and principles of the programme An array of metrics is in use to assess the effectiveness of ISA2 solutions, for example, the number of users, instances of use, the number of downloads and page views. The performance of ISA2 solutions can thus be characterised in multiple ways, depending on the type of output. Examples of the outputs include : ·16,092 professionals working in the field of e-Government are registered on Joinup to access interoperability solutions and collaborate with each other. ·Over 29,200 surveys have been created so far using the ISA2 EU Survey solution. ·The European Single Procurement Document (ESPD) solution has been downloaded over 35,000 times from Joinup. ·36 educational and training resources have been published in the Interoperability Academy Catalogue of Educational and Training Resources, designed to enhance digital skills and support the delivery of digital and interoperable public services. ·141 assessments of digital public services were performed using the Interoperability Maturity Assessment of a Public Service (IMAPS) solution between 2018 and 2020. Additional performance indicators, collected as part of the desk review of secondary data, are presented throughout the chapter. A full overview is included in Annex 6.c. The rolling work programme provides details on how the ISA2 actions should consider the principles listed in Article 4(b) of the ISA2 Decision. The overall judgment stemming from the desk analysis of the rolling work programme is positive (Annex 6.f). Examples of ISA2 actions contributing to the principles include: ·To ensure transparency, the ISA2 programme has taken several measures: i) the rolling work programme provides an overview of the objectives of the actions, the planned and developed solutions, the expected impacts, and the budget allocated to each action; ii) the ISA2 dashboard provides quarterly updates regarding the efficiency and effectiveness of actions in terms of costs, earned value management, effectiveness indicators and targets; iii) information on the level of take-up of solutions is made available via the ISA2 webpage dedicated to solutions; and iv) the solutions developed can be accessed via the Joinup platform. ·Concerning reusability and avoiding duplications, the process of submitting proposals for actions played an important part, as the descriptions of proposed actions had to specify (i) the extent to which the action re-uses other readily available solutions and (ii) the re-usability of the action outputs. In conclusion, the ISA2 programme achieved all its objectives at least to some extent. The main achievements of ISA2 consist in the support to the implementation of EU policies and actions through interoperability solutions, the facilitation of the re-use of interoperability solutions, and the contribution to the promotion of a holistic approach to interoperability in the EU. More could be done in the future to fully achieve the objectives that were pursued by the programme. According to the consulted stakeholder, the take-up of solutions could be further increased for example by focussing the efforts on developing a smaller set of key mature solutions, continually developing them and nurturing their take-up.

External factors could improve but also jeopardise the way in which the programme achieves its objectives and delivers its results. The need of public administrations to increase their efficiency is an external factor that contributes to the programme’s performance. In contrast, legal and institutional complexity are in many cases blocking cross-border and cross-sectoral interoperability. An area of improvement is the identification of legislation gaps, both at the EU and national levels, that hamper cross-border and cross-sectoral interoperability and the assessment of ICT implications of Union law. In addition, considering the importance of standardisation for interoperability, a stronger and more systematic link with standards developing organisations could further improve the results achieved so far. Efficiency Efficiency relates to the extent to which the programme’s objectives are achieved at a minimum cost. Throughout the duration of the ISA2 programme, its efficiency has been tracked using the Earned Value Management (EVM) methodology . Calculation of the EVM helped monitoring the work progress compared to planning, taking into account the costs, time needed, and scope of the ISA2 actions. To support this, the ISA2 dashboard provides quarterly data on the earned value, planned value, and actual costs of each monitored action. Based on data available on the ISA2 dashboard, which covers the timeframe between June 2016 and October 2020 (the latest available data point at the time of data collection), the progress made in implementing the programme between 2016 and 2020 can be analysed. The key indicators for this analysis are the following: ·Earned value, namely the value of actual progress made compared to the budget; ·Planned value, i.e., the benchmark against which the value of the actual progress is measured in terms of time and costs; and ·Actual cost, representing the budget spent. Figure 12 provides an overview of the main indicators for the third quarter of 2020. The 21 sampled actions were aggregated at the package level to provide a comparative view. The earned value is nearly equal to the planned value for all sampled actions, showing that the work has progressed as expected. Final activities are being undertaken during 2021 to finalise the programme’s activities and transition to the Digital Europe Programme. Figure 12 Earned Value, Actual Costs and Planned Value of sampled actions, by package (Q4 2020) Source: CEPS final study. To track how the programme’s implementation has evolved over its lifetime, computing the Schedule Performance Index (SPI), which represents the ratio of earned value to planned value, provides further insights ( Figure 13 ). An SPI value higher than one indicates that a given ISA2 package is ahead of schedule, while an SPI value below one suggests that the analysed package is behind schedule. From the programme’s start in 2016 to the fourth quarter of 2020, the SPIs of the sampled actions grouped by packages have shown fluctuations but have been converging on one. In Q4 2020, the computed indices for most packages were almost equal to one. The more striking fluctuations, as in the cases of the “1.Key and generic interoperability enablers” and “6.Decision making and legislation – Supporting instruments” packages, during the first two years of the programme can be explained by the fact that some of the actions rolled out under ISA2 are continuations of actions from the previous edition of the programme. The ‘continuity’ factor can play an important role in the implementation of actions and contribute in the beginning to faster progress than expected. Given that some actions were started later on in the programme, as is the case of the “4.Geospatial solutions” package, data for the EVM analysis are not always available from Q2 of 2016, but only from a later point in time. Figure 13 Schedule Performance Indices of the sampled actions grouped by packages Source: CEPS final study. The assessment of the programme’s cost-effectiveness relies on the analysis of the costs per output. However, the application of this method in the case of the ISA2 programme has an important limitation. Given the diversity of ISA2 actions and types of outputs, a full-scale analysis and comparison of the cost-effectiveness of ISA2 actions and solutions is difficult to carry out. The ISA2 dashboard provides an overview of the budget at completion for the ISA2 actions as well as the actual costs incurred for implementing the actions. Figure 14 presents the breakdown of the budget at completion compared to the actual costs for the sample of actions grouped by their respective packages for the period 2016 – 2020 (the latest data point available being October 2020). The actual costs of the packages shown tend to be close to their foreseen budgets. The average budget for an ISA2 action between 2016 and 2020 was of € 2,025,193.

Figure 14 Budget at completion and actual costs of the sampled actions grouped by packages Source: CEPS final study. To assess the costs per output the evaluation compares packages that use the same performance indicators. The sampled actions in two packages rely on the same indicator, namely the number of public administrations using the solutions that were developed: the “5. e-Procurement/e-invoicing-Supporting instruments” and “1. Key and generic interoperability enablers” packages. By taking the actual costs of the sampled actions in the two packages and the total number of public administrations using the solutions of the packages (the full overview of performance indicators can be consulted in Annex 3), the average actual cost per public administration using the solutions can be calculated (see Table 4 ). The average costs per public administration are estimated at €33,917 for the “e-Procurement/e-invoicing-Supporting instruments” package and €23,470 for the “Key and generic interoperability enablers” package. This analysis, however, does not capture the effectiveness of the actions in terms of end-users. The services of public administrations are used by thousands of citizens and businesses, meaning that ultimately the costs are much lower relative to the number of end-users, and thus the benefits greater than the costs. Table 4 Costs per user (public administrations) for ISA2 solutions Package Actual costs of sampled actions in the package (€) Number of public administrations using the solutions of the package Average actual cost (€) e-Procurement/e-invoicing-Supporting instruments 4,646,617 170 27,333 Key and generic interoperability enablers 4,975,794 212 23,470 Note: The calculations are based on the sampled actions of the indicated ISA2 packages over the period 2016 - 2020. The total number of public administrations using the solutions of the “Key and generic interoperability enablers” package consists of approximately 200 public administrations using “e-TrustEx” and 12 public administrations using “Core Public Service Vocabulary Application Profile (CPSV-AP)”. The total number of public administrations using the solutions of the “eProcurement / eInvoicing – Supporting instruments” package includes 70 public administrations using Open e-Prior, roughly 75 solutions either using the ESPD data model or the open source version of the ESPD service developed under ISA2, and 25 connections from Member State solutions to eCertis. The full overview of the number of users and other performance indicators is presented in Annex 3. The overview of costs per package is presented in Figure 14 . Source: CEPS final study. The annual process of selection of actions to be included in the Rolling Work Programme was a central part of the functioning of ISA2. The preparation and submission of a proposal to be included in the ISA2 Rolling Work Programme required the applicant to perform several activities such as: I)finding out about the call for proposals; II)studying the documents of the call for proposals and understanding the rules and procedures; III)preparing a concise description of the proposed action in compliance with the “work programme entry template”; IV)collecting the required internal authorisations to submit the proposal; and V)submitting the proposal by e-mail. Figure 15 shows that consulted action owners and stakeholders involved in the governance of the selection process for actions to be included in the programme was to a great extent fit for purpose (the assessment is based on the responses received from 12 action owners and six stakeholders involved in the governance of the programme). Respondents emphasised that one of the positive aspects was that EU countries could submit proposals, alongside European Commission services. Nevertheless, only a limited number of proposals were received from the EU countries. In the future new ways of fostering active involvement of EU countries and other public administrations should be explored. Figure 15 Extent to which the selection process of the actions is fit-for-purpose (number of respondents by stakeholder category) Note: Averages do not account for respondents answering “don’t know/no opinion” (DK/NO).

Source: CEPS final study . Total number of respondents: 18. In addition, twelve respondents (action owners and stakeholders involved in the governance of the programme) provided further details based on their experience in having submitted proposals and the proposals having been accepted and included in the rolling work programmes of ISA2. The time spent on preparing the proposals varied from two to three person-days up to 25 person-days to complete the proposal (see Figure 16 Figure 16 Number of person-days required for preparing a proposal for an ISA 2 action ). The differences in the reported time required to prepare a proposal reflect the types of actions put forward such as new actions or previous actions for which one can rely on previously accumulated experience. Further information from the ISA2 interim evaluation shows that the amount of time spent on preparing a new proposal – instead of updating an existing one – can be up to 30 person-days. This suggests that a greater effort may be expected at the beginning of the programme, as applicants get acquainted with the requirements of the selection process and draft initial proposals for actions. The subsequent applications would focus mainly on updating previously accepted proposals, reducing the time required. Figure 16 Number of person-days required for preparing a proposal for an ISA2 action Note: This chart presents the distribution of person-days needed to prepare an ISA2 proposal grouped in intervals of two days to facilitate the overview of data. This chart combines the feedback from action owners and programme governance stakeholders consulted for the final evaluation of the programme as well as the interim evaluation of the programme. Source: CEPS final study . To get an overview of the administrative costs associated with the preparation of proposals for ISA2 actions, the Standard Cost Model can be applied. Applying this methodology based on the respective Better Regulation Tool, the estimates of person-days spent preparing proposals for ISA2 actions are multiplied by a standard tariff, which in this case is represented by the hourly labour cost in the respective EU countries, factoring in an average person-day composed of eight hours. Building on the feedback from consulted stakeholders and the findings of the interim evaluation study, the standard cost model analysis in this case can distinguish between the updating of a proposal for an existing action (with an estimated average time spent of up to 6 days) and the preparation of a proposal for a new action, for which the time spent increases significantly, from 10 person-days to 30 person-days. Against this background, the time spent in the process of preparing proposals for ISA2 actions can thus be translated into costs as follows (the underlying data can be consulted in Annex 6.g): ·A renewed proposal has cost approximately € 1,126; ·A new proposal has cost approximately € 5,669. To put these numbers into context, it is worth noting that the average budget for an ISA2 action between 2016 and 2020 was € 2,025,193. The costs incurred in preparing a proposal are thus very small, as illustrated in the following scenario. Let us assume that a proposal for a new action was prepared and included in the 2016 Rolling Work Programme and then updated annually through 2020 (four updates in total). The total administrative costs throughout the programme’s duration would thus amount to an average of € 10,173, which in relative terms would represent only 0.5% of the average total budget of the action. In the case of ISA2, one respondent pointed out that the part of the application form regarding the work programme was complicated. The templates should be shorter, and the level of granularity should be reduced to provide more flexibility and ease in modifying the planned activities. In conclusion, the work across ISA2 packages has progressed as expected relative to the planned work and budget. The heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness. For those packages where it was possible to apply cost-effectiveness techniques, costs per end-user (e.g.: business, citizens, etc.) have been estimated as low and thus benefits are greater than costs for the ultimate beneficiaries of the solutions developed. The process of selecting actions to be funded by ISA2 is considered relatively efficient and fit for purpose. The costs of preparing and submitting proposals as part of the selection process were very small relative to the average budget of an ISA2 action. EU countries could submit proposals, alongside European Commission services. Nevertheless, only a limited number of proposals were received from the EU countries. Coherence Article 13(4) of the ISA2 Decision requires that the coherence of ISA2 actions is assessed; this is referred to as the programme’s ‘internal coherence’. In addition, Article 13(6) pays specific attention to the programme’s ‘external coherence’ in order to ‘identify potential overlaps, examine coherence with areas for improvement, and verify synergies with other Union initiatives, in particular with the CEF’. To assess both the internal and the external coherence of the programme, findings from desk research on links between ISA2 actions and the relationship between ISA2 and other relevant EU programmes/policies/initiatives were used in addition to the information provided by stakeholders.

Internal coherence Given that the ISA2 programme currently supports 54 actions across nine work packages with multiple Commission services involved in the implementation of the actions, coordination and coherence among the multiple activities rolled out was an essential part of the programme’s performance (internal coherence). To this end, assessing the level of synergies or overlaps between ISA2 actions is relevant. The stakeholder consultation and desk research show to some extent synergies between the actions. Consulted ISA2 action owners, stakeholders involved in the implementation of linked EU initiatives, and solution users (with the caveat that only three solution users expressed their views on this matter) see the most synergies ( Figure 17 ). Yet respondents also emphasised that the programme’s fragmentation into multiple different actions makes it difficult to fully tap into all synergies. Figure 17 Extent to which synergies and overlaps between ISA2 actions exist (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). Source: CEPS final study . Total number of respondents: 49 (Synergies) and 43 (Overlaps). Consulted stakeholders see only limited overlaps between the ISA2 actions (see Figure 17 ), which tend to be mostly functional overlaps needed for the cohesive functioning of the programme. Additional evidence from the desk review of information available on the ISA2 dashboard allows gaining an overview of the network of ISA2 actions and the links between them, i.e., actions being re-used or contributing to other actions. There are multiple instances of re-use and contributions, as shown in Table 5 . Several actions play a central role, with their solutions and outputs being widely re-used by other actions, such as is the case for the SEMIC and Joinup actions. Table 5 Number of links between the sampled actions Actions Number of other ISA² solutions that are used by the action Number of other ISA² actions that use the solutions of the action 1. Key and generic interoperability enablers Trusted Exchange Platform (e-TrustEx) 5 9 Catalogue of Services 3 3 2. Semantic Interoperability Public Multilingual Knowledge Management Infrastructure for the DSM (PMKI) 2 0 SEMIC: Promoting Semantic Interoperability Amongst the European Union EU countries 6 23 3. Access to data/data sharing/open data Big Data for Public Administrations 6 0 Sharing Statistical Production and Dissemination Services and Solutions in the European Statistical System 6 2 Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification 2 0 4. Geospatial Solutions European Location Interoperability Solutions for e-Government (ELISE) 9 2 5. eProcurement/eInvoicing-Supporting instruments European Public Procurement Interoperability Initiative 8 4 6. Decision making and legislation-Supporting instruments Legal interoperability 9 0 REFIT Platform 0 3 Inter-Institutional Register of Delegated Acts 1 1 7. EU Policies-Supporting instruments European Citizens' Initiatives and European Parliament Elections 3 0 8. Supporting instruments for public administrations Joinup - European Collaborative Platform and Catalogue 7 14 NIFO 9 0 EIA 7 9 EUSurvey 2 0 Interoperability Maturity Model 11 4 Standard-Based Archival Data Management, Exchange and Publication 0 1 9. Accompanying measures Raising Interoperability Awareness - Communication Activities 1 0 Source: CEPS final study Based on the findings from consulted desk research and the review of desk evidence on ISA2 actions and solutions, concrete examples of synergies include: ·As part of the ELISE action, instances of synergies include the supply of indicators from Location Interoperability Framework Observatory (LIFO) to the NIFO and the re-use of DCAT-AP , which is developed under the SEMIC action, for the Geo-DCAT, an extension of DCAT-AP which described geospatial datasets, dataset series and services. ·The Innovative Public Services action, exploring how new digital technologies can enhance the provision of public services, has synergies with actions including: the Legal interoperability action (which also experimented with potential applications of new technologies) and the FISMA: Financial data standardisation action (which explored the use of machine learning for reporting), as well as the ELISE action (which carried out studies on disruptive technological developments for linked-to-location data). ·The Interoperability Academy action is fostering re-use by increasing awareness of the solutions developed as part of other ISA2 actions.

·Through its support for the implementation and monitoring of the EIF, the ISA2 programme also contributed to actions #4 and #6 of the eGovernment Action Plan calling for support for the take-up of the new EIF. ·The role played by the “ELISE” action in implementing the INSPIRE Directive also fed into action #19 of the eGovernment Action Plan on the deployment and take-up of the INSPIRE Directive data infrastructure. In addition, it is important to discuss the role played by standardisation in the field of interoperability. In this context, it is worth assessing the coherence between the Multi-Stakeholder Platform for ICT standardisation and ISA2 to draw lessons for the future. The Platform brings together experts acting as an advisory group for the Commission on matters related to the implementation of ICT standardisation policies. Among their duties, the group members support the Commission in preparing the annual rolling plan for ICT standardisation which serves to outline those key EU policy areas for which ICT standards and specifications are necessary for the successful implementation of the policies. Regarding the work conducted under ISA2, the rolling plan puts the emphasis on further developing existing specifications and enhancing their take-up, for instance with regard to the Core Vocabularies and CPSV-AP, turning specifications into internationally accepted standards, such as is the case with the DCAT-AP specifications, and building on the existing work done in the field of e-Procurement solutions as part of ISA2. Stakeholders emphasised the need for ·systematic links between the public sector interoperability work and the EU standardisation to make sure that needs for public sector interoperability are taken into account, to help that future interoperability initiatives fully benefit from the outcome of ongoing standardisation and to promote mature interoperability solutions within the standardisation work. ·Increased awareness in the public sector on the key role of standards in the digital transformation strategy and in fully supporting the twin green and digital transitions, engaging national standards organisations and national trade associations. Finally, public sector interoperability is an issue that is brought up in international fora as well. At the international level, the work of the OECD on Digital Government lends itself to a comparison with the activities undertaken under ISA2 and the EIF. In particular, the “ Recommendation on Digital Government ”, adopted in 2014 and comprising high-level recommendations, and the “ Digital Government Toolkit ” are relevant, with the toolkit providing support for the implementation of the high-level recommendations. The OECD approach is similar to the EIF and ISA2 in the sense that a strategic framework is established through the high-level recommendations of the OECD, while the EIF serves as the strategic framework for the specific EU approach in the field. To complement the recommendations, the OECD brings in a more practical component through the “Digital Government Toolkit”. In the EU, the practical aspect is represented by ISA2. The OECD recommendations and the work done as part of the EIF and ISA2 are mutually reinforcing in their messages, particularly when it comes to: ·‘Ensuring greater transparency, openness and inclusiveness of government processes and operations’ as part of digital government strategies (Recommendation 1). ·‘Creating a data-driven culture in the public sector' as part of digital government strategies, with a strong focus on access and re-use of data (Recommendation 3). ·‘Ensuring coherent use of digital technologies across policy areas and levels of government’ in developing digital government strategies (Recommendation 6). Yet the approach taken by the OECD is broader and less granular than what the EIF and ISA2 set out to achieve. The toolkit provides an overview of principles, best practices related to the principles, and a self-assessment of the stages of development in digital government. The approach to the toolkit is again not as granular as the tools and solutions provided by ISA2 in support of the implementation of the EIF. Overall, while the OECD provides guidance and a set of recommendations to help public authorities design the main directions of their digital government strategies, such an approach does not guarantee a harmonised landscape of digital and interoperable public sectors. The approach taken through ISA2 brings added value through more concrete interoperable solutions that support enhanced cross-border interoperability (see Chapter 5.5 for a discussion of the EU added value of the programme). In conclusion, analysing the links between sampled actions, there are multiple instances of re-use and contributions across ISA2 actions. Several actions play a central role, with their solutions and outputs being widely re-used by other actions. ISA2 actions could to some extent create synergies with each other and limited overlaps. However, the programme’s fragmentation into multiple different actions makes it difficult to fully benefit from all synergies. This finding is reinforced through the analysis of the effectiveness of ISA2 (see Chapter 5.2 ). Multiple links are observed between ISA2 actions and other relevant EU programmes, policies and initiatives. Interoperability actions were an enabler for the implementation of different EU policies with high digital impacts. Overlaps remain relatively limited, but they can appear between ISA2 and funding instruments for public sector digitalisation. ISA2 actions have contributed to the work of the Multi-Stakeholder Platform for ICT standardisation . Looking at international initiatives, ISA2 takes an approach that is consistent with the recommendations on Digital Government put forward by the OECD. The ISA2 programme went further than international initiatives by providing concrete interoperable solutions to help harmonise interoperability in the public sector. EU added value The EU added value captures the programme's impacts additional to those that would be achieved if the issues addressed by ISA2 were left solely in the hands of national and sub-national authorities and the contribution to the advancement of common EU policies. A very tangible proof of the EU added value was the clear call for European interoperability solutions for the public sector with the still on-going COVID crisis. In response to some of the disruptions experienced, and to facilitate cooperation, the exchange of best practices and solution re-use, the ”Digital Response to COVID-19” collection was set up on the Joinup platform as part of the ISA2 programme. This collection has brought together a variety of resources including tools and data (such as open-source software, open data, websites, platforms and events) to support public administrations, citizens and businesses in managing the pandemic. There is a key finding from the pandemic experience: ongoing work on interoperability can help build resilience across the EU and equip the public sector to respond more effectively to the general needs of citizens, businesses and public administrations, but it can also become a key component in times of an unexpected crisis. In general, most respondents confirm that national or sub-national initiatives alone would have brought only limited contributions toward most of the objectives (see Figure 19 ). Consulted solution users, national and sub-national public administrations and stakeholders responsible for the programme governance themselves consider that initiatives taken at the Member State level as opposed to the EU level would have only brought limited achievements and thus an EU-level approach is better suited to address the challenges related to the interoperability of public administrations and services. The only exception is “Specific objective 3: To contribute to the development of more effective, simplified and user-friendly public e-administration at the national, regional and local levels”. As the focus in this case is substantially on the different levels of public administration within EU countries, the respondents pointed out that national or sub-national initiatives would be necessary at least to some extent to ensure that the objective is achieved.

Figure 19 Extent to which national or sub-national interventions would be able to achieve the ISA2 objectives in the absence of the programme (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). Source: CEPS final study . Total number of respondents: 96 (General objective), 96 (Specific objective 1), 95 (Specific objective 2), 95 (Specific objective 3), 92 (Specific objective 4), 94 (Specific objective 5). Respondents across all stakeholder groups confirm that ISA2 can probably achieve its objectives at costs that are lower than the costs of national or sub-national interventions (see Figure 20 ). In particular, consulted solution users and stakeholders involved in the implementation of linked EU policies/initiatives consider that the costs are almost definitely lower. Figure 20 Extent to which an EU-level intervention would be able to achieve the objectives of ISA2 at a lower cost than comparable national or sub-national interventions (breakdown by group of stakeholders; average score and number of respondents) Score: (-2) definitely would not; (-1) probably would not; (1) probably would; (2) definitely would; Note: Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). Source: CEPS final study . Total number of respondents: 89. The focus of the ISA2 programme is to apply a cross-cutting approach, aiming to enhance interoperability both across sectors and across borders. The latter aspect is particularly important in understanding to what extent the programme has brought additional EU value by contributing to a more coherent interoperability landscape. In this respect, consulted stakeholders consider that ISA2 has contributed to some extent to enhancing cross-border interoperability (see Figure 21 ). While noting the achievements of the programme, respondents also pointed out two aspects that need to be considered in this analysis. First, the programme is relatively small and is part of a wider landscape of initiatives and policies that are needed to support cross-border interoperability. Many EU instruments have complemented the programme (see chapter 3.2 ). Second, achievements in terms of cross-border interoperability also depend on their legal basis, the take-up of solutions and this aspect depends on the public administrations involved. The extent to which the programme can work closely with public administrations and promote public administrations’ take-up of the solutions play an important role. Figure 21 Extent to which ISA2 contributed to enhancing cross-border interoperability in the EU (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). Source: CEPS final study . Total number of respondents: 55.

The performance indicators for the sampled actions and solutions as well as the overview of the take-up of solutions by EU countries provide additional evidence of the cross-border contributions of ISA2. There are instances of re-use of ISA2 solutions by public administrations in the EU countries, with some solutions being consistently taken up (for further details see Annex 6.b). One of the key ways in which ISA2 contributed to cross-border interoperability was the awareness raising about the importance of interoperability. Through its communication activities, ISA2 reached varied stakeholders in different EU countries. The “Interoperability Academy” complemented these awareness raising efforts by trying to empower stakeholders with more information, help enhance skills and promote the ISA2 solutions in order to improve the overall interoperability landscape in the EU (even if it has only been implemented starting with 2019). Respondents across stakeholder groups generally consider that ISA2 contributed to some extent to the advancement of common EU policies (see Figure 22 ). Importantly, the extent of the contribution depends on the specific EU policies that are considered. Figure 22 Extent to which ISA2 contributed to the advancement of common EU policies (breakdown by group of stakeholders; average score and number of respondents) Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: Averages do not account for respondents answering “don’t know/no opinion” (DK/NO). Source: CEPS final study . Total number of respondents: 55. ISA2 played a central role in the implementation of the current EU interoperability policy. As one of the main implementing instruments of the European Interoperability Framework (EIF) and the Interoperability Action Plan (IAP), ISA2 has provided the operational arm of the strategic framework for public sector interoperability in the EU. The 2017 Communication on the new EIF defined five focus areas to guide the implementation of the EIF. The IAP annexed to the Communication put forward a list of 22 actions grouped under the five focus areas thus setting strategic priorities until 2020. The contributions of ISA2 actions range from overarching support for monitoring the implementation of the EIF to specific support to certain principles, recommendations, and parts of the IAP. The ISA2 governance was also overseeing the implementation of the EIF. Noteworthy examples, highlighting the different types of contributions, include the following: ·At the overarching level, the NIFO action played a central role. The NIFO action has helped implement the commitment stated in the 2017 EIF Communication to create a framework for monitoring the progress in implementing the EIF. This has led to the development of the EIF Monitoring Mechanism with the first results available for 2019. The Monitoring Mechanism takes stock of the three main components of the EIF and the accompanying 47 recommendations, basing the analysis on 68 KPIs. Beyond the EIF Monitoring Mechanism, NIFO also monitors the developments made across the EU in terms of digital public administrations and interoperability (through the yearly Digital Public Administration factsheets as well as digital policy reports) and developed the EIF Toolbox to support public administrations with the implementation of the EIF. ·When it comes to contributions to specific parts of the EIF, several examples are illustrative of the breadth of topics covered by ISA2 actions: oThe EIA action helped define the needs and shortcomings related to a common interoperability architecture for European public services and contribute to defining such an architecture as well as map reusable solutions and guidelines services as interoperability building blocks. oThe Public Multilingual Knowledge Management Infrastructure for the Digital Single Market action aimed to support EU public administrations in creating services that can be accessible and shareable regardless of the language actually used, as well as allowing SMEs to sell goods and service cross-border in a DSM. o Legal Interoperability : The Legal Interoperability action supports policymaking across policy areas, bringing to the forefront the importance of considering potential digital impacts and the role of interoperability when developing new legislation. This ISA2 action implements: §Action 3 of the IAP, by raising awareness on the importance of considering interoperability early on in the legislative process and by developing a methodology for legal interoperability screening; §Actions 19 and 20 of the IAP, by having put forward guidelines for ICT impact assessment as part of the Better Regulation Toolbox (Tool #27) and guidelines for digital-ready policy proposals. o Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification : The action supports open data initiatives by facilitating data re-use and sharing and offering tools to visualise data effectively. The action contributes to several priorities listed in the IAP: organisational interoperability (Actions 6 and 7); sharing of good practices (Action 11); governance structure (Action 2) and key enablers focused on EU open data initiative (Action 14). o Interoperability Academy : This action was established to help increase awareness of interoperability, the EIF and the solutions developed under ISA2. The action facilitates access to information and learning material in this sense. The Interoperability Academy contributes primarily to Actions 5, 8, 11 of the IAP.

·Concerning the governance of the implementation of the EIF and the IAP, the ISA2 governance played a crucial role. When moving to the Digital Europe Programme with a wider portfolio beyond public sector interoperability, the DEP governance body might not be the right place to provide an enhanced governance of interoperability in the EU (see also findings in Chapter 5.1 ). Further details on the contribution of sampled actions to the implementation of the EIF can be consulted in Annex 6.k. The contribution of ISA2 actions to the EIF was taken into account already in the design stage of the actions: the template used for submitting new or updated proposals for ISA2 actions for the yearly selection included the requirement to explain how the proposed action would contribute to the implementation of the EIF and the IAP. Beyond the EIF, the contribution of ISA2 to wider policies such as the Digital Single Market Strategy (DSM) is deemed more limited, due to the fact that ISA2 is only one of the instruments that are meant to contribute to such overarching policies. The programme’s actions have contributed to specific areas of the DSM Strategy from the perspective of interoperability as a key enabler of digitalisation. Importantly, a key action listed in the 2015 Communication on the DSM Strategy under the ambition of “Boosting competitiveness through interoperability and standardisation”, namely the revision of the EIF, was achieved under ISA2 in 2017, with subsequent monitoring of the implementation of the revised Framework having been ensured through the programme as well. Further details on how specifically ISA2 has contributed to the EIF and the DSM can be consulted in Annex 6.h. In conclusion, ISA2 provided clear EU added value. National or sub-national initiatives alone would have made only limited contributions towards most of the objectives. In addition, ISA2 was able to achieve these objectives at a lower cost than comparable national or sub-national initiatives. These conclusions hold true across the different stakeholder groups and are further corroborated by expert assessment. Instances of re-use of ISA2 solutions by public administrations in the EU countries contribute to enhancing cross-border interoperabili