EUROPEAN COMMISSION Brussels, 18.11.2022 SWD(2022) 720 final COMMISSION STAFF WORKING DOCUMENT Final evaluation of the European Interoperability Framework (EIF) Accompanying the document Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL laying down measures for a high level of public sector interoperability across the Union (Interoperable Europe Act) {COM(2022) 720 final} - {SEC(2022) 720 final} - {SWD(2022) 721 final} - {SWD(2022) 722 final}



Table of Contents Executive summary 1.Introduction 2.WHAT IS THE EXPECTED OUTCOME OF THE INTERVENTION? 2.1Introduction to interoperability 2.2Introduction to the EIF 2.3Description of the intervention and its objectives 2.4Points of comparison 3.How has the situation evolved over the evaluation period? 3.1Policy context 3.2State of play 4.Evaluation findings (analysis) 4.1To what extent was the 2017 EIF successful and why? 4.2How did EU intervention make a difference? 4.3Is the EIF still relevant? 5.What are the conclusions and lessons learnt? 5.1Conclusions 5.2Lessons learnt Annexes

Annex I. Procedural information Lead DG: Directorate General for Informatics Organisation and timing Consultation of the RSB Evidence, sources and quality Annex II. Methodology and analytical models used Overview of information and data sources Primary data Overview of the level of knowledge of respondents Secondary data Data validation Methods Main limitations Annex III. Evaluation framework for the EIF evaluation Annex IV. Overview of benefits and costs Annex V. Stakeholder consultation Summary of the stakeholder strategy Summary of the feedback on the Inception impact assessment Consultation activities for the EIF evaluation and Impact assessment Results: EIF Evaluation

Input of the Expert Group co-design process Annex VI. Supporting evidence from desk research Annex VI.1. Scoreboards monitoring the implementation of the EIF Annex VI.2. Overview of the adoption of NIFs Annex VI.3. Overview of the mentions of EIF in a sample of EU pieces of legislation and policy documents Annex VI.4. Evolution of the quality of digital public services and citizens’ use of the Internet to interact with public administrations between 2012 and 2019 Annex VI.5. Academic and grey literature outlining the needs and problems in the field of interoperability Annex VII. Solutions contributing to the EIF implementation Annex VIII. Mapping of legal initiatives to the EIF Executive summary Interoperability enables data exchanges and facilitates connections between IT systems and platforms in the delivery of digital services. It covers broader organisational, legal and governance aspects enabling fundamental user-centric principles like the once-only principle. The EU has been supporting interoperability between public administrations since the 1990s through a series of funding programmes such as IDA, IDABC, ISA, ISA² and now Interoperable Europe . As part of this support, a European interoperability framework (EIF) was first produced in 2004 and then updated in 2010 and 2017 . With the introduction of the EIF for the first time in 2004, there was a first agreement on a common vision and non-binding guidelines to support data exchanges and digitalisation in public administrations in Europe. The implementation of a new EIF, adopted in 2017 as COM(2017) 134, was recognised as a key element in enhancing interoperable public services, especially in a cross-border context. It contains a set of principles, models and recommendations to guide public administrations across the EU and at all levels of administration in the design and provision of key digital and interoperable public services. The EIF recognises the importance of cross-border interoperability as a means of implementing EU-wide policies. It provides guidance to public administrations on the design and update of national interoperability frameworks (NIFs), or national policies, strategies and guidelines promoting interoperability. The evaluation shows that the EIF has certainly helped inspire European public administrations, on a strategic level, to design and deliver digital public services, and in the design and updating of (NIFs) or digital strategies. Quite a lot of Member States have aligned their digitalisation policies and plans with the EIF, or at least with its guiding principles. The framework has provided high-level advice through concepts and models to help guide the creation of interoperable digital public services. However, there is a disconnect between high-level guidance and actual execution or operational implementation, particularly at regional and local level. The concepts, layered interoperability architecture, and conceptual model for integrated public service delivery set out in the EIF are generally synergic, and the suggestions mutually reinforce each other. The significance of the EIF is attested to by the direct references to it in EU policy and strategy documents, but it is less often cited in EU legal acts like directives and regulations. Furthermore, the EIF is found to be consistent with initiatives such as Single Digital Gateway Regulation, the digital single market strategy, and the data strategy. The voluntary nature of the framework is reflected in its uneven and ad hoc uptake at EU and Member State level. As a result, the availability of comparable data is limited, making comparisons between countries and sound cost-benefit analysis difficult. Several steps for improving the EIF can be derived from experience with implementation: ·clarify the recommendations with a better structure, clarity, granularity, and implementation guidance, reducing some overlaps between them;

·promote the framework across polices and Member States, encouraging best-practice sharing and awareness-raising activities; ·reinforce the EIF’s role as a strategic guide by: oenhancing EU interoperability governance and its fundamental role; oincreasing the references to it in related EU legal acts; ofostering coordination across EU and national polices through common interoperability rules; oreinforcing EU policy for interoperability with more binding requirements; ousing conditionality in EU funding instruments supporting digitalisation, enforcing interoperability by design and reuse of interoperability solutions; ·promote sound monitoring of progress in the field of interoperability and of the EIF’s effects. A future enhanced EU policy for public administration interoperability could address those issues by: ocreating binding interoperability rules; oestablishing a system of governance that supports a more practical and actionable EIF; oidentifying common reusable specifications and standards, tools, and solutions to support digital checks for new EU policies.



1. Introduction Digitalisation holds significant promise for improving public service delivery in Europe and generating benefits for public administrations, individuals, and businesses alike, such as time savings. Reforming internal processes, transforming the design and delivery of services, boosting data sharing and collaboration, adopting a user-centric approach, and adopting timely policies that keep up with technological changes are all part of the path to digital government. All these changes and digital transformation are highly dependent on the level of interoperability of the underlying IT systems and platforms supporting digital services and data exchanges. If services are digitised in silos, data cannot be reused, shared and exchanged. Interoperability can be defined as ‘the ability of organisations to interact towards mutually beneficial goals, involving the sharing of information and knowledge between these organisations, through the business processes they support, by means of the exchange of data between their ICT systems’.Legal, organisational, semantic and technical interoperability is crucial in the implementation of digital government policies. It enables the design of reusable specifications for data exchanges, or the agreement to use existing ones, and facilitates connections between IT systems. Both are key in the delivery of digital services. It goes beyond the design of IT systems, also covering broader organisational, legal and governance aspects. Interoperability is the foundation for the digitalisation of services, helping to put fundamental user-centric principles such as the once-only principle into practice. The once-only-principle is an e-government concept that aims to ensure that citizens, institutions, and companies only have to provide certain standard information to the authorities and administrations once. The once-only principle is part of the European Union’s plans to further develop the Digital Single Market by reducing the administrative burden on citizens and businesses. As emphasised in various EU-level initiatives in recent years, ensuring the interoperability of digital government has become a top priority, to maximise the potential of digital solutions for the single market. The European interoperability framework (EIF) adopted in 2017 as COM(2017) 134 (‘the EIF Communication’), was recognised as a key element in enhancing interoperable public services, especially across borders. It contains a set of principles, models and recommendations to guide public administrations across the EU and at all levels of administration in the design and provision of key digital and interoperable public services. Ultimately, it supports the free movement of goods, people, services and data throughout the EU. The EIF is a set of guidelines – principles – similar to the better regulation guidelines, aimed at guiding and enhancing interoperability of digital public services at all levels across borders in the EU and within countries (at national, regional, local level). It is meant to be used by administrations at every level from EU-wide to local, to design and deliver seamless, integrated digital public services to be used by other public administrations, individuals and businesses. Furthermore, the EIF Communication was accompanied by support for implementation of the framework from 2016 to 2020, drawing on the ISA² and the Connecting Europe Facility programmes. Through the ISA² Programme, the Commission created and promoted interoperability solutions that support implementation of the EIF by the Member States, as described in the ISA² final evaluation report. According to the 2017 Communication on the new EIF, its implementation needs to be evaluated to assess whether relevant recommendations should become part of a mandatory instrument. This need for a back-to-back evaluation and impact assessment was reaffirmed in a series of Communications in 2020, stating the need for a reinforced interoperability strategy for EU governments to foster coordination and the adoption of common standards for public services and data flows. Following the ‘evaluate first’ principle, this evaluation focuses on the new EIF introduced in 2017 and assesses its implementation, seeking to understand the achievements of the framework and the lessons that can be learnt to support the overarching goals of enhancing interoperability to support the digital transformation of the EU’s public sector.

The evaluation was based on five evaluation criteria stemming from the Commission’s better regulation requirements and nine evaluation questions in Annex III. A mix of data collection and desk research was used to gather both primary and secondary data (for further details on the methodology, see Annex II). In this, the Commission was assisted by a team of researchers led by the Centre for European Policy Studies (CEPS), which summarised its findings in an independent evaluation study. The collected data were validated via triangulation to ensure the robustness of the evidence. Data were collected from multiple sources, and several different tools were used to analyse the data according to the evaluation criteria. Data were also used from the EIF monitoring mechanism exercise that takes place annually as part of the National Interoperability Observatory (NIFO). The monitoring mechanism is meant to provide a macro-level view of how Member States are implementing the framework. More specifically, as the evaluation relies on both qualitative and quantitative data, a variety of analysis methods was used. Qualitative information was aggregated, compared and summarised to substantiate the evidence resulting from the evaluation criteria. To analyse the data collected during the consultation activities, descriptive statistics were applied. Finally, to complement the findings, other quantitative analyses, including a standard cost model, an econometric model and text mining, were carried out when evaluating the effectiveness, efficiency and coherence criteria. The mix of data sources and methods provided for solid conclusions. Nevertheless, several limitations may have affected the findings. These include the scarcity of data for assessing the costs and benefits of implementing the EIF at national level, the need for more time to be able to discern the longer-term results, and potential consultation fatigue among respondents. 2. WHAT IS THE EXPECTED OUTCOME OF THE INTERVENTION? 2.1Introduction to interoperability Interoperability is defined as ‘the ability of organisations to interact towards mutually beneficial goals, involving the sharing of information and knowledge between these organisations, through the business processes they support, by means of the exchange of data between their ICT systems.’ Legal, organisational, semantic and technical interoperability is crucial in the implementation of digital government policies. Interoperability is what makes digitalisation of government services efficient, enables the seamless and trusted exchange of data and connectivity of services across levels of administration, borders and sectors. It is a foundational element of all modern digitalisation strategies. Digitalisation of government services without interoperability is possible, however, and unfortunately still very much the norm. Governments can certainly be digitalises in a way that does not allow for data exchange across borders or even inside the same country. However, this means foregoing the benefits of learning from best practices, and of reusing existing tools and solutions for lower development and maintenance costs; the wheel must be reinvented again and again. For example, a well-designed online website for a national tax-return system may not be able to connect to the population registry, which means that taxpayers need to fill in the same data every year. And, possibly, if their signature is not valid in an electronic format, they will have to print out the declaration and go in person to the tax office to file it. And, if the tax database is not connected to a separate VAT database, businesses will need to enter the same information yet again, losing precious time and money. But there are also digitally transformed systems where interoperability is built in by default, as in Estonia. Individuals and businesses there spend an average of 3 minutes a year on their tax declaration. All the necessary information is already available, reused and exchanged between systems seamlessly thanks to the interoperability of data and systems. Another example are the national mobile contact tracing and warning applications, which were set up in record time and work seamlessly for citizens thanks to a European interoperability gateway. At its peak, 19 European countries out of 22 that deployed national contact tracing and warning applications were connected to this gateway and were able to exchange information. The EU Digital COVID Certificate is also an outstanding example. It is based on common interoperability data models to enable a common reading, interpretation and verification of the data inside. It enables the verification of data originating from different national/regional health systems; semantic alignment (first, second, third dose, type of vaccine, types of tests); as well as organisational and legal cooperation certainty (authorised entities, business rules related to the acceptance of certificates). The system has been a global success, as in addition to EU/EEA Member States, also many third countries from five continents have joined it. Interoperability enhances to the design of reusable specifications, the quality of seamless data exchanges, and facilitates connections between IT systems. All three are key in the delivery of digital services. Interoperability goes beyond the design of IT systems; it also covers broader organisational, legal and governance aspects. Interoperability is the foundation for the digitalisation of services, upholds fundamental digital rights and principles (like the once-only principle), and supports the design, interconnection, and delivery of digital services in a coordinated way so that data can flow across different domains and across borders. However, this requires strong interoperability foundations. Sound interoperability governance at both national and European level is a fundamental part of the legal, organisational and technical specifications towards those common goals. The ambitious Communication on a digital single market strategy, put forward by the Juncker Commission in 2015, the interoperability agenda needed to be revised and brought into line with the emerging challenges in the field of ICT and European public services. Among other relevant digital initiatives to interoperability, there is the Directive on reuse of Public Sector Information (subsequently revised and renamed as the Open Data Directive in 2019), the INSPIRE directive, the eIDAS Regulation , the EU eGovernment action plan 2016-2020 , the single digital gateway (proposed in 2017 and subsequently adopted as a Regulation in 2019), and the European Cloud initiative . 2.2Introduction to the EIF The EU has been supporting interoperability between public administrations since the 1990s through a series of funding programmes such as IDA, IDABC, ISA, ISA² , and now Interoperable Europe . As part of this support, the first EIF was produced in 2004 and then updated in 2010 and 2017 . With the introduction of the EIF in 2004, there was for the first time an agreement on a common vision and non-binding guidelines to support data exchanges and digitalisation in Public Administrations in Europe. Responding to the call for a revised framework for interoperability in the EU, a new EIF was adopted in 2017 as part of the Communication from the European Commission adopted on 23 March 2017. The new 2017 EIF expanded on the previous 2004 version adopted by the PEGSCO committee of the IDABC programme and the 2010 version adopted by the Commission. It brought more targeted recommendations, taking new technological and policy developments into account. The number of recommendations increased from 25 to 47 in the 2017 EIF and several recommendations were updated or newly developed to support relevant EU policies and initiatives. The EIF establishes a set of guidelines and principles, similar to the better regulation guidelines. It aims to guide and enhance the interoperability of digital public services at all levels across borders in the EU and within countries (at national, regional and local level). The framework has three goals: ·to inspire and guide European public administrations in designing and delivering seamless European public services to other public administrations, individuals and businesses in a way that is digital, cross-border by default (i.e. accessible to the public in the EU) and open by default; ·provide guidance to public administrations on the design and update of national interoperability frameworks (NIFs), or national policies, strategies and guidelines promoting interoperability;

·contribute to the establishment of the digital single market by fostering cross-border and cross-sectoral interoperability for the delivery of European public services. The EIF Communication was accompanied by an interoperability action plan to support the implementation of the EIF from 2016 to 2020, with a series of solutions supported by the ISA² and the Connecting Europe facility (CEF). Through the ISA² programme, the Commission created and promoted interoperability solutions that support implementation of the EIF by the Member States as described in the ISA² final evaluation report. The EIF focuses on cross-border interoperability between governments needing support to implement EU-wide policies. Given that digitalisation applies to all public administrations in all domains and at all levels, it is also applied directly in the countries for their own digital services and serves as inspiration for the creation of national interoperability frameworks (NIFs). Positive spillover effects were anticipated from providing Member States with common guidance to start working on interoperable services within their borders. Common interoperable governance structures, tools, specifications and services support seamless interactions across the various administrative levels in the country (central, regional, local levels), but also across some domains (justice, education, health and taxation). Most of all, the framework is a step towards developing interaction and cooperation required in a cross-border setting. Having a proper interoperability set-up within each Member State will help mitigate the complexities of coordinating interactions across borders that involve several domains and competent authorities. The EIF helps remove barriers to the exchange of data, connect digital services across the various administrative levels of the country and across borders and sectors in the EU. The framework is focused on supporting public organisations in working together on defining, designing and implementing digital services in a coordinated and efficient manner – starting by commonly designing and aligning back-office protocols, specifications, infrastructures, platforms and IT tools. However, it also covers fundamental issues for individual and corporate end users, such as accessibility, user-centric design and the once-only principle. At EU level, relevant stakeholders are the EU institutions and policy units responsible for formulating, designing and implementing EU policies with digital implications. This includes units in DG GROW, DG CONNECT and the Joint Research Centre (JRC). As an example of EU policies, the Single Digital Gateway Regulation led by DG GROW implements a pan-European technical system using the principles of the EIF and some solutions from the interoperability action plan, like eDelivery and common vocabularies for cross-border and cross-sector evidence exchanges. Member States’ digital government authorities and experts (government chief information officers, practitioners and academic experts) are longstanding close stakeholders, but the public bodies (ministries and agencies) mainly work on digitalisation and interoperability of digital services. In its factsheets on digital public administration and interoperability (Chapter 5), NIFO provides regular and up-to-date information on those authorities. The information is also available in a dedicated section of the NIFO website. Regional and local authorities have recently become more involved, particularly through the definition of an EIF-inspired framework for smart cities and local communities in the context of the Living-in.EU movement. The EIF, through the ISA² programme, has supported and encouraged the take-up of solutions by private companies involved in public sector development. Other important stakeholders are academia, and standard-setting, consulting and other private sector organisations (e.g. GovTech) that work closely with governments providing technical support. The revised version of the EIF, which is subject of this evaluation, was created in a collaborative way with public administration experts and the national authorities in the Member States that are involved in designing and coordinating implementation of digital service policies or strategies. The text was discussed and agreed upon in several meetings between Member States’ representatives for the ISA² programme, from national digitalisation departments, and policy departments (Directorates-General) responsible for single market digitalisation. In addition, academia, standardisation organisations and ICT companies gave their views in several workshops. Finally, a public consultation was held. Their needs were considered and are an integral part of the EIF adopted in 2017. 2.3Description of the intervention and its objectives To address the challenges identified, the EIF outlined several key objectives together with a set of inputs that comprise the main components of the EIF: the underlying interoperability principles of European public services, the layered interoperability model and the conceptual model for integrated public services provision, as well as the recommendations attached to each of the three components. The Communication on the EIF also included an interoperability action plan, detailing a set of key actions to be undertaken between 2017 and 2020 to support implementation of the EIF. As set out in the Communication, the Commission ensured the promotion, support and implementation of the Interoperability Action Plan mainly through the ISA² programme. At the time of its adoption, the framework was expected to produce a series of outcomes and impacts. The logical links between the needs and problems, objectives, inputs, and expected outcomes and impacts are summarised in Figure 1 (EIF intervention logic). The operational objectives and outputs are not included, as they were assessed during the parallel running evaluation of ISA². The needs, problems and drivers that the EIF intended to address were identified by analysing the 2017 EIF Communication, the accompanying Staff Working Documents, and the results of the consultation activities carried out in 2016 to inform the revision process. The EIF was established to address one key need experienced by public administrations in the EU: ·the need for more specific guidance for public administrations on how to improve the governance of their interoperability activities. The Staff Working Document accompanying the 2017 EIF Communication outlined two main problems in the field of interoperable digital public services: ·the fragmented delivery of digital public services in the EU and ·fragmentation in the organisation and format of public data in the EU. These problems were attributable to five main drivers: ·incoherent governance of action to implement interoperability; ·cross-organisational barriers;

·resource constraints in relation to interoperability; ·scarcity of tools and solutions for interoperability in the delivery of public services; and ·legal issues affecting interoperability. Source: Study supporting the evaluation of the implementation of the EIF . The objectives of the EIF were presented in a hierarchical order; furthermore, the achievement of specific objectives was seen as a pre-condition for attaining the general objective. General objective ·To help public administrations provide key interoperable, user-centric, digital public services to businesses and individuals, at EU, national, regional and local levels, thus supporting the free movement of goods, people, services and data throughout the EU. Specific objectives ·Specific objective 1: to inspire European public administrations at all levels in their efforts to design and deliver seamless European public services to other public administrations, individuals and businesses which, where possible, are digital by default (i.e. providing services and data preferably via digital channels), cross-border by default (i.e. accessible for all in the EU), open by default (i.e. enabling reuse, participation/access and transparency), and which follow the once-only principle. ·Specific objective 2: to provide guidance to public administrations on the design and update of NIFs, or national policies, strategies and guidelines promoting interoperability. ·Specific objective 3: to contribute to the establishment of the digital single market (DSM) by fostering cross-border and cross-sectoral interoperability for the delivery of European public services. ·Specific objective 4: to account for technological developments and trends in the development of interoperability guidance and tools. As mentioned, the intervention logic of the EIF does not cover operational objectives, as they are covered in a parallel evaluation study on ISA². What activities are supported by the 2017 EIF? The EIF provides 47 recommendations that are grouped under the three main components of the EIF (an overview of the interrelations between the components is provided in Figure 2): ·Underlying principles of European public services. The EIF puts forward 12 interoperability principles to guide the design of interoperable European public services. The principles include the following: othe principles of subsidiarity and proportionality (a case for EU action has to be made); oa set of core interoperability principles: openness, transparency, reusability, technological neutrality, and data portability;

oprinciples on user needs and expectations: user-centricity, inclusion and accessibility, security and privacy and multilingualism; and oprinciples for cooperation among public administrations: administrative simplification, preservation of information and assessment of effectiveness and efficiency. ·Layered interoperability model. The layered interoperability model includes the four layers of interoperability (organisational, semantic, legal, and technical interoperability) as well as a crosscutting layer relating to integrated public service governance, brought together into a comprehensive approach through a final and crucial component: interoperability governance. ·Conceptual model for interoperable public services. The concept is that European public services should be interoperable by design. The model consists of several basic components: coordination function, internal information sources and services, base registries, open data, catalogues, external information sources and services, and security and privacy protection. Source: COM(2017) 134 final, Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, European Interoperability Framework - Implementation Strategy, Annex 2. What was the expected outcome of the 2017 EIF?’ When created, the EIF was meant to deliver short-, medium- and long-term effects for certain categories of stakeholders. While the most immediate and operational results (the ‘outputs’) of the EIF were expected to emerge from relevant ISA² measures, it was thought that implementation of the EIF should in itself be able to generate results affecting European public administrations at all levels, as well as the public and businesses in the EU. These expected results are summarised below. Outcomes (expected) ·European public administrations at EU, national, regional and local levels use the guidance provided by the EIF to effectively deliver digital public services responding to the needs of people and businesses, and to establish cross-organisational relationships supporting cross-border and cross-sector cooperation. ·National policies, strategies and guidelines consider and promote interoperability. EU legislation and policies consider interoperability implications, increasing the interoperability potential of the national follow-up measures. ·A coherent cross-border and cross-sector interoperability environment is achieved, supporting the digital single market in the EU. ·New technological developments are considered when designing interoperability guidance and tools. Impacts (expected) ·Available interoperable, user-centric public services in the EU. ·Developing a European public services ecosystem in which: owners and designers of systems and public services become aware of interoperability requirements; public administrations are ready to collaborate with each other and with businesses and the public; information flows seamlessly across borders, supporting the single market, particularly the digital single market. 2.4Points of comparison This evaluation focuses on the achievements and areas for improvement of the 2017 EIF since its adoption. By 2016, there was a relatively good alignment between the Member States’ NIFs and the guidance provided by the 2010 EIF. The average EIF-NIF alignment level across the EU in 2016 was estimated at 75%. However, at the level of actual implementation, more work remained to be done. The average NIF implementation level across the Member States in 2016 was estimated at 56%. In the run-up to the revision of the EIF, a public consultation was carried out to gather the views of stakeholders on existing problems and the future of the EIF. Among the key problems identified by stakeholders were:

·lack of a comprehensive overview of existing interoperability initiatives; ·limited uptake of common standards in the development of solutions; ·lack of monitoring of implementation of interoperability initiatives; ·lack of both financial and human resources to support implementation of interoperability initiatives and solutions; and ·no assessment of costs and benefits of interoperability in the process of developing legislation at national level. In addition, the public consultation brought to light the main expectation of consulted stakeholders for the future EIF: time and cost savings. In the consultation, 49 stakeholders (74% of the respondents) said they expected time savings and 46 (70%) expected cost savings and greater transparency. 3. How has the situation evolved over the evaluation period? This chapter gives an overview of how the interoperability landscape has changed since the new EIF was adopted, both in terms of legislation and in terms of monitoring. This Chapter examines the policy context surrounding the adoption of the EIF before addressing the level of implementation. 3.1Policy context The policy context of the EIF has been very dynamic since the adoption of the 2017 version. In recent years, digitalisation of public administrations and services has risen up the political agenda at EU and national level, as demonstrated by the Resilience and Recovery Funds and the national plans: EU Member States are planning to invest more in the digitalisation of their public administrations. Back in 2017, the priorities for interoperability were reinforced through the Tallinn Declaration on eGovernment. The high-level Ministerial Declaration emphasised the role of interoperability as a strategic component for ensuring the effective delivery of public services in the EU. A similar emphasis on interoperability is seen in a European Parliament resolution of 2017. It underlines that the digitisation of public administrations should aim to ‘promote the better exercise of citizenship, improve the quality of life for citizens and the social and economic development of the regions, enhance citizens’ understanding of and involvement in public services’, while improving their efficiency and cost-effectiveness. The digitalisation of public services has been progressing relentlessly, gaining even more momentum due to the pandemic. T he digital economy and society (DESI ) index and eGovernment benchmark report show that all EU Member States have made progress in the area of digitalisation, but the overall picture across Member States is mixed, and despite some convergence, the gap between the EU’s frontrunners and those with the lowest DESI scores remains large. In 2019, the Council underlined in its conclusions on the future of a highly digitised Europe beyond 2020 that interoperability will continue to play a key role in the EU’s digital policy. The Von der Leyen Commission has set strong ambitions for digital transformation in the EU, as emphasised in the following Communications published in 2020 and 2021. ·Shaping Europe’s digital future, which sets out the Commission’s vision for the next five years on harnessing the potential of the digital transformation for the EU and its citizens, underpinned by European values. One of the key actions in the Communication is the development of a ‘reinforced EU governments interoperability strategy’ to foster coordination and the adoption of common standards for public services and data flows by 2021. ·A European strategy for data, which proposes nine European data spaces that rely on interoperability to enable seamless data exchange. ·The white Paper on artificial intelligence (AI) and the subsequent proposal for a regulation laying down harmonised rules on AI. ·A new industrial strategy for Europe for innovation and a greener and more digital industry, for which building EU leadership in key areas including data applications is essential. ·The SME strategy for a sustainable and digital Europe, which identifies the opportunities that digitalisation holds for small and medium-sized enterprises (SMEs) in the EU; wider access to data and the reusability of data can support the development of innovative SMEs. ·The long-term action plan for better implementation and enforcement of single market rules, focusing on better using digital tools to improve access to information, fostering cooperation between the Member States, and testing and applying advanced digital tools for public services. ·The 2030 Digital Compass: the European way for the digital decade, setting key priorities and targets for the digital transition in the EU over the next decade. When it comes to delivering digital public services, the Communication emphasises the need to ‘ensure interoperability across all levels of government and public services’.

In 2020, the Member States built on the Tallinn Declaration with the Berlin Declaration on digital society and value-based digital government. The Member States and the EU institutions committed themselves to developing a more vigorous and more interoperability-focused digital transformation policy for government and to supporting the digital transformation of public administrations. In achieving these ambitions, interoperability plays a key enabling role, ensuring a level of coordination, common processes and models for the achievement of further goals. The commitments included in these documents are also reflected in the budgetary allocations for the next several years. In particular, the new Multiannual Financial Framework includes a new programme to bolster the digital agenda in the EU called the Digital Europe Programme. With a planned budget of EUR 7.5 billion (at current prices) from 2021 to 2027, this new financial instrument will support five key areas of the digital transition: supercomputing, artificial intelligence, cybersecurity, advanced digital skills and European digital innovation hubs (EDIHs). The Digital Europe programme will ensure continued of funding for measures to increase interoperability in the EU. 3.2State of play To understand the current state of play in the implementation of the EIF, it is important to note that the EIF is a non-binding instrument providing guidance that can be followed voluntarily by the Member States without a precise timeline. Therefore, delays in the implementation of the EIF are difficult to identify, given the lack of a specific timeframe (for more details, see Annex VI, Supporting evidence from desk research). Most of the Member States (15) have created an NIF based on the EIF, namely Belgium, Bulgaria, Estonia, Greece, Spain, Croatia, Italy, Cyprus, Luxembourg, Malta, Netherlands, Austria, Romania, Slovenia and Sweden. The others have an NIF or similar strategic documents partially inspired by the EIF. Only three countries, Ireland, Lithuania and Finland, have not implemented an EIF-inspired NIF. Figure 3 reflects the extent to which the EIF has been taken up, at least conceptually, across the EU, based on data collected in January 2022. Note: The legend corresponds to the extent to which EU countries have adopted NIFs that are fully or partly based on the EIF, as follows: (0) Light blue: countries that have not adopted an EIF-inspired NIF; (1) Middle blue: countries that have adopted NIFs or similar strategic documents partly inspired by the EIF; and (2) Dark blue: countries that have adopted NIFs based on the EIF. For each country, the year of the first adoption is displayed. Source: Study supporting the evaluation of the implementation of the EIF . The EIF monitoring mechanism (NIFO) delivers a high-level assessment of the progress in implementing the three main components of the EIF. It examines how the components have been adopted as guidance principles in the national strategies and polices. Three scoreboards have been developed: the principles of interoperability, the interoperability layers, and the conceptual model for integrated public services provision. A set of 68 key performance indicators (KPIs) is used in compiling the scoreboards. The final output (see Annex VI.2) consists of a ‘traffic light’ assessment whereby the relevant KPIs are aggregated for each component of the scoreboard as follows: [1] for the first scoreboard, the aggregation is done for each of the 12 principles; [2] for the second scoreboard, the relevant KPIs are aggregated for each layer of the EIF interoperability model; and [3] for the third scoreboard, the aggregation of the KPIs is carried out for each component of the conceptual model for integrated public services provision. The ‘traffic light’ assessment produces a score value scale between 1 and 4, based on the aggregation of the KPIs mentioned above. The first results of the EIF monitoring mechanism are based on data collected in 2019. The results are presented in the 2020 digital public administration factsheets for each Member State. To gain an overview of the implementation of the EIF at EU level, the results of the monitoring mechanism exercise can be presented based on the progress in each scoreboard for all EU Member States (the full overview is presented in Annex VI.2). Overall averages per Member State per scoreboard were calculated, to provide a panoramic overview of those areas of the EIF in which most Member States score well and the areas where further efforts would be needed (see Figure 4, Figure 5 and Figure 6). Note: For a full description of the principles, see Annex 2 to the 2017 EIF Communication. Source: Study supporting the evaluation of the implementation of the EIF based on the data available on Joinup concerning the EIF monitoring mechanism and the underlying database provided by DIGIT.D2.





Source: Study conducted by Wavestone, 2022 Note: Countries are marked in grey where missing data for some sub-indicators make it impossible to calculate an overall average (see Annex VI.1 for a full overview). Source: Study supporting the evaluation of the implementation of the EIF based on the data available on Joinup concerning the EIF monitoring mechanism and the underlying database provided by DIGIT.D2. Note: Countries are marked in grey where missing data for some sub-indicators make it impossible to calculate an overall average (see Annex VI.1 for a full overview). Source: Study supporting the evaluation of the implementation of the EIF on Joinup concerning the EIF monitoring mechanism and the underlying database provided by DIGIT.D2.



Implementation of the 2017 EIF over time The pre-2017 EIF version was much more theoretical, with narrower scope. At that time, the monitoringmainly involved checking how well aligned the main principles in the national interoperability strategies were. Under the 2017 EIF, the level of implementation is also assessed. This is one of the reasons why it is not possible to make a direct comparison of progress in the Member States’ performance under the previous framework with their progress under the current one. Nevertheless, small and positive improvements are apparent from a high-level comparative analysis of results from the EIF monitoring mechanism for 2019 and 2020. Figure 4.b presents the improvements made by the Member States in implementing the recommendations linked to the principles of interoperability. Several countries have improved their implementation of the principles. Portugal has seen the biggest improvement. Implementation is assessed by gauging how well aligned NIFs and interoperability strategies are with the EIF and, if needed, tailored and extended to address the national context and needs. Along with Portugal, the number of countries reporting a high level of alignment increased from 14 in 2019 to 16 in 2020. With regards to interoperability layers, Greece and Poland have made most progress in implementing the recommendations, through increased efforts on governance. Results show significant improvements, particularly on holistic governance across all administrative levels and sectors, and the implementation of well-defined processes for selecting and adopting standards and specifications.

In the implementation of the recommendations related to the conceptual model, Sweden and Lithuania have made particular efforts to increase their level of alignment with the EIF conceptual model for integrated public services provision. They both increased their score by an average of 0.57 points, by putting in place catalogues of public services, open data and interoperability solutions. Several external factors can affect the overall performance and implementation of the EIF, either positively, supporting the overall goals of the framework, or negatively, hindering the achievement of expected results. The factors were selected based on findings from the relevant literature and analyses of key trends relevant for interoperability initiatives in the EU. The following external factors were found to have a potential positive impact on the implementation of the EIF: ·technological advances bringing new opportunities to improve and foster the delivery of public services, which can make the need for interoperability more salient and thus provide a further incentive to ensure that interoperability principles and models are taken into account; ·the development of national initiatives on interoperability in the public sector, which can show the commitment to making progress on interoperability; and ·the deployment of cross-sectoral initiatives such as smart city initiatives, showcasing the relevance of interoperability at the intersection of different sectors. Other external factors were found to have a potential negative impact and thus jeopardise implementation of the EIF. These factors were: ·technical challenges, such as legacy systems; ·limited resources, reflecting, for example, difficulties in accessing funding and attracting skilled workers in the field of interoperability; ·institutional complexity, referring to the complex system of roles distributed between the different levels of public administration and of complexity stemming from legal requirements; and ·changing political priorities, which may impact measures to improve interoperability in a given country. The impact of these factors on the performance of the EIF was tested through consultations with targeted stakeholders as described in Annex V. EIF supporting instruments The EIF is a theoretical framework, not a set of solutions. However, as a result of the associated interoperability action plan, a set of EU reusable IT solutions have been developed under the ISA² and CEF funding programmes to support the EIF implementation. Furthermore, the EIF toolbox published on Joinup provides a catalogue of reusable solutions linked to the main areas, principles and recommendations of the EIF. Moreover, any interested stakeholder can take them up as they are based on open source specifications. In fact, the EIF Communication was accompanied by action to support the implementation of the European interoperability framework over 2016-2020, through a series of solutions supported by the ISA² and CEF programmes. The Commission, through the ISA² programme, then created and promoted interoperability solutions that support implementation of the EIF by the Member States as described in the ISA² final evaluation report. Up to the end of 2020, the ISA² programme supported a total of 54 actions, grouped in nine packages, which were devised on a yearly basis through an annual rolling work programme. The nine packages are: 1.key and generic interoperability enablers; 2.semantic interoperability; 3.access to data / data sharing / open data; 4.geospatial solutions; 5.eProcurement / eInvoicing;

6.decision-making and legislation; 7.EU policies – supporting instruments; 8.supporting instruments for public administrations; and 9.accompanying measures. ISA² managed 39 actions in its first year of operation, 43 actions in 2017, 53 actions in 2018, and 54 actions under both the 2019 and 2020 rolling work programmes. In its digital policy hub , the National Interoperability Framework Observatory provides examples and case studies of national IT solutions and services implemented in public administration following the EIF principles. EU solutions that follow the EIF principles can be found in the EIF toolbox, which is now also starting to include national solutions. Annex VII gives a comprehensive overview of solutions supporting EIF implementation that are available to Member States, as part of the interoperability action plan described above. Most of them are also accessible through the EIF toolboxcreated to that end. 4. Evaluation findings (analysis) 4.1To what extent was the 2017 EIF successful and why? To assess whether the EIF has been successful, this chapter assesses how well it meets the following three criteria: i) effectiveness; ii) efficiency; and iii) coherence. The assessment draws on desk research, stakeholder and public consultations, and data collected from annual EIF monitoring across the 27 Member States. The EIF has certainly helped to inspire European public administrations at all levels to design and deliver digital public services, as well as providing guidance to public administrations on the design and updating of NIFs or digital strategies. Figure 3 in Chapter 3 shows that most Member States have drawn up a NIF or a similar national strategic or policy document that incorporates, at least in part, the EIF’s recommendations, principles, and models for delivering digital services. However, the EIF’s final achievement in helping to provide interoperable, user-centric public services in the EU and the development of a European public services ecosystem were limited, highlighting the need for more action and practical insights. EIF Monitoring mechanism The EIF monitoring mechanism, used for the annual monitoring exercise by the National Interoperability Framework Observatory , is one of the Commission’s most important data sources for tracking roll-out of the EIF across the Member States. It provides an overview of NIFs (see Figure 3 in Section 3), which currently present a positive picture of EIF implementation progress. However, there is an important caveat: the EIF monitoring mechanism focuses on the higher level of conceptual take-up of the guidance and commitment to the framework’s principles, recommendations, layers, and high-level components. The depth and breadth of EIF monitoring is limited by the cross-cutting nature of interoperability, deeply embedded in broader digitalisation efforts, and by the fact that public services are implemented and delivered by hundreds, or even thousands, of different authorities in some countries. Monitoring is mainly done by gathering data directly from Member States through an online questionnaire and transforming the replies into score values for the various KPIs. The methodology for this is established under the guidance of the JRC’s Composite Indicator Unit. For some few KPIs only, the data are collected directly from other EU observatories that target specific domains or areas overlapping with the EIF, such as DESI, the eGovernment Benchmark report, the open data portal, or the Accessibility Directive. Similarly, the overview of NIFs captures only the conceptual commitment, while stakeholders’ expectations are focused on the implementation level – implementing interoperability guidance at the service level. This is where a gap is apparent: between the conceptual level, where progress is generally positive, and practical implementation, where there is room for improvement in specific areas and where additional efforts are deemed necessary. Evaluation of EIF’s main components The EIF’s main components – the conceptual model, the layered interoperability model and the recommendations – are all useful, but are perceived as difficult to implement. As a result, more assistance appears to be needed for Member States in the form of guidance and systematic steps toward implementing the EIF’s recommendations. The risk is that the lack of granularity will allow for multiple interpretations.

There is undoubtedly a divide between acceptance of the EIF’s conceptual aspects and the need for more practical guidance in implementing interoperability requirements. There is also a need to raise awareness, particularly at local level. National public authorities are relatively confident in the EIF’s ability to provide public administrations with guidance on the design and updating of NIFs or strategies. However, they report that the framework’s voluntary nature and the scope of its recommendations restrict its ability to foster cross-border and cross-sector interoperability or take technological developments into account. This limits its contribution to the establishment of the digital single market. Two factors can explain the limited success in achieving the objectives. 1.The EIF’s objectives are extremely broad and the recommendations usually high-level. Further clarity and guidance would be required to ensure that they are implemented by Member States. 2.In some sectors, a lack of collaboration between the private and public sectors impedes the growth of synergies. To begin, the EIF’s principles are applicable to the development of interoperable digital public services in general. On the plus side, the principles of openness, technology neutrality, and data portability, as well as the reusability of IT solutions, transparency, and administrative simplification are deemed particularly critical, based on input from both targeted and open consultations, and are crucial for enhancing interoperable digital services. Indeed, the EIF monitoring mechanism’s findings suggest that progress has been made toward implementing several of the principles identified as beneficial by stakeholders consulted. Most Member States are particularly aligned with the EIF in areas such as principle 2: ‘Openness’, principle 8: ‘Security and privacy’ and principle 10: ‘Administrative simplicity’ (see Annex VI.2). However, several concepts are either too abstract (information preservation, effectiveness and efficiency assessment, and subsidiarity and proportionality), or they require additional clarification and practical focus to maximise their usefulness (transparency, technological neutrality and user-centricity). Further guidance is needed for a number of principles (including the assessment of effectiveness and efficiency, multilingualism, and subsidiarity and proportionality), as demonstrated by monitoring mechanism finding that better implementation is called for at Member State level. There are also principles from the EIF that require further development, such as multilingualism, information preservation, effectiveness and efficiency assessment, and subsidiarity and proportionality. This feedback is also corroborated by the EIF monitoring mechanism’s results, particularly in relation to principle 1: ‘Subsidiarity and proportionality’, principle 9: ‘Multilingualism’, principle 12: ‘Effectiveness and efficiency assessment’, and principle 7: ‘Inclusion and accessibility.’ Numerous countries have scores on the scale’s lower limit, (1 or 2), indicating the need for additional efforts to comply with the EIF’s principles and recommendations in their particular areas (see Annex VI.2). While layered interoperability and the conceptual model are deemed beneficial for enhancing interoperable digital public services and are adequately addressed by the EIF, additional guidance could make them more actionable, as confirmed by both consulted stakeholders and EIF monitoring mechanism findings. Layered interoperability could be further enhanced by applying the recommendations on the governance, organisational, and legal levels of interoperability. The evaluation study demonstrates that greater emphasis should be placed on the non-technical aspects of interoperability, acknowledging that interoperability is a multifaceted notion, not limited to technical difficulties. Certain features of the semantic layer could also be addressed, such as the low rate of semantic adoption due to a lack of awareness or knowledge about the services end consumers. Moreover, it emerged from the evaluation study that some stakeholders judged the organisational and legal layers to be incomplete and under-implemented. The organisational layer, in particular, requires clarification because it is believed to be too basic for practical applications in its current state. Finally, the layered model could be improved by elaborating on the interactions between the technological, organisational, and legal layers. The monitoring mechanism’s findings corroborate these observations. The scorecard for the EIF’s layered model of interoperability demonstrates that Member States have made significant efforts to implement the EIF and its recommendations in this area, most notably technological interoperability at EU level. However, the scoreboard identifies a need to improve implementation of suggestions relating to the governance levels of interoperability, organisational interoperability, and legal interoperability. Stakeholders contacted about the conceptual model confirm that it has helped to enhance interoperable digital public services to some extent. However, the conceptual paradigm is regarded as overly abstract and inoperative. As a result, Member States may interpret and hence implement it inconsistently, jeopardising the goal of interoperability. As such, experts and representatives from public administrations call for the establishment of an end-to-end operational approach to assist Member States in implementing interoperability by design in order to develop a comprehensive interoperability approach. As regards the EIF’s coherence, the EIF’s principles, layered interoperability model, and conceptual model for integrated public service delivery are generally synergic. To improve the framework’s overall cohesiveness, the conceptual model might be more closely related to the other two components. According to the research, the strength of the national frameworks is contingent on an in-depth understanding and design of these three EIF components. The components’ interdependence is exemplified by the connection between the semantic layer, the ideals of openness and transparency, and the section of the conceptual model on openness. The principles and layered interoperability model are extremely synergistic, while enhancements to the conceptual model may result in more synergies. The recommendations made throughout the EIF are largely self-reinforcing. Nonetheless, there are several instances where EIF guidelines overlap, at least thematically. While the overlaps may not cause problems, by better organising the recommendations, the important messages could be clarified, making the advice more actionable. On a more detailed level, the EIF monitoring mechanism’s findings suggest that various enhancements might be made to further support the conceptual model’s implementation in some Member States, notably, Germany, Ireland and Romania (see Figure 6 in Chapter 3). Almost all countries do admirably in the field of ‘internal information sources and services’ (with scores of 4 almost everywhere), but efforts in some countries like Germany, Ireland and Poland would need to be stepped up (as signalled by scores of 1; see Annex VI.2). Additionally, progress on open data is encouraging, with all countries rating towards the top of the scale (scores of 3 and 4). Improvements could be made to cataloguing, external information sources and services, as well as security and privacy. External drivers External drivers such as technological advancements and national and cross-sectoral interoperability initiatives have contributed to some extent to the EIF’s observed achievements. The main constraining factors, to be minimised, are seen as limited resources and institutional complexity. According to respondents in targeted consultations, external drivers do, on average, contribute to the framework’s implementation (see Figure 7).

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: Averages do not include respondents answering ‘don’t know/no opinion’ (DK/NO). Source: Study supporting the evaluation of the implementation of the EIF However, the low average score assigned to cross-sectoral projects by national public agencies is particularly noteworthy. This score indicates primarily a lack of coordination between the public administrations of the various Member States. The COVID-19 pandemic has highlighted the critical nature of interoperable solutions. Nonetheless, several expert studies determined that cross-sectoral initiatives can play a significant catalytic role in advancing interoperability adoption. In this context, smart city efforts are instructive. There is significant opportunity to enhance such projects by ensuring that interoperability is a topic of discussion and, more significantly, a requirement, thereby reinforcing the EIF’s fundamental messages. There are considerable opportunities to achieve economies of scale for smart cities by reusing tools, data, and applications. Some unfavourable external influences have a similar impact on the EIF according to the respondents in the EIF evaluation consultations (see Figure 8). These are described below. 1.Scarce resources. budgetary limits, organisational difficulties, and a skills shortage. These obstacles stem mostly from a lack of human and financial resources, but also from the incompatibility of national databases. 2.Institutional complexity. This is undoubtedly the most challenging factor to solve, as it may be necessary to adjust the legislative framework in order to execute the EIF successfully, and institutional changes are slow to materialise. A new analysis on the influence of open source software on technological independence, competitiveness, and innovation in the EU economy identifies a fragmentation of duties in the field of digital policy, impeding collaboration on digital law-making. This issue has a range of consequences for different countries, as some countries’ legal systems are more amenable to reform than others. Interoperability and institutional complexity are inextricably linked; this issue can be partially resolved by enhancing interoperability. 3.Limited awareness is another barrier that is deemed to jeopardise the EIF’s take-up, particularly at regional and municipal level and in the design of digital public services. Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. Note: Averages do not include respondents answering ‘don’t know/no opinion’ (DK/NO). Source: Study supporting the evaluation of the implementation of the EIF . EIF beneficial effects The framework has had some beneficial effects. The EIF has to a certain extent contributed to the improvement of the quality of public administration services and to the promotion of free movement of products, services, capital, and labour across Member States. A strong alignment between the NIF and the EIF would surely boost the direct and indirect benefits for stakeholders (i.e. residents, enterprises, and public administrations) through better digital public services, particularly at cross-border level. As explained above, some Member States have created an NIF based on the EIF, such as Belgium, Estonia, Greece, Spain, Cyprus, Luxembourg, Malta, Netherlands, Austria, Portugal, Romania and Slovenia. Spain for example has passed an EIF-based NIF into law through a Royal Decree. The purpose is to ensure that the systems and applications used by public administrations are technically, semantically and organisationally interoperable enough to enable rights to be exercised and duties fulfilled through electronic access to public services, while also enhancing effectiveness and efficiency. Thanks to the NIF, sound governance has been stablished across various administrative levels, and a set of supporting common services (eID, eSignature, eDelivery, eInvoicing…), infrastructures and technical specificationshave been devised for common and more efficient digital service provisioning. Thanks to this, public services were not disrupted during Spain’s first COVID-19 lock-down (the most severe in Europe along with Italy’s). The Spanish Chief Information Officer (CIO) highlighted common infrastructure and services as being among the key elements for the smooth delivery of public services during the first wave of the COVID-19 crisis. See below further details. In addition, thanks to Spain’s EIF-based NIF, data on taxation, social security, justice etc. data can be exchanged through a common interoperable platform, used by all digital services in the country on a daily basis, reducing the administrative burden on people and businesses.

Likewise, on the 1 March 2019, Luxembourg’s Government Council adopted an NIF which guides public sector bodies within its remit on how to attain a higher level of interoperability. This was Luxembourg’s first NIF bringing together in the same text all the elements needed for a coherent and clearly structured base for public sector interoperability. The NIF builds on the 2017 EIF whilst taking account of context and specific national needs. The NIF thus complies with the requirements set out in the Commission’s EIF and enables the Luxembourg government to make progress towards greater interoperability. The Slovenian framework provides for a website for publishing public sector interoperability solutions and products. It connects a catalogue of interoperability solutions with best practices for reusing its content. Some NIFs set up the necessary governance to work together in a coordinated matter on interoperability-related topics supporting common goals and priorities laid down by the national digitisation policies, as well as relevant EU legislation like eIDAS, the Single Digital Gateway Regulation and the Open Data Directive. Such governance arrangements typically bring on board all ministries at central level, the regions and even the local government in the design and implementation of common processes, IT systems and technical specifications. The report on public administrations’ digital responses to COVID-19 in Europe shows that five countries (Czechia, Denmark, Estonia, Spain and the Netherlands) reported no disruption to their public services. This appears to be in line with what was outlined in the 2020 edition of the European Commission eGovernment benchmark as four of these five countries are considered to have highly digitalised public services, scoring well above the European average. The delivery of public services in four of these five countries is also considered to be highly interoperable with scores above the European average on the EIF. This is also further corroborated by the Danish, Estonian Spanish and Dutch national CIOs, who stated that the already high level of digitalisation in public services delivery helped to ensure the continuity of these services throughout the first wave of the COVID-19 crisis. An example of an EU solution which puts these generic interoperability guidelines into practice is the EU interoperability gateway for COVID-19 contact-tracing and vaccines. A national/bi-national solution is Estonia and Finland’s X-Road platform , linking all public digital services. Estonian experience demonstrates that the EIF aided in the development of a wide variety of digital and cross-border public services relating to company and property registration (for more details, see also the above section covering the impacts of external factors). Additionally, the EIF contributes to the free movement of commodities, services, capital, and labour throughout Member States. Experts and academia, as well as businesses and business organisations, are generally more optimistic about the framework’s potential benefits for research, development, and innovation. With regard to implementation of the EIF, IMAPS is an online survey developed under the ISA² programme, helping public service owners evaluate, consider and improve all key interoperability aspects of their digital public service (legal, semantic, organisational, or technical), in compliance with the EIF . Some public bodies have used the tool on a voluntary basis to make their administrative procedures fit for purpose for the single digital gateway, assessing the level of digital maturity of a public service that needs to be provided online by 2023 and accessible via the gateway. Examples include online submission of corporate tax declarations or applications for a European health insurance card (EHIC). Some countries (Belgium, Czechia and Slovenia) and the city of Valencia have used the survey to detect areas for improvement and to make digital services more mature from the EIF perspective. IMAPS publishes an annual report on the interoperability maturity of European digital public services assessed using IMAPS. The latest IMAPS report (2020) reveals that all assessed digital public services are on average below the essential interoperability level (IMAPS maturity level 3). However, this EIF evaluation shows the limitations on providing a deeper and wider analysis of the level of EIF implementation across the hundreds and thousands of bodies in the 27 Member States. To do systematically, sound interoperability governance would be needed, involving the main national authorities dealing with digitalisation, in close coordination with the regions, local authorities and the European Commission. A common process for apply IMAPS would be beneficial, or a similar tool aligned with main interoperability and digitalisation goals stemming from the national and |EU strategies and policies. The aim would be to ensure that any interoperability maturity level assessment helped to monitor, rationalise and steer all implementation plans and investments, and find gaps, overlaps, inefficiencies and ways to bridge them. The EIF can serve as a genuine catalyst for public-government dialogue. It is possible to quantify both the impact of adopting an EIF-based NIF and the maturity level of digital, online public services to gain additional insight into the range of possible EIF impacts and improvements in digital public services. An economic analysis demonstrates that an EIF-based NIF, or an NIF with EIF-based components, is more conducive to public-government interaction than no EIF-based NIF. Additionally, a 10% improvement in the quality of online public services would encourage an additional 4.3 million to 5 million EU residents to connect with their governments via the internet each year. Figure 9 and Figure 10 respectively report the results obtained under two different regressions: 1.a ‘complete’ EIF-based NIF compared to an ‘incomplete’ EIF-based NIF or no EIF-based NIF; and 2.a ‘complete’ or ‘incomplete’ EIF-based NIF compared no EIF-based NIF. Note: The results (i.e. when NIF ‘complete’=1 and NIF ‘incomplete’=0) compare countries having a fully EIF-based NIF with countries having a partially EIF-based NIF or no EIF-based NIF at all. The results assume linearity over time, i.e. the average effect of having an NIF on online interaction with public administrations is assumed to be constant over time. Source: Study supporting the evaluation of the implementation of the EIF

Note: The results (i.e. when NIF ‘complete’=1 and NIF ‘incomplete’=1) compare countries having a fully or partially EIF-based NIF with countries having no EIF-based NIF or similar strategies. The results assume linearity over time, i.e. the average effect of having an NIF on online interaction is assumed to be constant over time. Source: Study supporting the evaluation of the implementation of the EIF . This analysis is conducted at the macroeconomic level and is based on binary variables indicating whether or not the countries analysed have approved an NIF or comparable national strategic documents based entirely or partially on the EIF. As a result, the analysis does not account for the effects of the practical implementation of specific interoperability standards, but rather for the effects of strategic advice in general. The analysis determines the average impact throughout the sample examined. The magnitude of the effects may vary by country. Additionally, the findings confirm the association between increased service quality and the fact that improved eGovernment services enable people to interact more with their government via the internet. More precisely, a 10 % increase in the eGovernment index results in an increase of 1.5-1.7 % in people’s use of the internet to connect with public authorities while all other variables remain constant. On the efficiency front, the data indicate that benefits may outweigh costs in some cases. While public administrations bear the costs of implementing the EIF, streamlining procedures and facilitating data exchange are seen as benefiting a broader range of stakeholders by improving interactions (i.e. saving time and money) between public administrations on the one hand and between public administrations and users of public services on the other. Most of the EIF’s benefits stem from two factors. 1.Affirming the importance of administrative simplicity. By streamlining and simplifying the digital delivery of public services, EIF implementation costs were negligible in comparison to the cost savings associated with interactions between public administrations, and between administrations and their constituents. 2.Encouraging data reuse. The EIF’s goal is to assist public administrations in avoiding duplication of efforts and effectively leveraging existing resources and information, resulting in better service quality. Concerning expenditure by public administrations in implementing the EIF, national and EU public administrations gave consistent responses. Supporting the framework’s roll-out is regarded to be reasonably inexpensive, at both national and EU level. In the Netherlands, for example, an independent and external advisory body (ACTAL, the Dutch advisory board on regulatory burden) has been set up to assist the government and parliament on minimising regulatory burdens. Additionally, short-term advantages must be weighed against long-term benefits. While initial investments can be costly, as procedures are optimised, operating costs tend to decrease over time. The deployment of NIFs inspired by the EIF clearly also resulted in various benefits. 1.The EIF facilitates synergies in the implementation and design of new services, contributing to the reuse of a diverse range of software programmes collected in the Italian reuse catalogue, for example. 2.Interoperability has resulted in cost savings associated with various ICT projects. Centralised authorisation of ICT projects is estimated to have saved Czechia more than EUR 39 billion in public investment in 2019. Additionally, some ICT projects filed by public agencies were rejected due to their failure to meet interoperability criteria. Notably, public administrations were able to rethink and adapt their project ideas, resulting in further savings, by depending on the sharing and reuse of existing digital solutions and shared government services. While voluntary adoption of the EIF and its recommendations has undoubtedly resulted in benefits recognised by public administrations, more should be done to foster an EU-wide strategy that is really integrated. However, a comprehensive evaluation is impossible to make due to the framework’s broad scope and non-binding character. Cost-benefit analysis In measuring and comparing the expense of adopting the EIF and applying these costs to NIFs, many difficulties arise due to the voluntary nature of the framework. These difficulties are due to the varying cost of digitalisation across different public sector bodies and the different approaches taken by countries in implementing the EIF and the varying degrees of digitalisation. Comparisons between Member States are complicated, as costs may vary in terms of coverage and over time, even when cost differences are clear (see Box 1). Moreover, interoperability is an underlying support function for digital transformation; for this reason, many of the assessments conducted by Member States are based on digital government transformation costs. Defining specific interoperability costs requires complex research and analysis. The consultation with the stakeholders nevertheless provided some reference points for the costs linked to producing a strategy or an NIF document and bound mainly to consultation services. The complexity of EIF implementation make it hard for Member States to estimate the associated costs. Many different authorities and public services are involved and it is difficult to isolate EIF implementation from the overall digital transformation processes. This is also the case for the implementation of EU policies. The team working on the single digital gateway conducted a studyto estimate the cost of implementing the once-only principle where interoperability plays a major role. Its conclusions, summarised below, show that the costs were hard to quantify. 1.The scarcity and low comparability of data makes it difficult to assess the costs and benefits of incorporating the EIF into national frameworks. The voluntary nature of the framework is reflected in its uneven and ad hoc uptake across the EU, limiting the availability of data and making comparison between countries difficult. Furthermore, implementation of the EIF has not followed the same timeline in the different countries. To overcome this limitation, the study team contacted the national public authorities that responded to the consultation activities and indicated their availability to participate in a follow-up interview, to gain insight on the time spent (in terms of person-days) on the incorporating the EIF, or part of it, into their national framework.

2.The overarching scope of the EIF hinders the assessment of the direct costs and benefits stemming from its implementation. Costs and benefits deriving from the EIF do not relate exclusively to implementation of the framework; costs and benefits also depend Member States’ levels of digitalisation, the degree of centralisation and varying organisational structures involved in delivering public services. The costs related to interoperability initiatives are borne at different levels of administration in different countries. The cost discrepancies are due to the following factors. ·The extent to which information technology has been digitised. Member States that have already deployed interoperable solutions in line with the EIF can quickly build on pre-existing solutions, resulting in reduced costs and responsibilities. Additionally, governments that rely on open source solutions mitigate the risk of lock-in. As a result, solutions can be more easily transferred or altered, resulting in cost savings. ·Catch-up effects. Because public services have differing degrees of centralisation and organisational structures, the costs of interoperability projects are borne at varying levels of administration in different countries, as stated in the expert assessments accompanying this study. In Sweden, for example, costs are borne primarily by individual agencies. In comparison, the role of national government initiatives and support is substantially stronger in neighbouring Norway and Denmark. It is often difficult to assess the cost of ICT projects since they can contain significant hidden costs such as the involvement of existing workers in modifications required for the new operation design, such as training, or time spent on transitional operations. These constraints underline the need to identify common criteria so that Member States’ costs can be assessed consistently. Furthermore, incorporating EIF guidance into the national framework is likely to have delivered different benefits in the different Member States (see Box 2). Several representatives of public authorities in the Member States provided additional feedback on the number of person-days required to take up the EIF in the development of national frameworks or strategies for interoperability. The number of person-days is translated into a costing based on average labour costs in all sectors (based on NACE rev.2 activity) in the respective countries. In line with the better regulation toolbox, an extra 25% is added to labour costs to account for overheads (e.g. rent costs, utilities, insurance, office equipment and supplies, and travel costs).. On that basis, time and costs can amount to 150-205 person-days (equivalent to approximately EUR 24 000 –EUR 30 000) for creating updated drafts of existing national documents and strategies based on the EIF, and 550 person-days (equivalent to approximately EUR 169 000) for designing EIF-based national documents and strategies from scratch. Costs also differ according to how fully the EIF has already been implemented. Countries that have not implemented the EIF before may incur more up-front costs to ‘catch up’ with the other Member States. Limited interoperability impedes the digitalisation of the public sector and makes it difficult to achieve benefits such as time and cost savings. Addressing this problem and enabling digital transformation in public services could save result in significant savings. Several studies have carried out cost-savings analyses in different countries or geographical areas following the implementation of different solutions. Table 1 reports the findings on estimated cost savings collected during the desk research. Geographical coverage Implemented solutions Cost savings Belgium Digital government solutions In 2016: 1.EUR 4.6 million for businesses 2.EUR 28 million for the public UK Digital strategies and implementation of key principles: digital-by-default public services GBP 1.7 billion to GBP 1.8 billion every year Netherlands and Belgium Digital strategies and implementation of key principles: once-only principle In 2017: EUR 163 million EUR 100 million a year Spain Digital strategies and implementation of key principles: e-administrations, once-only principle, and interoperability platforms EUR 22 billion between 2008 and 2011, with: 3.60% linked to e-administration 4.20% to the implementation of the once-only principle and 5.20% to interoperability platforms EU Digital strategies and implementation of key principles: once-only principle and digital-by-default public services In 2014: 1.EUR 5 billion a year linked to the once-only-principle; and 2.EUR 10 billion a year linked to the digital-by-default principle Source: Study supporting the evaluation of the implementation of the EIF based on official reports. The expense of translating the EIF, or parts of it, into national frameworks is difficult to assess and ranges from EUR 24 000 to EUR 169 000 (based on the estimates of time translated into costs provided by representatives of several public authorities in the Member States). The impacts of a public sector interoperability policy are more difficult to identify and quantify than those of many other EU initiatives. A report by the JRCanalysed the economic impact of public sector interoperability overall. It found that, due to its very nature as an enabling factor across many policy fields, the impact of interoperability is hard to isolate from other aspects of a policy. Interoperability in its many forms is an important enabler but it is interoperable policies and public services that deliver benefits (and consume resources). Nevertheless, the evidence is quite conclusive. For instance, in terms of economic impact, the study by the JRC found that, at a conservative estimate, the public could save up to 24 million hours a year through improved interoperability. This would represent monetary savings in the order of EUR 543 million annually. For businesses, improved interoperability would lead to time savings of 30 billion hours. In monetary terms, this translates into savings of EUR 568 billion annually. Improvements in public sector performance due to interoperability could lead to an increase of 0.4% in GDP. Overall, the potential positive benefits derived from increased interoperability for the public sector in the EU are considerable. For instance, converting these estimates into monetary terms, a 1 % improvement in the indicators used, through greater interoperability, would boost EU GDP by EUR 21 billion - EUR 56 billion. Future monitoring should focus on reusing developed technologies and providing better estimates of costs. In this regard, a set of EIF-specific criteria could make it easier to compare costs across the EU (including, for instance, the daily cost of work by the team tasked with implementing the EIF at national level, the cost of partial implementation of the EIF, the number of consultations between the European Commission and the Member States when the EIF was agreed on and introduced, the time spent by annual reporting under the NIFO initiative, etc.). The present evaluation study also shows the current limits to proper correlational analysis between digitalisation and interoperability, and the difficulties in decoupling the two effects in the estimates. Future work is needed to come up with sound methodologies for estimating the cost of interoperability, as opposed to the costs of digital transformation in a broader setting. Coherence The EIF is found to be consistent with initiatives such as the Single Digital Gateway Regulation, the digital single market strategy, and the data strategy. Both strategies highlight the EIF’s role in digitalising EU public services. In addition, the 2017 EIF and measures to monitor its implementation reflected measures outlined in the eGovernment action plan 2016-2020. Moreover, the 2015 DSM strategycalled for revision of the EIF to better complement the strategy’s goals of digitalising public services in the EU. The 2017 EIF heeded this call. The interoperability action plan included key steps to be taken to further support the vision of digital and interoperable public services. The ISA² final evaluation study highlights the programme’s contributions (as the main EIF implementation instrument). Moreover, the adoption of a revised EIF Communication in 2017 helped with the successful completion of two actions in the e-government action plan for 2016-2020, namely actions 4 and 6. It was developed as part of the NIFO initiative, which is funded by the ISA² programme to monitor the framework’s implementation in the Member States. The NIFO action also included an EIF toolbox . The toolbox maps possible interoperability solutions produced as part of the ISA² and CEF programmes, which public administrations can use to implement EIF recommendations. The toolbox also links the EIF’s suggestions to their accompanying solutions. Many of the EIF principles or 47 recommendations have been targeted by other provisions (examples given already include open data, SDG for once-only, privacy and data protection (GDPR), accessibility, etc.) This can be seen from a first mapping of digital-relevant EU initiatives onto the EIF pillars and individual recommendations, part of the EIF toolbox resources (see Annex VIII). However, some parts of the EIF are not yet specifically targeted by the EU regulator, such as interoperability governance or reusability. As examples of coherence at EU level, the SDG Regulation mentions a number of ideas and fundamental concepts promoted by the EIF, such as the potential for reusing an ISA² solution and the core public services vocabulary application profile (CPSV-AP). The Regulation itself implements a pan-European technical system using EIF principles and some solutions from the interoperability action plan, such as e-delivery and common vocabularies.

To implement the once-only principle, Article 14 of the Regulation obliges Member States to exchange evidence for 21 administrative procedures listed in Annex II to the Regulation. The EIF is fundamental to this endeavour. Implementation of harmonised common data models has already started for eight evidence types, to enable data to be shared according to semantic interoperability principles and fully in line with EIF guidance (semantic layer principles). These evidence types include the birth certificate, the absence of criminal records and academic qualifications. At Member State level, this involves firstly the central state digitalisation units. The last mile is taken care of by regional and local governments who host digital services in most cases and are strongly impacted by EU policies. Solutions for eDelivery and for interconnecting authoritative sources of data on people and businesses are also designed in line with principles and solutions stemming from the EIF conceptual model for integrated digital public service delivery. The data.europa.eu portal has also been set-up using the EIF as inspiration, reusing a data model created under the ISA² programme, to catalogue data and link up the open data sets from EU institutions, agencies and bodies, as well as European countries national and regional catalogues in an interoperable manner, according to the principles of the EIF conceptual model Data portability is one of the main driving forces of the EIF. As another example, Article 28 of the data act draft proposal provides for common specifications and core vocabularies to underpin the interoperability of cross-sector and cross-border data spaces, fully in line with EIF principles and recommendations, to ensure interoperable data sharing services and flows. To ensure strong interoperability governance, the draft assigns a major role to the European Data Innovation Board, at least for public sector services. Where there are different approaches between sectors and even Member States on privacy, data protection, responsibilities, common use cases, data flows and data specifications, this can result in a lack of legal certainty for the sharing of data across domains, policy areas and borders. The data act proposal is meant to address this lack of legal certainty, in conjunction with EU data policy and data governance. In this regard, it is aligned with the legal interoperability concept promoted in the EIF, ensuring that organisations operating under different legal frameworks, policies and strategies can work together. Health is another practical example. Patients typically have medical records that are hosted in a hospital or medical centre as a digital record. If the person moves to another location or country, transferring the file to another hospital digitally can be problematic since there are no, or few, common formats or solutions widely used in Europe. The solution is to agree on the pieces of information for the record, and then on common formats for the records. This will be part of the European Health Data Space. It has to be done in a way that not only health service providers, such as hospitals, but also other data users, for instance research institutions conducting medical research, can easily interpret the data and combine it with other data sources, such as mobility, socio-economic or environmental. This is one of the reasons why data spaces should be interoperable across themselves, and minimum technical requirements are needed. Cooperation and ongoing work in the area of digital health and health data use is applying EIF principles and actively seeking interoperability support. ISA² solutions also help to implement the EIF and more broadly interoperability initiatives at EU and Member State level. Other than those mentioned, the EIF aligns with Horizon 2020, the INSPIRE Directive, and the smart cities/communities projects. Moreover, the EIF’s overlaps and inconsistencies with other projects are restricted. More direct references to the EIF could help reinforce the idea that such an EU-level framework has broad consequences, and that interoperability is vital for projects wanting to leverage the benefits of digital transformation. An examination of rules, directives, decisions, and Commission communications from 2004 to 2021 demonstrates that the EIF’s interconnections with other EU programmes might be strengthened. We discovered that: 1.the EIF has been considered in the preparation of some legislation in areas where interoperability is vital; 2.the EIF is mentioned in relevant communications, with a spike in 2020, indicating that public sector interoperability is becoming a more important policy issue. Most references are found after the 2010 EIF was adopted, although a few more references have been identified since the 2017 EIF was published. Figure 11 lists the EIF and related keywords in regulations, directives, and decisions from 2004 to April 2021. Source: Study supporting the evaluation of the implementation of the EIF based on the EurLex dataset and the EurLex website Finally, more could be done to strengthen the position of the EIF as a guiding, strategic document. It is important to ensure that alignment with the EIF is not limited only to the declarative level, but that it is fully taken on board in implementing new policies. A future interoperability strategy and policy could definitely contribute in this regard. 4.2How did EU intervention make a difference? National or sub-national initiatives would make a very limited contribution to the EIF’s aims. Consulted public authorities saw only limited potential for national and sub-national efforts to contribute to the EIF’s existing aims, with substantial reliance on national expertise and improvements in digital public services. Indeed, there are countries at both ends of the spectrum: those that are more advanced and experienced and those that are less advanced and experienced. This heterogeneity may have a detrimental effect on interoperability at EU level. Heterogeneity of solutions and methods may impede EU-wide interoperability. More crucially, the cross-border dimension cannot be addressed sole through the efforts of national or sub-national governments. Even if national, regional, and local public administrations strive to improve their approach to digital and interoperable public services, without coordination, the strategies and solutions selected may impede cross-border interoperability. Thus, the EU can serve both as a coordinator, ensuring a consistent approach to interoperability throughout the EU, and as a stimulant, offering direction and support. The Tallinn Declaration of 2017 and the Berlin Declaration of December 2020both emphasised the importance of a unified strategy to interoperability. Additional evidence from desk research corroborates these conclusions. The list of Member States that have adopted an NIF or comparable strategic documents (Figure 3 and Annex VI.2) demonstrates that several countries have incorporated the EIF into their national frameworks, and in some cases, the EIF served as the starting point for such papers (as it is the case, for instance, for Croatia).

Benefits at EU level By using economies of scale (via shared technologies) and economies of learning, the EIF is almost certain to achieve its objectives at a lesser cost than equivalent national or sub-national projects (dialogue and shared solutions). The EIF decreases costs through economies of scale (via common tools) and economies of learning, as confirmed by responses to the evaluation consultations (dialogue and shared solutions). Initiatives at the national and/or local level would be substantially more expensive. Additionally, the EIF generates positive externalities as a result of stakeholder collaboration. Notably, the ISA² programme has aided in the development and sharing of tools and solutions by serving as one of the primary instruments for executing the EIF, namely the interoperability action plan. A uniform EU strategy eliminates the costs associated with each Member State assessing and devising actions to address challenges on an individual basis. Additionally, a lack of coordination is likely to result in divergent paths being taken, resulting in increased fragmentation rather than a shared vision. On average, the EIF has aided in the advancement of shared EU policy. However, more could have been accomplished. The EIF is underutilised across sectors and by Commission departments. Cooperation between other Commission departments should be strengthened to enable the EIF to produce stronger outcomes. A future interoperability governance, which is now being evaluated as part of the back-to-back study of a future interoperability policy, may help to close this gap. Although the EIF has given value to the EU, cross-border interoperability remains limited and driven by sectoral needs. The benefits of cross-border interoperability, as promoted by the EIF, have not been shared equitably throughout Member States. Without a doubt, the framework might make a greater contribution to the development of a European interoperability environment. Identifying and addressing significant cross-border demands would very certainly result in increased EIF take-up and consequently increased EU added value. The EIF’s primary contribution to cross-border interoperability is in raising awareness of the need for interoperability and in encouraging countries to implement NIFs based on a common model. The EIF contributes to cross-border interoperability in three ways. 1.It has increased awareness of the multi-tiered approach to interoperability that goes beyond technical considerations. The EIF establishes a consistent vocabulary and conceptualisation of interoperability (including the interoperability layers and conceptual model), which can serve as a common starting point for future projects in the field. 2.It encourages Member States to implement NIFs based on a common model, fostering an EU-wide approach to interoperability. The EIF promotes conversation and cooperation. eIDAS, the electronic personal identity recognition and management system, has been adopted by 15 Member States, supporting people when moving permanently or temporarily, or completing official procedures. E-invoices are another example of an EU-developed interoperable solution successfully reused at national level. 3.Specific major digitalisation polices in the EU – such as the Single Digital Gateway Regulation (SDGR) or the Open Data Directive – have been designed in the implementation phase of the polices, following th