COMMISSION STAFF WORKING DOCUMENT Final evaluation of the European Interoperability Framework (EIF) Accompanying the document Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL laying down measures for a high level of public sector interoperability across the Union (Interoperable Europe Act)
Date: 2022-11-18
Institutions: Directorate-General for Digital Services, European Commission
Subjects: European Interoperability Framework, administrative cooperation, cross-border cooperation, data processing, digital transformation, electronic government, exchange of information, information system, public sector, public service

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 18.11.2022  SWD(2022) 720 final  COMMISSION STAFF WORKING DOCUMENT Final evaluation of the European Interoperability Framework (EIF) Accompanying the document Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL laying down measures for a high level of public sector interoperability across the Union (Interoperable Europe Act) {COM(2022) 720 final} - {SEC(2022) 720 final} - {SWD(2022) 721 final} - {SWD(2022) 722 final}  
 
 
Table of Contents 

EXECUTIVE SUMMARY ................................................................................................. 3 

1. 

INTRODUCTION ....................................................................................................... 5 

2.  WHAT IS THE EXPECTED OUTCOME OF THE INTERVENTION? .................. 7 

2.1  Introduction to interoperability ................................................................................ 7 

2.2  Introduction to the EIF ............................................................................................ 8 

2.3  Description of the intervention and its objectives ................................................. 10 

2.4  Points of comparison ............................................................................................. 15 

3.  HOW  HAS  THE  SITUATION  EVOLVED  OVER  THE  EVALUATION 

PERIOD? ................................................................................................................... 15 

3.1  Policy context ........................................................................................................ 16 

3.2  State of play ........................................................................................................... 17 

4.  EVALUATION FINDINGS (ANALYSIS) .............................................................. 24 

4.1  To what extent was the 2017 EIF successful and why? ........................................ 24 

4.2  How did EU intervention make a difference? ....................................................... 40 

4.3  Is the EIF still relevant? ......................................................................................... 43 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNT? .......................... 45 

5.1  Conclusions ........................................................................................................... 45 

5.2  Lessons learnt ........................................................................................................ 47 

ANNEXES ........................................................................................................................ 50 

Annex I. Procedural information .................................................................................. 50 

Lead DG: Directorate General for Informatics .......................................................... 50 

Organisation and timing ............................................................................................. 50 

Consultation of the RSB ............................................................................................ 50 

Evidence, sources and quality .................................................................................... 51 

Annex II. Methodology and analytical models used .................................................... 53 

Overview of information and data sources ................................................................ 53 

Primary data ............................................................................................................... 54 

Overview of the level of knowledge of respondents.................................................. 55 

Secondary data ........................................................................................................... 56 

Data validation ........................................................................................................... 57 

Methods ...................................................................................................................... 58 

Main limitations ......................................................................................................... 67 

Annex III. Evaluation framework for the EIF evaluation ............................................ 68 

Annex IV. Overview of benefits and costs ................................................................... 85 

1 

 
Annex V. Stakeholder consultation .............................................................................. 87 

Summary of the stakeholder strategy ......................................................................... 87 

Summary of the feedback on the Inception impact assessment ................................. 88 

Consultation activities for the EIF evaluation and Impact assessment ...................... 89 

Results: EIF Evaluation ............................................................................................. 92 

Input of the Expert Group co-design process ............................................................ 95 

Annex VI. Supporting evidence from desk research .................................................... 99 

Annex VI.1. Scoreboards monitoring the implementation of the EIF .......................... 99 

Annex VI.2. Overview of the adoption of NIFs ......................................................... 106 

Annex VI.3. Overview of the mentions of EIF in a sample of EU pieces of 

legislation and policy documents ........................................................................... 113 

Annex VI.4. Evolution of the quality of digital public services and citizens’ use of 

the Internet to interact with public administrations between 2012 and 2019 ........ 115 

Annex VI.5. Academic and grey literature outlining the needs and problems in the 

field of interoperability .......................................................................................... 116 

Annex VII. Solutions contributing to the EIF implementation .................................. 120 

Annex VIII. Mapping of legal initiatives to the EIF .................................................. 125 

2 

 
EXECUTIVE SUMMARY 

Interoperability  enables  data  exchanges  and  facilitates  connections  between  IT  systems  and 
platforms  in  the  delivery  of  digital  services.  It  covers  broader  organisational,  legal  and 
governance aspects enabling fundamental user-centric principles like the once-only principle. 

The EU has been supporting interoperability between public administrations since the 1990s 
through a series of funding programmes such as IDA, IDABC, ISA, ISA² and now Interoperable 
Europe. As part of this support, a European interoperability framework (EIF) was first produced 
in 2004 and then updated in 2010 and 2017. With the introduction of the EIF for the first time 
in 2004, there was a first agreement on a common vision and non-binding guidelines to support 
data exchanges and digitalisation in public administrations in Europe. The implementation of a 
new EIF, adopted in 2017 as COM(2017) 134, was recognised as a key element in enhancing 
interoperable  public  services,  especially  in  a  cross-border  context.  It  contains  a  set  of 
principles, models and recommendations to guide public administrations across the EU and at 
all levels of administration in the design and provision of key digital and interoperable public 
services. 

The EIF recognises the importance of cross-border interoperability as a means of implementing 
EU-wide policies. It provides guidance to public administrations on the design and update of 
national  interoperability  frameworks  (NIFs),  or  national  policies,  strategies  and  guidelines 
promoting interoperability. 

The evaluation shows that the EIF has certainly helped inspire European public administrations, 
on a strategic level, to design and deliver digital public services, and in the design and updating 
of (NIFs) or digital strategies. Quite a lot of Member States have aligned their digitalisation 
policies  and  plans  with  the  EIF,  or  at  least  with  its  guiding  principles.  The  framework  has 
provided  high-level  advice  through  concepts  and  models  to  help  guide  the  creation  of 
interoperable  digital  public  services.  However,  there  is  a  disconnect  between  high-level 
guidance and actual execution or operational implementation, particularly at regional and local 
level. 

The concepts, layered interoperability architecture, and conceptual model for integrated public 
service delivery set out in the EIF are generally synergic, and the suggestions mutually reinforce 
each other. 

The significance of the EIF is attested to by the direct references to it in EU policy and strategy 
documents, but it is less often cited in EU legal acts like directives and regulations. Furthermore, 
the EIF is found to be consistent with initiatives such as Single Digital Gateway Regulation, 
the digital single market strategy, and the data strategy. 

The voluntary nature of the framework is reflected in its uneven and ad hoc uptake at EU and 
Member  State  level.  As  a  result,  the  availability  of  comparable  data  is  limited,  making 
comparisons between countries and sound cost-benefit analysis difficult. 

Several steps for improving the EIF can be derived from experience with implementation: 

  clarify  the  recommendations  with  a  better  structure,  clarity,  granularity,  and 

implementation guidance, reducing some overlaps between them; 

  promote the framework across polices and Member States, encouraging best-practice 

sharing and awareness-raising activities; 
reinforce the EIF’s role as a strategic guide by: 

 

o  enhancing EU interoperability governance and its fundamental role; 
o 

increasing the references to it in related EU legal acts; 

3 

 
o  fostering  coordination  across  EU  and  national  polices  through  common 

interoperability rules; 

o  reinforcing EU policy for interoperability with more binding requirements; 
o  using  conditionality  in  EU  funding  instruments  supporting  digitalisation, 
enforcing interoperability by design and reuse of interoperability solutions; 

  promote sound monitoring of progress in the field of interoperability and of the EIF’s 

effects. 

A  future  enhanced  EU  policy  for  public  administration  interoperability  could  address  those 
issues by: 

o  creating binding interoperability rules; 
o  establishing  a  system  of  governance  that  supports  a  more  practical  and 

o 

actionable EIF; 
identifying common reusable specifications and standards, tools, and solutions 
to support digital checks for new EU policies. 

4 

 
 
 
1.  INTRODUCTION 

Digitalisation holds significant promise for improving public service delivery in Europe and 
generating benefits for public administrations, individuals, and businesses alike, such as time 
savings.  Reforming  internal  processes,  transforming  the  design  and  delivery  of  services, 
boosting data sharing and collaboration, adopting a user-centric approach, and adopting timely 
policies that keep up with technological changes are all part of the path to digital government1. 
All  these  changes  and  digital  transformation  are  highly  dependent  on  the  level  of 
interoperability of the underlying IT systems and platforms supporting digital services and data 
exchanges. If services are digitised in silos, data cannot be reused, shared and exchanged. 

Interoperability  can  be  defined  as  ‘the  ability  of  organisations  to  interact  towards  mutually 
beneficial  goals,  involving  the  sharing  of  information  and  knowledge  between  these 
organisations, through the business processes they support, by means of the exchange of data 
between their ICT systems’2. Legal, organisational, semantic and technical interoperability 
is crucial in the implementation of digital government policies. It enables the design of reusable 
specifications  for  data  exchanges,  or  the  agreement  to  use  existing  ones,  and  facilitates 
connections between IT systems. Both are key in the delivery of digital services. It goes beyond 
the design of IT systems, also covering broader organisational, legal and governance aspects. 
Interoperability is the foundation for the digitalisation of services, helping to put fundamental 
user-centric principles such as the once-only principle into practice. The once-only-principle is 
an e-government concept  that  aims  to  ensure  that  citizens,  institutions,  and  companies  only 
have to provide certain standard information to the authorities and administrations once3. The 
once-only principle is part of the European Union’s plans to further develop the Digital Single 
Market by reducing the administrative burden on citizens and businesses. 

As emphasised in various EU-level initiatives in recent years, ensuring the interoperability of 
digital government has become a top priority, to maximise the potential of digital solutions for 
the  single  market.  The  European  interoperability  framework  (EIF)  adopted  in  2017  as 
COM(2017) 1344 (‘the EIF Communication’), was recognised as a key element in enhancing 
interoperable  public  services,  especially  across  borders.  It  contains  a  set  of  principles, 
models and recommendations to guide public administrations across the EU and at all levels of 
administration  in  the  design  and  provision  of  key  digital  and  interoperable  public  services. 
Ultimately, it supports the free movement of goods, people, services and data throughout the 
EU. 

The EIF is a set of guidelines – principles – similar to the better regulation guidelines, aimed at 
guiding and enhancing interoperability of digital public services at all levels across borders in 
the  EU  and  within  countries  (at  national,  regional,  local  level).  It  is  meant  to  be  used  by 
administrations at every level from EU-wide to local, to design and deliver seamless, integrated 
digital public services to be used by other public administrations, individuals and businesses. 
Furthermore, the EIF Communication was accompanied by support for implementation of the 
framework  from  2016  to  2020,  drawing  on  the  ISA²  and  the  Connecting  Europe  Facility 
programmes.  Through  the  ISA²  Programme,  the  Commission  created  and  promoted 

1 

2 

3 
4 

OECD (2016), Digital Government Strategies for  Transforming Public Services in the  Welfare Areas, 
p. 10, http://www.oecd.org/gov/digital-government/Digital-Government-Strategies-Welfare-
Service.pdf. 
COM(2017) 134 final, Communication from the Commission to the European Parliament, the Council, 
the  European  Economic  and  Social  Committee  and  the  Committee  of  the  Regions,  European 
Interoperability Framework - Implementation Strategy. 
https://en.wikipedia.org/wiki/Once-only_principle 
See footnote 2. 

5 

 
                                                           
interoperability  solutions  that  support  implementation  of  the  EIF  by  the  Member  States,  as 
described in the ISA² final evaluation report5. 

According  to  the  2017  Communication  on  the  new  EIF6,  its  implementation  needs  to  be 
evaluated  to  assess  whether  relevant  recommendations  should  become  part  of  a  mandatory 
instrument. This need for a back-to-back evaluation and impact assessment was reaffirmed in a 
series of Communications in 2020, stating the need for a reinforced interoperability strategy for 
EU  governments  to  foster  coordination  and  the  adoption  of  common  standards  for  public 
services and data flows7. 

Following the ‘evaluate first’ principle, this evaluation focuses on the new EIF introduced in 
2017 and assesses its implementation, seeking to understand the achievements of the framework 
and the lessons that can be learnt to support the overarching goals of enhancing interoperability 
to support the digital transformation of the EU’s public sector. 

The evaluation was based on five evaluation criteria stemming from the Commission’s better 
regulation requirements and nine evaluation questions in Annex III. A mix of data collection 
and desk research was used to gather both primary and secondary data (for further details on 
the methodology, see Annex II). In this, the Commission was assisted by a team of researchers 
led by the Centre for European Policy Studies (CEPS), which summarised its findings in an 
independent evaluation study8. 

The collected data were validated via triangulation to ensure the robustness of the evidence9. 
Data were collected from multiple sources, and several different tools were used to analyse 
the  data  according  to  the  evaluation  criteria.  Data  were  also  used  from  the  EIF  monitoring 
mechanism  exercise  that  takes  place  annually  as  part  of  the  National  Interoperability 
Observatory (NIFO). The monitoring mechanism is meant to provide a macro-level view of 
how Member States are implementing the framework. More specifically, as the evaluation relies 
on both qualitative and quantitative data, a variety of analysis methods was used. Qualitative 
information was aggregated, compared and summarised to substantiate the evidence resulting 
from  the evaluation criteria. To analyse the data  collected during the  consultation activities, 
descriptive  statistics  were  applied.  Finally,  to  complement  the  findings,  other  quantitative 
analyses, including a standard cost model, an econometric model and text mining, were carried 
out when evaluating the effectiveness, efficiency and coherence criteria. 

The  mix  of  data  sources  and  methods  provided  for  solid  conclusions.  Nevertheless,  several 
limitations may have affected the findings. These include the scarcity of data for assessing the 
costs and benefits of implementing the EIF at national level, the need for more time to be able 
to discern the longer-term results, and potential consultation fatigue among respondents. 

5 
6 
7 

8 

9 

https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2021:965:FIN 
See footnote 2. 
COM(2020) 67 final, Communication from the Commission to the European Parliament, the Council, the 
European Economic and Social Committee and the Committee of the Regions, Shaping Europe’s digital 
future. 
CEPS  (2021),  Study  supporting  the  evaluation  of  the  implementation  of  the  EIF.  Available  at: 
https://op.europa.eu/en/publication-detail/-/publication/29d694d4-4696-11ec-89db-
01aa75ed71a1/language-en/format-PDF/source-search. 
European  Commission  (2017),  Tool  #4  Evidence-based  better  regulation  in  the  Better  Regulation 
Toolbox,  available  at  https://ec.europa.eu/info/files/better-regulation-toolbox-4_en.  Last  accessed: 
10 August 2021. 

6 

 
                                                           
2.  WHAT IS THE EXPECTED OUTCOME OF THE INTERVENTION? 

2.1  Introduction to interoperability 

Interoperability is defined as ‘the ability of organisations to interact towards mutually beneficial 
goals,  involving  the  sharing  of  information  and  knowledge  between  these  organisations, 
through the business processes they support, by means of the exchange of data between their 
ICT systems.’10 Legal, organisational, semantic and technical interoperability is crucial in 
the implementation of digital government policies. 

Interoperability  is  what  makes  digitalisation  of  government  services  efficient,  enables  the 
seamless  and  trusted  exchange  of  data  and  connectivity  of  services  across  levels  of 
administration, borders and sectors.  It  is  a foundational  element of  all modern digitalisation 
strategies. 

Digitalisation  of  government  services  without  interoperability  is  possible,  however,  and 
unfortunately still very much the norm. Governments can certainly be digitalises in a way that 
does not allow for data exchange across borders or even inside the same country. However, this 
means foregoing the benefits of learning from best practices, and of reusing existing tools and 
solutions for lower development and maintenance costs; the wheel must be reinvented again 
and again. 

For example, a well-designed online website for a national tax-return system may not be able 
to connect to the population registry, which means that taxpayers need to fill in the same data 
every year. And, possibly, if their signature is not valid in an electronic format, they will have 
to print out the declaration and go in person to the tax office to file it. And, if the tax database 
is not connected to a separate VAT database, businesses will need to enter the same information 
yet again, losing precious time and money. 

But there are also digitally transformed systems where interoperability is built in by default, as 
in Estonia11. Individuals and businesses there spend an average of 3 minutes a year on their tax 
declaration. All the necessary information is already available, reused and exchanged between 
systems seamlessly thanks to the interoperability of data and systems.  

Another example are the national mobile contact tracing and warning applications, which were 
set up in record time and work seamlessly for citizens thanks to a European interoperability 
gateway. At its peak, 19 European countries out of 22 that deployed national contact tracing 
and  warning  applications  were  connected  to  this  gateway  and  were  able  to  exchange 
information12. 

The  EU  Digital  COVID  Certificate  is  also  an  outstanding  example.  It  is  based  on  common 
interoperability data models to enable a common reading, interpretation and verification of the 
data inside. It enables the verification of data originating from different national/regional health 
systems; semantic alignment (first, second, third dose, type of vaccine, types of tests); as well 
as organisational and legal cooperation certainty (authorised entities, business rules related to 
the acceptance of certificates). The system has been a global success, as in addition to EU/EEA 
Member States, also many third countries from five continents have joined it. 

Interoperability enhances to the design of reusable specifications, the quality of seamless data 
exchanges, and facilitates connections between IT systems. All three are key in the delivery of 

10 

11 

12  

COM(2017) 134 final, Communication from the Commission to the European Parliament, the Council, 
the  European  Economic  and  Social  Committee  and  the  Committee  of  the  Regions,  European 
Interoperability Framework - Implementation Strategy. 
https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/digital-
public-administration-factsheets-2021 
https://health.ec.europa.eu/ehealth-digital-health-and-care/ehealth-and-covid-19_en 

7 

 
                                                           
digital services. Interoperability goes beyond the design of IT systems; it also covers broader 
organisational,  legal  and  governance  aspects.  Interoperability  is  the  foundation  for  the 
digitalisation of services, upholds fundamental digital rights and principles (like the once-only 
principle),  and  supports  the  design,  interconnection,  and  delivery  of  digital  services  in  a 
coordinated way so that data can flow across different domains and across borders. However, 
this  requires  strong  interoperability  foundations.  Sound  interoperability  governance  at  both 
national  and  European  level  is  a  fundamental  part  of  the  legal,  organisational  and  technical 
specifications towards those common goals. 

The ambitious Communication on a digital single market strategy, put forward by the Juncker 
Commission in 201513, the interoperability agenda needed to be revised and brought into line 
with the emerging challenges in the field of ICT and European public services. Among other 
relevant digital initiatives to interoperability, there is the Directive on reuse of Public Sector 
Information14  (subsequently  revised  and  renamed  as  the  Open  Data  Directive  in  2019),  the 
INSPIRE directive15, the eIDAS Regulation16, the EU eGovernment action plan 2016-2020, the 
single digital gateway (proposed in 2017 and subsequently adopted as a Regulation in 2019)17, 
and the European Cloud initiative. 

2.2  Introduction to the EIF 

The EU has been supporting interoperability between public administrations since the 1990s 
through  a  series  of  funding  programmes  such  as  IDA,  IDABC,  ISA18,  ISA²,  and  now 
Interoperable  Europe.  As  part  of  this  support,  the  first  EIF  was  produced  in  2004  and  then 
updated in 2010 and 2017. With the introduction of the EIF in 2004, there was for the first time 
an agreement on a common vision and non-binding guidelines to support data exchanges and 
digitalisation in Public Administrations in Europe. 

Responding to the call for a revised framework for interoperability in the EU, a new EIF was 
adopted  in  2017  as  part  of  the  Communication  from  the  European  Commission  adopted  on 
23 March 201719. The new 2017 EIF expanded on the previous 2004 version adopted by the 
PEGSCO  committee  of  the  IDABC  programme  and  the  2010  version  adopted  by  the 
Commission20. It brought more targeted recommendations, taking new technological and policy 
developments into account. The number of recommendations increased from 25 to 47 in the 
2017 EIF and several recommendations were updated or newly developed to support relevant 
EU policies and initiatives. 

The EIF establishes a set of guidelines and principles, similar to the better regulation guidelines. 
It aims to guide and enhance the interoperability of digital public services at all levels across 
borders in the EU and within countries (at national, regional and local level). The framework 
has three goals: 

 

to  inspire  and  guide  European  public  administrations  in  designing  and  delivering 
seamless  European  public  services  to  other  public  administrations,  individuals  and 

13 
14 
15 
16 

17 

18  

19 
20 

COM(2015) 192 final. 
Directive 2013/37/EU. 
Directive 2007/2/EC. 
Regulation (EU) No 910/2014 on electronic identification and trust services for electronic transactions in 
the internal market. 
See  the  Staff  Working  Document  accompanying  the  2017  EIF  Communication:  https://eur-
lex.europa.eu/legal-content/ES/TXT/?uri=CELEX:52017SC0112. 
IDA  – interchange of data between administrations; IDABC  - interoperable delivery of pan-European 
eGovernment services to public administrations, businesses and citizens; ISA – Interoperability Solutions 
for Public Administrations. 
COM(2017) 134 final. 
COM(2010) 744 final. 

8 

 
                                                           
businesses in a way that is digital, cross-border by default (i.e. accessible to the public in 
the EU) and open by default; 

  provide  guidance  to  public  administrations  on  the  design  and  update  of  national 
interoperability  frameworks  (NIFs),  or  national  policies,  strategies  and  guidelines 
promoting interoperability; 

  contribute to the establishment of the digital single market  by  fostering cross-border 

and cross-sectoral interoperability for the delivery of European public services. 

The EIF Communication was accompanied by an interoperability  action plan to  support the 
implementation of the EIF from 2016 to 2020, with a series of solutions supported by the ISA² 
and  the  Connecting  Europe  facility  (CEF).  Through  the  ISA²  programme,  the  Commission 
created and promoted interoperability solutions that support implementation of the EIF by the 
Member States as described in the ISA² final evaluation report21.  

The  EIF  focuses  on  cross-border  interoperability  between  governments  needing  support  to 
implement EU-wide policies. Given that digitalisation applies to all public administrations in 
all domains and at  all levels,  it is also  applied directly in  the countries for their own digital 
services  and  serves  as  inspiration  for  the  creation  of  national  interoperability  frameworks 
(NIFs). Positive spillover effects were anticipated from providing Member States with common 
guidance to start working on interoperable services within their borders. Common interoperable 
governance structures, tools, specifications and services support seamless interactions across 
the various administrative levels in the country (central, regional, local levels), but also across 
some domains (justice, education, health and taxation). Most of all, the framework is a step 
towards developing interaction and cooperation required in  a  cross-border setting. Having a 
proper interoperability set-up within each Member State will help mitigate the complexities of 
coordinating  interactions  across  borders  that  involve  several  domains  and  competent 
authorities. 

The  EIF  helps  remove  barriers  to  the  exchange  of  data,  connect  digital  services  across  the 
various administrative levels of the country and across borders and sectors in the EU. 

The framework is focused on supporting public organisations in working together on defining, 
designing and implementing digital services in a coordinated and efficient manner – starting by 
commonly  designing  and  aligning  back-office  protocols,  specifications,  infrastructures, 
platforms and IT tools. However, it also covers fundamental issues for individual and corporate 
end users, such as accessibility, user-centric design and the once-only principle. 

At  EU  level,  relevant  stakeholders  are  the  EU  institutions  and  policy  units  responsible  for 
formulating, designing and implementing EU policies with digital implications. This includes 
units in DG GROW, DG CONNECT and the Joint Research Centre (JRC). As an example of 
EU  policies,  the  Single  Digital  Gateway  Regulation  led  by  DG  GROW  implements  a  pan-
European  technical  system  using  the  principles  of  the  EIF  and  some  solutions  from  the 
interoperability  action  plan,  like  eDelivery  and  common  vocabularies  for  cross-border  and 
cross-sector evidence exchanges. 

Member  States’  digital  government  authorities  and  experts  (government  chief  information 
officers, practitioners and academic experts) are longstanding close stakeholders, but the public 
bodies (ministries and agencies) mainly work on digitalisation and interoperability of digital 
services. In its factsheets on digital public administration and interoperability (Chapter 5), NIFO 
provides  regular  and  up-to-date  information  on  those  authorities22.  The  information  is  also 

21 
22 

https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2021:965:FIN. 
https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/digital-
public-administration-factsheets. 

9 

 
                                                           
available  in  a  dedicated  section  of  the  NIFO  website23.  Regional  and  local  authorities  have 
recently  become  more  involved,  particularly  through  the  definition  of  an  EIF-inspired 
framework for smart cities and local communities in the context of the Living-in.EU movement. 
The EIF, through the ISA² programme, has supported and encouraged the take-up of solutions 
by private companies involved in public sector development. Other important stakeholders are 
academia,  and  standard-setting,  consulting  and  other  private  sector  organisations  (e.g. 
GovTech) that work closely with governments providing technical support. 

The revised version of the EIF, which is subject of this evaluation, was created in a collaborative 
way with public administration experts and the national authorities in the Member States that 
are  involved  in  designing  and  coordinating  implementation  of  digital  service  policies  or 
strategies. The text was discussed and agreed upon in several meetings between Member States’ 
representatives for the ISA² programme, from national digitalisation departments, and policy 
departments  (Directorates-General)  responsible  for  single  market  digitalisation.  In  addition, 
academia,  standardisation  organisations  and  ICT  companies  gave  their  views  in  several 
workshops. Finally, a public consultation was held24. 

Their needs were considered and are an integral part of the EIF adopted in 2017. 

2.3  Description of the intervention and its objectives 

To address the challenges identified, the EIF outlined several key objectives together with a set 
of  inputs  that  comprise  the  main  components  of  the  EIF:  the  underlying  interoperability 
principles of European public services, the layered interoperability model and the conceptual 
model for integrated public services provision, as well as the recommendations attached to each 
of  the  three  components.  The  Communication  on  the  EIF  also  included  an  interoperability 
action plan, detailing a set of key actions to be undertaken between 2017 and 2020 to support 
implementation  of  the  EIF.  As  set  out  in  the  Communication,  the  Commission  ensured  the 
promotion, support and implementation of the Interoperability Action Plan mainly through the 
ISA²  programme25. At  the  time  of  its  adoption,  the  framework  was  expected  to  produce  a 
series of outcomes and impacts. 

The logical links between the needs and problems, objectives, inputs, and expected outcomes 
and impacts are summarised in Figure 1 (EIF intervention logic). The operational objectives 
and outputs are not included, as they were assessed during the parallel running evaluation of 
ISA²26. 

23 

24 

25 

26 

https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/digital-
public-administrations-and-eif-national-responsible-bodies. 
https://ec.europa.eu/isa2/consultations/results/result_impact-assessment-for-the-revision-of-the-eis-
eifl_en. 
The  EIF  Communication  states  that,  ‘The  Commission  will  support,  promote  and  monitor  the 
implementation  of  the  interoperability  action  plan,  and  the  European  interoperability  framework  in 
general,  primarily  through  the  ISA²  programme.’  The  Communication  also  mentions  that  additional 
support may come from other instruments such as Horizon 2020, the CEF, the European structural and 
investment funds and the structural reform support programme (SRSP). See: COM(2017) 134 final, p. 8. 
See SWD/2021/965 final (Staff working document), COM/2021/965 final (Report), CEPS (2021) Study 
supporting the final evaluation of the ISA2 programme. Available at: https://op.europa.eu/en/publication-
detail/-/publication/afa4297a-0acc-11ec-adb1-01aa75ed71a1/language-en/format-PDF/source-
229005953. 

10 

 
                                                           
The needs, problems27 and drivers that the EIF intended to address were identified by analysing 
the 2017 EIF Communication28, the accompanying Staff Working Documents29, and the results 
of the consultation activities carried out in 2016 to inform the revision process30. The EIF was 
established to address one key need experienced by public administrations in the EU: 

 

the need for more specific guidance for public administrations on how to improve the 
governance of their interoperability activities31. 

The  Staff  Working  Document  accompanying  the  2017  EIF  Communication32  outlined  two 
main problems in the field of interoperable digital public services: 

 
 

the fragmented delivery of digital public services in the EU and 
fragmentation in the organisation and format of public data in the EU. 

These problems were attributable to five main drivers: 

incoherent governance of action to implement interoperability; 

 
  cross-organisational barriers; 
 
  scarcity of tools and solutions for interoperability in the delivery of public services; and 
 

resource constraints in relation to interoperability; 

legal issues affecting interoperability. 

27 

28 
29 

30 

31 
32 

Needs refer to prerequisites for the efficient delivery of European public services, more specifically for 
interoperability. Problems consist of specific bottlenecks that prevent need from being addressed. The 
drivers are the underlying causes that lead to the problems identified. 
COM(2017) 134 final. 
SWD(2017) 112 final, Commission Staff Working Document: Revision of the European Interoperability 
Framework – Analysis; SWD(2017) 113 final, Commission Staff Working Document: Revision of the 
European Interoperability Framework - Synopsis report of the consultation activities. 
Factual summary of the contributions received during the public consultation undertaken by the European 
Commission  to  assess  the  impacts  of  the  revision  of  the  European  Interoperability  Framework  And 
Strategy,  available  at:  https://ec.europa.eu/ISA²/sites/isa/files/eif-public-consultation-factual-summary-
en.pdf. 
COM(2017) 134 final, p. 4. 
SWD(2017) 112 final. 

11 

 
                                                           
Figure 1: EIF intervention logic  

Source: Study supporting the evaluation of the implementation of the EIF.

12 

 
 
The objectives of the EIF were presented in a hierarchical order; furthermore, the achievement 
of specific objectives was seen as a pre-condition for attaining the general objective. 

General objective 

  To  help  public  administrations  provide  key  interoperable,  user-centric,  digital  public 
services to businesses and individuals, at EU, national, regional and local levels135, thus 
supporting the free movement of goods, people, services and data throughout the EU136. 

Specific objectives137 

  Specific objective 1:  to inspire European public  administrations at  all levels  in  their 
efforts  to  design  and  deliver  seamless  European  public  services  to  other  public 
administrations, individuals and businesses which, where possible, are digital by default 
(i.e. providing services and data preferably via digital channels), cross-border by default 
(i.e.  accessible  for  all 
the  EU),  open  by  default  (i.e.  enabling  reuse, 
participation/access and transparency), and which follow the once-only principle. 
  Specific objective 2: to provide guidance to public administrations on the design and 
update  of  NIFs,  or  national  policies,  strategies  and  guidelines  promoting 
interoperability. 

in 

  Specific  objective  3:  to  contribute  to  the  establishment  of  the  digital  single  market 
(DSM) by fostering cross-border and cross-sectoral interoperability for the delivery of 
European public services. 

  Specific  objective  4:  to  account  for  technological  developments  and  trends  in  the 

development of interoperability guidance and tools138. 

As mentioned, the intervention logic of the EIF does not cover operational objectives, as they 
are covered in a parallel evaluation study on ISA²139. 

What activities are supported by the 2017 EIF? 

The EIF provides 47 recommendations that are grouped under the three main components of 
the EIF (an overview of the interrelations between the components is provided in Figure 2): 

  Underlying  principles  of  European  public  services.  The  EIF  puts  forward 
12 interoperability  principles  to  guide  the  design  of  interoperable  European  public 
services. The principles include the following: 

o 

the principles of subsidiarity and proportionality (a case for EU action has to be 
made); 

o  a  set  of  core  interoperability  principles:  openness,  transparency,  reusability, 

technological neutrality, and data portability; 

o  principles  on  user  needs  and  expectations:  user-centricity,  inclusion  and 

accessibility, security and privacy and multilingualism; and 

o  principles  for  cooperation  among  public  administrations:  administrative 
simplification, preservation of information and assessment of effectiveness and 
efficiency. 

135 
136 
137 
138 
139 

Based on COM(2017) 134 final, Annex 2, p. 6. 
COM(2017) 134 final, p. 5. 
Ibid, Annex 2, p. 5. 
COM(2017) 134 final, p. 3. 
See SWD/2021/965 final (Staff working document), COM/2021/965 final (Report), CEPS (2021), Study 
supporting the final evaluation of the ISA2 programme. Available at: https://op.europa.eu/en/publication-
detail/-/publication/afa4297a-0acc-11ec-adb1-01aa75ed71a1/language-en/format-PDF/source-
229005953. 

13 

 
                                                           
  Layered interoperability model. The layered interoperability model includes the four 
layers of interoperability (organisational, semantic, legal, and technical interoperability) 
as well as a crosscutting layer relating to integrated public service governance, brought 
together  into  a  comprehensive  approach  through  a  final  and  crucial  component: 
interoperability governance. 

  Conceptual model for interoperable public services. The concept is that European 
public services should be interoperable by design. The model consists of several basic 
components:  coordination  function,  internal  information  sources  and  services,  base 
registries, open data, catalogues, external information sources and services, and security 
and privacy protection. 

Figure 2: The EIF conceptual model 

Source: COM(2017) 134 final, Communication from the Commission to the European  Parliament, the Council, 
the  European  Economic  and  Social  Committee  and  the  Committee  of  the  Regions,  European  Interoperability 
Framework - Implementation Strategy, Annex 2. 
What was the expected outcome of the 2017 EIF?’ 

When created, the EIF was meant to deliver short-, medium- and long-term effects for certain 
categories of stakeholders. While the most immediate and operational results (the ‘outputs’) of 
the  EIF  were  expected  to  emerge  from  relevant  ISA²  measures,  it  was  thought  that 
implementation of the EIF should in itself be able to generate results affecting European public 
administrations at all levels, as well as the public and businesses in the EU. These expected 
results are summarised below. 
Outcomes (expected)140 

  European  public  administrations  at  EU,  national,  regional  and  local  levels  use  the 
guidance provided by the EIF to effectively deliver digital public services responding 
to the needs of people and businesses, and to establish cross-organisational relationships 
supporting cross-border and cross-sector cooperation. 

  National policies, strategies and guidelines consider and promote interoperability. EU 
the 

legislation  and  policies  consider 
interoperability potential of the national follow-up measures. 

interoperability 

implications, 

increasing 

140 

The outcomes reflect the specific objectives. The outcomes were derived from: COM(2017) 134 final, 
Annex 2, pp. 5 and 7. 

14 

 
 
                                                           
  A  coherent  cross-border  and  cross-sector  interoperability  environment  is  achieved, 

supporting the digital single market in the EU. 

  New  technological  developments  are  considered  when  designing  interoperability 

guidance and tools. 

Impacts (expected) 

  Available interoperable, user-centric public services in the EU141. 
  Developing a European public services ecosystem in which: owners and designers of 
systems  and  public  services  become  aware  of  interoperability  requirements;  public 
administrations are ready  to  collaborate with  each other and with  businesses and the 
public;  information  flows  seamlessly  across  borders,  supporting  the  single  market, 
particularly the digital single market142. 

2.4  Points of comparison 

This evaluation focuses on the achievements and areas for improvement of the 2017 EIF since 
its adoption. By 2016, there was a relatively good alignment between the Member States’ NIFs 
and the guidance provided by the 2010 EIF. The average EIF-NIF alignment level across the 
EU in 2016 was estimated at 75%. However, at the level of actual implementation, more work 
remained to be done. The average NIF implementation level across the Member States in 2016 
was estimated at 56%143. 

In the run-up to the revision of the EIF, a public consultation was carried out to gather the views 
of  stakeholders  on  existing  problems  and  the  future  of  the  EIF.  Among  the  key  problems 
identified by stakeholders were144: 

 
 
 
 

lack of a comprehensive overview of existing interoperability initiatives; 
limited uptake of common standards in the development of solutions; 
lack of monitoring of implementation of interoperability initiatives; 
lack  of  both  financial  and  human  resources 
interoperability initiatives and solutions; and 

to  support 

implementation  of 

  no  assessment  of  costs  and  benefits  of  interoperability  in  the  process  of  developing 

legislation at national level. 

In  addition,  the  public  consultation  brought  to  light  the  main  expectation  of  consulted 
stakeholders for the future EIF: time and cost savings. In the consultation, 49 stakeholders (74% 
of the respondents) said they expected time savings and 46 (70%) expected cost savings and 
greater transparency145. 

3.  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

This chapter gives an overview of how the interoperability landscape has changed since the 
new EIF was adopted, both in terms of legislation and in terms of monitoring. This Chapter 

141 
142 
143 

144 

145 

COM(2017) 134 final, p. 9. 
COM(2017) 134 final, Annex 2, p. 7. 
Gatti, R., Carbone L., Mezzapesa, V. (2017), State of Play of Interoperability in Europe - Report 2016, 
European Commission, pp. 15-22. Available at: 
https://ec.europa.eu/isa2/sites/default/files/report_2016_rev10_single_pages_0.pdf. 
Factual summary of the contributions received during the public consultation undertaken by the European 
Commission  to  assess  the  impacts  of  the  revision  of  the  European  Interoperability  Framework  And 
Strategy,  p. 2.  Available  at:  https://ec.europa.eu/ISA²/sites/isa/files/eif-public-consultation-factual-
summary-en.pdf. 
Ibid., p. 3. 

15 

 
                                                           
examines the policy context surrounding the adoption of the EIF before addressing the level of 
implementation. 

3.1  Policy context 

The policy context of the EIF has been very dynamic since the adoption of the 2017 version. In 
recent  years,  digitalisation  of  public  administrations  and  services  has  risen  up  the  political 
agenda at EU and national level, as demonstrated by the Resilience and Recovery Funds and 
the national plans: EU Member States are planning to invest more in the digitalisation of their 
public administrations. 

Back in 2017, the priorities for interoperability were reinforced through the Tallinn Declaration 
on  eGovernment146.  The  high-level  Ministerial  Declaration  emphasised 
the  role  of 
interoperability as a strategic component for ensuring the effective delivery of public services 
in the EU. A similar emphasis on interoperability is seen in a European Parliament resolution 
of 2017147. It underlines that the digitisation of public administrations should aim to ‘promote 
the  better  exercise  of  citizenship,  improve  the  quality  of  life  for  citizens  and  the  social  and 
economic development of the regions, enhance citizens’ understanding of and involvement in 
public services’, while improving their efficiency and cost-effectiveness. 

The  digitalisation  of  public  services  has  been  progressing  relentlessly,  gaining  even  more 
momentum  due  to  the  pandemic.  The  digital  economy  and  society  (DESI)  index  and 
eGovernment benchmark report show that all EU Member States have made progress in the 
area of digitalisation, but the overall picture across Member States is mixed, and despite some 
convergence, the gap between the EU’s frontrunners and those with the lowest DESI scores 
remains large. 

In 2019, the Council underlined in its conclusions on the future of a highly digitised Europe 
beyond 2020148 that interoperability will continue to play a key role in the EU’s digital policy. 
The Von der Leyen Commission has set strong ambitions for digital transformation in the EU, 
as emphasised in the following Communications published in 2020 and 2021. 

  Shaping Europe’s digital future, which sets out the Commission’s vision for the next 
five years on harnessing the potential of the digital transformation for the EU and its 
citizens, underpinned by European values. One of the key actions in the Communication 
is the development  of a  ‘reinforced EU governments interoperability  strategy’ to 
foster coordination and the adoption of common standards for public services and data 
flows by 2021149. 

  A European strategy for data, which proposes nine European data spaces that rely on 

interoperability to enable seamless data exchange150. 

  The white Paper on artificial intelligence (AI)151 and the subsequent proposal for a 

regulation laying down harmonised rules on AI152. 

146 

147 
148 

149 
150 
151 
152 

Tallinn Declaration on eGovernment at the ministerial meeting during Estonian Presidency of the Council 
of the EU on 6 October 2017. 
European Parliament resolution of 16 May 2017 on the EU eGovernment Action Plan 2016-2020. 
Council  conclusions  on  the  future  of  a  highly  digitised  Europe  beyond  2020:  ‘Boosting  digital  and 
economic competitiveness across the Union and digital cohesion’, Brussels, 7 June 2019. 
COM(2020) 67 final. 
COM(2020) 66 final. 
COM(2020) 65 final. 
COM/2021/206 final. 

16 

 
                                                           
  A new industrial strategy for Europe for innovation and a greener and more digital 
industry, for which building EU leadership in key areas including data applications is 
essential153. 

  The  SME  strategy  for  a  sustainable  and  digital  Europe,  which  identifies  the 
opportunities that digitalisation holds for small and medium-sized enterprises (SMEs) 
in the EU; wider access to data and the reusability of data can support the development 
of innovative SMEs154. 

  The  long-term  action  plan  for  better  implementation  and  enforcement  of  single 
market rules, focusing on better using digital tools to improve access to information, 
fostering cooperation between the Member States, and testing and applying advanced 
digital tools for public services155. 

  The 2030 Digital Compass: the European way for the digital decade156, setting key 
priorities and targets for the digital transition in the EU over the next decade. When it 
comes to delivering digital public services, the Communication emphasises the need to 
‘ensure interoperability across all levels of government and public services’157. 

In 2020, the Member States built  on the Tallinn Declaration with  the  Berlin Declaration on 
digital  society  and  value-based  digital  government158.  The  Member  States  and  the  EU 
institutions committed themselves to developing a more vigorous and more interoperability-
focused  digital  transformation  policy  for  government  and  to  supporting  the  digital 
transformation of public administrations. In achieving these ambitions, interoperability plays a 
key  enabling  role,  ensuring  a  level  of  coordination,  common  processes  and  models  for  the 
achievement of further goals. 

The commitments included in these documents are also reflected in the budgetary allocations 
for the next several years. In particular, the new Multiannual Financial Framework includes a 
new programme to bolster the digital agenda in the EU called the Digital Europe Programme. 
With  a  planned  budget  of  EUR 7.5 billion  (at  current  prices)  from  2021  to  2027,  this  new 
financial  instrument  will  support  five  key  areas  of  the  digital  transition:  supercomputing, 
artificial  intelligence,  cybersecurity,  advanced  digital  skills  and  European  digital  innovation 
hubs  (EDIHs)159.  The  Digital  Europe  programme  will  ensure  continued  of  funding  for 
measures to increase interoperability in the EU. 

3.2  State of play 

To understand the current state of play in the implementation of the EIF, it is important to note 
that the EIF is a non-binding instrument providing guidance that can be followed voluntarily 
by the Member States without a precise timeline. Therefore, delays in the implementation of 
the EIF are difficult to identify, given the lack of a specific timeframe (for more details, see 
Annex  VI,  Supporting  evidence  from  desk  research).  Most  of  the  Member  States  (15)  have 
created an NIF based on the EIF, namely Belgium, Bulgaria, Estonia, Greece, Spain, Croatia, 
Italy, Cyprus, Luxembourg, Malta, Netherlands, Austria, Romania, Slovenia and Sweden. The 

153 
154 
155 
156 
157 
158 

159 

COM(2020) 102 final. 
COM(2020) 103 final. 
COM(2020) 94 final. 
COM(2021) 118 final. 
Ibid., p. 11. 
Berlin Declaration on digital society and value-based digital government at the ministerial meeting during 
the German Presidency of the Council of the European Union on 8 December 2020. 
The reported sum allocated to the Digital Europe programme is based on the European Council agreement 
on 
the  2021–2027  Multiannual  Financial  Framework  and  Next  Generation  EU.  See: 
https://www.ceps.eu/ceps-publications/reading-between-the-lines-of-council-agreement-on-the-mff-
and-next-generation-eu/. 

17 

 
                                                           
others  have  an  NIF  or  similar  strategic  documents  partially  inspired  by  the  EIF.  Only  three 
countries, Ireland, Lithuania and Finland, have not implemented an EIF-inspired NIF. 

Figure 3 reflects the extent to which the EIF has been taken up, at least conceptually, across the 
EU, based on data collected in January 2022. 

Figure 3: Implementation of the EIF across Europe 

Note: The legend corresponds to the extent to which EU countries have adopted NIFs that are fully or partly based 
on the EIF, as follows: (0) Light blue: countries that have not adopted an EIF-inspired NIF; (1) Middle blue: 
countries that have adopted NIFs or similar strategic documents partly inspired by the EIF; and (2) Dark blue: 
countries that have adopted NIFs based on the EIF. For each country, the year of the first adoption is displayed. 
Source: Study supporting the evaluation of the implementation of the EIF. 

The  EIF  monitoring  mechanism  (NIFO)  delivers  a  high-level  assessment  of  the  progress  in 
implementing the three main components of the EIF. It examines how the components have 
been adopted as guidance principles in the national strategies and polices. 

Three scoreboards have been developed: the principles of interoperability, the interoperability 
layers,  and  the  conceptual  model  for  integrated  public  services  provision.  A  set  of  68  key 
performance  indicators  (KPIs)  is  used  in  compiling  the  scoreboards.  The  final  output  (see 
Annex VI.2) consists of a ‘traffic light’ assessment whereby the relevant KPIs are aggregated 
for each component of the scoreboard as follows: 

[1]  
[2]  

[3]  

for the first scoreboard, the aggregation is done for each of the 12 principles; 
for the second scoreboard, the relevant KPIs are aggregated for each layer of the EIF 
interoperability model; and 
for the third scoreboard, the aggregation of the KPIs is carried out for each component 
of the conceptual model for integrated public services provision. 

The  ‘traffic  light’  assessment  produces  a  score  value  scale  between  1  and  4,  based  on  the 
aggregation of the KPIs mentioned above. The first results of the EIF monitoring mechanism 

18 

 
 
are  based  on  data  collected  in  2019.  The  results  are  presented  in  the  2020  digital  public 
administration factsheets for each Member State160. 

To  gain  an  overview  of  the  implementation  of  the  EIF  at  EU  level,  the  results  of  the 
monitoring mechanism exercise can be presented based on the progress in each scoreboard for 
all EU Member States (the full overview is presented in Annex VI.2). Overall averages per 
Member  State  per  scoreboard  were  calculated,  to  provide  a  panoramic  overview  of  those 
areas of the EIF in which most Member States score well and the areas where further efforts 
would be needed (see Figure 4, Figure 5 and Figure 6). 

Figure 4.a: Average scores in 2020 for Scoreboard 1: Principles of interoperability 

Note: For a full description of the principles, see Annex 2 to the 2017 EIF Communication. 
Source: Study supporting the evaluation of the implementation of the EIF based on the data available on Joinup 
concerning the EIF monitoring mechanism and the underlying database provided by DIGIT.D2. 

160 

details 

further 

For 
https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/digital-
public-administration-factsheets. 

‘Digital  Public  Administration  Factsheets’, 

Joinup, 

please 

see: 

19 

 
 
 
 
                                                           
Figure  4.b:  Change  in  average  scores  2019-2020  for  Scoreboard  1:  Principles  of 
interoperability 

Source: Study conducted by Wavestone, 2022 

20 

 
 
 
Figure 5: Average scores in 2020 for Scoreboard 2: Interoperability layers 

Note: Countries are marked in grey where missing data for some sub-indicators make it impossible to calculate 
an overall average (see Annex VI.1 for a full overview). 
Source: Study supporting the evaluation of the implementation of the EIF based on the data available on Joinup 
concerning the EIF monitoring mechanism and the underlying database provided by DIGIT.D2. 

Figure 6: Average scores in 2020 for Scoreboard 3: Conceptual model 

Note: Countries are marked in grey where missing data for some sub-indicators make it impossible to calculate 
an overall average (see Annex VI.1 for a full overview). 
Source: Study supporting the evaluation of the implementation of the EIF on Joinup concerning the EIF monitoring 
mechanism and the underlying database provided by DIGIT.D2. 

21 

 
 
 
 
 
Implementation of the 2017 EIF over time 

The pre-2017 EIF version was much more theoretical, with narrower scope. At that time, the 
monitoring161mainly involved checking how well aligned the main principles in  the national 
interoperability  strategies  were.  Under  the  2017  EIF,  the  level  of  implementation  is  also 
assessed.  This  is  one  of  the  reasons  why  it  is  not  possible  to  make  a  direct  comparison  of 
progress in the Member States’ performance under the previous framework with their progress 
under the current one. 

Nevertheless,  small  and  positive  improvements  are  apparent  from  a  high-level  comparative 
analysis of results from the EIF monitoring mechanism for 2019 and 2020. 

Figure  4.b  presents  the  improvements  made  by  the  Member  States  in  implementing  the 
recommendations linked to the principles of interoperability. 

Several countries have improved their implementation of the principles. Portugal has seen the 
biggest  improvement.  Implementation  is  assessed  by  gauging  how  well  aligned  NIFs  and 
interoperability strategies are with the EIF and, if needed, tailored and extended to address the 
national context and needs. Along with Portugal, the number of countries reporting a high level 
of alignment increased from 14 in 2019 to 16 in 2020. 

With  regards  to  interoperability  layers,  Greece  and  Poland  have  made  most  progress  in 
implementing  the  recommendations,  through  increased  efforts  on  governance.  Results  show 
significant improvements, particularly on holistic governance across all administrative levels 
and  sectors,  and  the  implementation  of  well-defined  processes  for  selecting  and  adopting 
standards and specifications. 

In the implementation of the recommendations related to the conceptual model, Sweden and 
Lithuania  have  made  particular  efforts  to  increase  their  level  of  alignment  with  the  EIF 
conceptual model for integrated public services provision. They both increased their score by 
an  average  of  0.57  points,  by  putting  in  place  catalogues  of  public  services,  open  data  and 
interoperability solutions. 

Several  external  factors  can  affect  the  overall  performance  and  implementation  of  the  EIF, 
either positively, supporting the overall goals of the framework, or negatively, hindering the 
achievement of expected results. The factors were selected based on findings from the relevant 
literature  and  analyses  of  key  trends  relevant  for  interoperability  initiatives  in  the  EU.  The 
following  external  factors  were  found  to  have  a  potential  positive  impact  on  the 
implementation of the EIF: 

 

 

 

technological advances bringing new opportunities to improve and foster the delivery 
of public services, which can make the need for interoperability more salient and thus 
provide  a  further  incentive  to  ensure  that  interoperability  principles  and  models  are 
taken into account; 
the development of national initiatives on interoperability in the public sector, which 
can show the commitment to making progress on interoperability; and 
the deployment of cross-sectoral initiatives such as smart city initiatives, showcasing 
the relevance of interoperability at the intersection of different sectors. 

Other external factors were found to have a potential negative impact and thus jeopardise 
implementation of the EIF162. These factors were: 

161 

162 

https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/past-nifo-
analytical-models-and-factsheets. 
Interoperability initiatives are faced with a variety of challenges. The key factors were selected taking 
into account the relevant literature, including, for instance: Leosk N., Põder I., Schmidt C., Kalvet T., 

22 

 
                                                           
 
 

 

technical challenges, such as legacy systems; 
limited  resources,  reflecting,  for  example,  difficulties  in  accessing  funding  and 
attracting skilled workers in the field of interoperability; 
institutional complexity, referring to the complex system of roles distributed between 
the  different  levels  of  public  administration  and  of  complexity  stemming  from  legal 
requirements; and 

  changing political priorities, which may impact measures to improve interoperability in 

a given country. 

The impact of these factors on the performance of the EIF was tested through consultations 
with targeted stakeholders as described in Annex V. 

EIF supporting instruments 

The EIF is a theoretical framework, not a set of solutions. However, as a result of the associated 
interoperability action plan, a set of EU reusable IT solutions have been developed under the 
ISA² and CEF funding programmes to support the EIF implementation. Furthermore, the EIF 
toolbox163 published on Joinup provides a catalogue of reusable solutions linked to the main 
areas, principles and recommendations of the EIF. Moreover, any interested stakeholder can 
take them up as they are based on open source specifications. 

In fact, the EIF Communication was accompanied by action to support the implementation of 
the  European  interoperability  framework  over  2016-2020,  through  a  series  of  solutions 
supported by the ISA² and CEF programmes. The Commission, through the ISA² programme, 
then created and promoted interoperability solutions that support implementation of the EIF by 
the Member States as described in the ISA² final evaluation report164. 

Up to the end of 2020, the ISA² programme supported a total of 54 actions, grouped in nine 
packages, which were devised on a yearly basis through an annual rolling work programme. 
The nine packages are: 

1.  key and generic interoperability enablers; 
2.  semantic interoperability; 
3.  access to data / data sharing / open data; 
4.  geospatial solutions; 
5.  eProcurement / eInvoicing; 
6.  decision-making and legislation; 
7.  EU policies – supporting instruments; 
8.  supporting instruments for public administrations; and 
9.  accompanying measures. 

ISA² managed 39 actions in its first year of operation, 43 actions in 2017, 53 actions in 2018, 
and 54 actions under both the 2019 and 2020 rolling work programmes. 

In  its  digital  policy  hub,  the  National  Interoperability  Framework  Observatory  provides 
examples  and  case  studies  of  national  IT  solutions  and  services  implemented  in  public 

Krimmer  R.  (2021),  Drivers  for  and  Barriers  to  the  Cross-border  Implementation  of  the  Once-Only 
Principle.  In:  Krimmer  R.,  Prentza  A.,  Mamrot  S.  (eds)  The  Once-Only  Principle.  Lecture  Notes  in 
Computer  Science,  vol  12621.  Springer,  Cham.  https://doi.org/10.1007/978-3-030-79851-2_3;  Kalvet, 
T., Toots, M., & Krimmer, R. (2018), Contributing to a digital single market for Europe: barriers and 
drivers of an EU-wide once-only principle. In Proceedings of the 19th Annual International Conference 
on  Digital  Government  Research:  Governance 
1-8). 
https://dl.acm.org/doi/10.1145/3209281.3209344. 
https://joinup.ec.europa.eu/collection/national-interoperability-framework-observatory/eif-toolbox. 
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=COM:2021:965:FIN. 

the  Data  Age 

(pp. 

in 

163 
164 

23 

 
                                                           
administration following the EIF principles. EU solutions that follow the EIF principles can be 
found in the EIF toolbox, which is now also starting to include national solutions. 

Annex VII gives a comprehensive overview of solutions supporting EIF implementation that 
are available to Member States, as part of the interoperability action plan described above. Most 
of them are also accessible through the EIF toolbox165created to that end. 

4.  EVALUATION FINDINGS (ANALYSIS) 

4.1  To what extent was the 2017 EIF successful and why? 

To  assess  whether  the  EIF  has  been  successful,  this  chapter  assesses  how  well  it  meets  the 
following  three  criteria:  i)  effectiveness;  ii)  efficiency;  and  iii)  coherence.  The  assessment 
draws on desk research, stakeholder and public consultations, and data collected from annual 
EIF monitoring across the 27 Member States. 

The  EIF  has  certainly  helped  to  inspire  European  public  administrations  at  all  levels  to 
design  and  deliver  digital  public  services,  as  well  as  providing  guidance  to  public 
administrations on the design and updating of NIFs or digital strategies. 

Figure 3 in Chapter 3 shows that most Member States have drawn up a NIF or a similar national 
strategic  or  policy  document  that  incorporates,  at  least  in  part,  the  EIF’s  recommendations, 
principles, and models for delivering digital services166. 

However, the EIF’s final achievement in helping to provide interoperable, user-centric public 
services in the EU and the development of a European public services ecosystem were limited, 
highlighting the need for more action and practical insights. 

EIF Monitoring mechanism 

The  EIF  monitoring  mechanism,  used  for  the  annual  monitoring  exercise  by  the  National 
Interoperability Framework Observatory167, is one of the Commission’s most important data 
sources for tracking roll-out of the EIF across the Member States. It provides an overview of 
NIFs  (see  Figure  3  in  Section  3),  which  currently  present  a  positive  picture  of  EIF 
implementation  progress.  However,  there  is  an  important  caveat:  the  EIF  monitoring 
mechanism focuses on the higher level of conceptual take-up of the guidance and commitment 
to the framework’s principles, recommendations, layers, and high-level components. 

The  depth  and  breadth  of  EIF  monitoring  is  limited  by  the  cross-cutting  nature  of 
interoperability, deeply embedded in broader digitalisation efforts, and by the fact that public 
services are implemented and delivered by hundreds, or even thousands, of different authorities 
in some countries. Monitoring168 is mainly done by gathering data directly from Member States 
through an online questionnaire and transforming the replies into score values for the various 
KPIs.  The  methodology  for  this  is  established  under  the  guidance  of  the  JRC’s  Composite 
Indicator  Unit.  For  some  few  KPIs  only,  the  data  are  collected  directly  from  other  EU 
observatories that target specific domains or areas overlapping with the EIF, such as DESI, the 
eGovernment Benchmark report, the open data portal, or the Accessibility Directive. 

165 

166 

167 

168 

https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-
observatory/solution/eif-toolbox/eif-toolbox. 
https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/national-
interoperability-initiatives. 
https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/eif-
monitoring. 
https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/eif-
monitoring. 

24 

 
                                                           
Similarly, the overview of NIFs captures only the conceptual commitment, while stakeholders’ 
expectations are focused on the implementation level – implementing interoperability guidance 
at  the  service  level.  This  is  where  a  gap  is  apparent:  between  the  conceptual  level,  where 
progress  is  generally  positive,  and  practical  implementation,  where  there  is  room  for 
improvement in specific areas and where additional efforts are deemed necessary. 

Evaluation of EIF’s main components 

The EIF’s main components – the conceptual model, the layered interoperability model and 
the recommendations – are all useful, but are perceived as difficult to implement. As a result, 
more assistance appears to be needed for Member States in the form of guidance and systematic 
steps toward implementing the EIF’s recommendations. The risk is that the lack of granularity 
will allow for multiple interpretations. 

There is undoubtedly a divide between acceptance of the EIF’s conceptual aspects and the need 
for more practical guidance in implementing interoperability requirements. There is also 
a need to raise awareness, particularly at local level. National public authorities are relatively 
confident in the EIF’s ability to provide public administrations with guidance on the design and 
updating of NIFs or strategies. However, they report that the framework’s voluntary nature and 
the  scope  of  its  recommendations  restrict  its  ability  to  foster  cross-border  and  cross-sector 
interoperability or take technological developments into account. This limits its contribution to 
the establishment of the digital single market. 

Two factors can explain the limited success in achieving the objectives. 

1.  The EIF’s objectives are extremely broad and the recommendations usually high-
level. Further clarity and guidance would be required to ensure that they are 
implemented by Member States. 

2.  In some sectors, a lack of collaboration between the private and public sectors 

impedes the growth of synergies. 

To  begin,  the  EIF’s  principles  are  applicable  to  the  development  of  interoperable  digital 
public  services  in  general.  On  the  plus  side,  the  principles  of  openness,  technology 
neutrality, and data portability, as well as the reusability of IT solutions, transparency, 
and  administrative  simplification  are  deemed  particularly  critical,  based  on  input  from 
both  targeted  and  open  consultations,  and  are  crucial  for  enhancing  interoperable  digital 
services. 

Indeed, the EIF monitoring mechanism’s findings suggest that progress has been made toward 
implementing several of the principles identified as beneficial by stakeholders consulted. Most 
Member States are particularly aligned with the EIF in areas such as principle 2: ‘Openness’, 
principle  8:  ‘Security  and  privacy’  and  principle  10:  ‘Administrative  simplicity’  (see 
Annex VI.2). 

However, several concepts are either too abstract (information preservation, effectiveness 
and efficiency assessment, and subsidiarity and proportionality), or they require additional 
clarification  and  practical  focus  to  maximise  their  usefulness  (transparency,  technological 
neutrality and user-centricity). Further guidance is needed for a number of principles (including 
the  assessment  of  effectiveness  and  efficiency,  multilingualism,  and  subsidiarity  and 
proportionality), as demonstrated by monitoring mechanism finding that better implementation 
is called for at Member State level. 

25 

 
 
There  are  also  principles  from  the  EIF  that  require  further  development169,  such  as 
multilingualism,  information  preservation,  effectiveness  and  efficiency  assessment,  and 
subsidiarity and proportionality. This feedback is also corroborated by the EIF monitoring 
mechanism’s results, particularly in relation to principle 1: ‘Subsidiarity and proportionality’, 
principle  9:  ‘Multilingualism’,  principle  12:  ‘Effectiveness  and  efficiency  assessment’,  and 
principle 7: ‘Inclusion and accessibility.’ Numerous countries have scores on the scale’s lower 
limit, (1 or 2), indicating the need for additional efforts to comply with the EIF’s principles and 
recommendations in their particular areas (see Annex VI.2). 

While  layered  interoperability  and  the  conceptual  model  are  deemed  beneficial  for 
enhancing  interoperable  digital  public  services  and  are  adequately  addressed  by  the  EIF, 
additional  guidance  could  make  them  more  actionable,  as  confirmed  by  both  consulted 
stakeholders and EIF monitoring mechanism findings. 

Layered interoperability could be further enhanced by applying the recommendations on the 
governance,  organisational,  and  legal  levels  of  interoperability.  The  evaluation  study170 
demonstrates  that  greater  emphasis  should  be  placed  on  the  non-technical  aspects  of 
interoperability,  acknowledging  that  interoperability  is  a  multifaceted  notion,  not  limited  to 
technical difficulties. Certain features of the semantic layer could also be addressed, such as the 
low rate of semantic adoption due to a lack of awareness or knowledge about the services end 
consumers. 

Moreover,  it  emerged  from  the  evaluation  study  that  some  stakeholders  judged  the 
organisational and legal layers to be incomplete and under-implemented. The organisational 
layer, in particular, requires clarification because it is believed to be too  basic for practical 
applications in its current state. Finally, the layered model could be improved by elaborating 
on the interactions between the technological, organisational, and legal layers. 

The monitoring mechanism’s findings corroborate these observations. The scorecard for the 
EIF’s  layered  model  of  interoperability  demonstrates  that  Member  States  have  made 
significant  efforts to  implement  the EIF  and its recommendations in  this  area, most notably 
technological interoperability at EU level. However, the scoreboard identifies a need to improve 
implementation  of  suggestions  relating  to  the  governance  levels  of  interoperability, 
organisational interoperability, and legal interoperability. 

Stakeholders  contacted  about  the  conceptual  model  confirm  that  it  has  helped  to  enhance 
interoperable  digital  public  services  to  some  extent.  However,  the  conceptual  paradigm  is 
regarded as overly abstract and inoperative. As a result, Member States may interpret and hence 
implement  it  inconsistently,  jeopardising  the  goal  of  interoperability.  As  such,  experts  and 
representatives  from  public  administrations  call  for  the  establishment  of  an  end-to-end 
operational  approach  to  assist  Member  States  in  implementing  interoperability  by  design  in 
order to develop a comprehensive interoperability approach. 

As  regards  the  EIF’s  coherence,  the  EIF’s  principles,  layered  interoperability  model,  and 
conceptual model for integrated public service delivery are generally synergic. To improve the 
framework’s overall cohesiveness, the conceptual model might be more closely related to the 

169 

170 

Interestingly,  the  scores  given  to  these  principles  show  disparities  between  the  consulted  groups  of 
stakeholders. Overall, EU and non-EU citizens are relatively unenthusiastic about the usefulness of these 
principles in enhancing interoperable digital public services (average scores of 3.47, 3.24, 3.29 and 3.13 
out of 5 respectively, based on 17 responses to the targeted and the public consultations). By contrast, 
these  principles  are  generally  deemed  more  useful  by  companies  and  business  associations  (average 
scores of 3.80, 3.53, 3.73 and 3.38 out of 5 respectively, based on 15 responses to the targeted and the 
public consultations), and by experts and academia (average scores of 4.00, 3.71, 3.36 and 3.64 out of 5 
respectively, based on 14 responses to the targeted and the public consultations). 
CEPS  EIF  final  evaluation  final  study,  available  at  https://op.europa.eu/en/publication-detail/-
/publication/29d694d4-4696-11ec-89db-01aa75ed71a1/language-en/format-PDF/source-search. 

26 

 
                                                           
other two components. According to the research, the strength of the national frameworks is 
contingent  on  an  in-depth  understanding  and  design  of  these  three  EIF  components.  The 
components’  interdependence  is  exemplified  by  the  connection  between  the  semantic 
layer, the ideals of openness and transparency, and the section of the conceptual model on 
openness. The principles and layered interoperability model are extremely synergistic, while 
enhancements to the conceptual model may result in more synergies. 

The recommendations made throughout the EIF are largely self-reinforcing. Nonetheless, there 
are several instances where EIF guidelines overlap, at  least thematically. While the overlaps 
may not cause problems, by better organising the recommendations, the important messages 
could  be  clarified,  making  the  advice  more  actionable.  On  a  more  detailed  level,  the  EIF 
monitoring mechanism’s findings suggest that various enhancements might be made to further 
support  the  conceptual  model’s  implementation  in  some  Member  States,  notably,  Germany, 
Ireland and Romania (see Figure 6 in Chapter 3). 

Almost all countries do admirably in the field of ‘internal information sources and services’ 
(with scores of 4 almost everywhere), but efforts in some countries like Germany, Ireland and 
Poland would need to be stepped up (as signalled by scores of 1; see Annex VI.2). Additionally, 
progress  on  open  data  is  encouraging,  with  all  countries  rating  towards  the  top  of  the  scale 
(scores of 3 and 4). Improvements could be made to cataloguing, external information sources 
and services, as well as security and privacy. 

External drivers 

External  drivers  such  as  technological  advancements  and  national  and  cross-sectoral 
interoperability initiatives have contributed to some extent to the EIF’s observed achievements. 
The main constraining factors, to be minimised, are seen as limited resources and institutional 
complexity. 

According to respondents in targeted consultations, external drivers do, on average, contribute 
to the framework’s implementation (see Figure 7). 

27 

 
 
Figure 7: Extent to which the following external factors contribute to implementation of 
the EIF (breakdown by group of stakeholders; average score and number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Note: Averages do not include respondents answering ‘don’t know/no opinion’ (DK/NO). 
Source: Study supporting the evaluation of the implementation of the EIF 

However, the low average score assigned to cross-sectoral projects by national public agencies 
is particularly noteworthy. This score indicates primarily a lack of coordination between 
the  public  administrations  of  the  various  Member  States.  The  COVID-19  pandemic  has 
highlighted the critical  nature of interoperable solutions.  Nonetheless,  several  expert studies 
determined  that  cross-sectoral  initiatives  can  play  a  significant  catalytic  role  in  advancing 
interoperability adoption. In this context, smart city efforts are instructive. There is significant 
opportunity to enhance such projects by ensuring that interoperability is a topic of discussion 
and, more significantly, a requirement, thereby reinforcing the EIF’s fundamental messages. 
There are considerable opportunities to achieve economies of scale for smart cities by reusing 
tools, data, and applications171. 

Some  unfavourable  external  influences  have  a  similar  impact  on  the  EIF  according  to  the 
respondents in the EIF evaluation consultations (see Figure 8). These are described below. 

1.  Scarce  resources.  budgetary  limits,  organisational  difficulties,  and  a  skills 
shortage. These obstacles stem mostly from a lack of human and financial resources, 
but also from the incompatibility of national databases172. 

171 

172 

07/04/2021, 

Joinup.  Available 

A new EIF for Smart Cities/Communities (EIF4SCC) is currently under development at EU level. See 
‘Proposed European Interoperability Framework for Smart Cities & Communities (EIF4SCC)’, published 
on: 
https://joinup.ec.europa.eu/collection/nifo-national-
interoperability-framework-observatory/news/eif4scc-smart-cities-communities. 
European Commission, How do Member States and Norway ensure accurate, timely and interoperable 
flash  #18  –  2021.  Available  at: 
data  management 
https://www.emn.lv/wp-content/uploads/2021/07/EMN-Flash_data-management-in-asylum-
procedure_final_24062021.pdf  

the  asylum  procedure?,  EMN 

at: 

in 

28 

 
 
                                                           
2.  Institutional complexity173. This is undoubtedly the most challenging factor to solve, 
as it may be necessary to adjust the legislative framework in order to execute the EIF 
successfully, and institutional changes are slow to materialise. A new analysis on the 
influence of open source software on technological independence, competitiveness, and 
innovation in the EU economy identifies a fragmentation of duties in the field of digital 
policy,  impeding  collaboration  on  digital  law-making174.  This  issue  has  a  range  of 
consequences  for  different  countries,  as  some  countries’  legal  systems  are  more 
amenable  to  reform  than  others.  Interoperability  and  institutional  complexity  are 
inextricably linked; this issue can be partially resolved by enhancing interoperability. 
3.  Limited awareness is another barrier that is deemed to jeopardise the EIF’s take-up, 
particularly at regional and municipal level and in the design of digital public services. 

Figure  8:  Extent  to  which  the  following  external  factors  are  jeopardising  the 
implementation  of  the  EIF  (breakdown  by  group  of  stakeholders;  average  score  and 
number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Note: Averages do not include respondents answering ‘don’t know/no opinion’ (DK/NO). 
Source: Study supporting the evaluation of the implementation of the EIF. 

EIF beneficial effects 

The framework has had some beneficial effects. The EIF has to a certain extent contributed to 
the improvement of the quality of public administration services and to the promotion of free 
movement of products, services, capital, and labour across Member States. A strong alignment 

173 

174 

Institutional complexity captures the difficulty and slowness of change in the legal system, as well as the 
complexity  arising  from  a  fragmented  and  multi-layered  distribution  of  competences  between  the 
different levels of public administration. 
Fraunhofer  ISI,  OpenForum  Europe  (2021),  The  impact  of  Open  Source  Software  and  Hardware  on 
technological independence, competitiveness and innovation in the EU economy, p. 307-308. 

29 

 
 
 
                                                           
between  the  NIF  and  the  EIF  would  surely  boost  the  direct  and  indirect  benefits  for 
stakeholders  (i.e.  residents,  enterprises,  and  public  administrations)  through  better  digital 
public services, particularly at cross-border level. 

As  explained  above,  some  Member  States  have  created  an  NIF  based  on  the  EIF,  such  as 
Belgium, Estonia, Greece, Spain, Cyprus, Luxembourg, Malta, Netherlands, Austria, Portugal, 
Romania and Slovenia. 
Spain  for  example  has  passed  an  EIF-based  NIF  into  law  through  a  Royal  Decree175.  The 
purpose  is  to  ensure  that  the  systems  and  applications  used  by  public  administrations  are 
technically,  semantically  and  organisationally  interoperable  enough  to  enable  rights  to  be 
exercised and duties fulfilled through electronic access to public services, while also enhancing 
effectiveness and efficiency. Thanks to the NIF, sound governance has been stablished across 
various  administrative  levels,  and  a  set  of  supporting  common  services  (eID,  eSignature, 
eDelivery, eInvoicing…), infrastructures and technical specifications176 have been devised for 
common and more efficient digital service provisioning. Thanks to this, public services were 
not disrupted during Spain’s first COVID-19 lock-down (the most severe in Europe along with 
Italy’s). The Spanish Chief Information Officer (CIO) highlighted common infrastructure and 
services as being among the key elements for the smooth delivery of public services during the 
first wave of the COVID-19 crisis. See below further details. In addition, thanks to Spain’s EIF-
based  NIF,  data  on  taxation,  social  security,  justice  etc.  data  can  be  exchanged  through  a 
common interoperable platform177, used by all digital services in the country on a daily basis, 
reducing the administrative burden on people and businesses. 
Likewise, on the 1 March 2019, Luxembourg’s Government Council adopted an NIF178 which 
guides public sector bodies within its remit on how to attain a higher level of interoperability. 
This was Luxembourg’s first NIF bringing together in the same text all the elements needed for 
a coherent and clearly structured base for public sector interoperability. The NIF builds on the 
2017 EIF whilst taking account of context and specific national needs. The NIF thus complies 
with  the  requirements  set  out  in  the  Commission’s  EIF  and  enables  the  Luxembourg 
government to make progress towards greater interoperability. 
The Slovenian framework provides for a website179 for publishing public sector interoperability 
solutions and products. It connects a catalogue of interoperability solutions with best practices 
for reusing its content. 

Some  NIFs  set  up  the  necessary  governance  to  work  together  in  a  coordinated  matter  on 
interoperability-related  topics  supporting  common  goals  and  priorities  laid  down  by  the 
national digitisation policies, as well as relevant EU legislation like eIDAS, the Single Digital 
Gateway  Regulation  and  the  Open  Data  Directive.  Such  governance  arrangements  typically 
bring on board all ministries at central level, the regions and even the local government in the 
design and implementation of common processes, IT systems and technical specifications. 

175 
176

177

178 
179 

https://www.boe.es/buscar/act.php?id=BOE-A-2010-1331. 

  https://administracionelectronica.gob.es/pae_Home/pae_Estrategias/pae_Interoperabilidad_Inicio/pae
_Normas_tecnicas_de_interoperabilidad.html. 

  https://administracionelectronica.gob.es/pae_Home/pae_Estrategias/Racionaliza_y_Comparte/element
os_comunes/Intermediacion_de_datos.html. 
https://digital.gouvernement.lu/en/dossiers/2019/NIF-2019.html. 
https://nio.gov.si/nio/vstopna.nio?lang=en. 

30 

 
                                                           
The report on public administrations’ digital responses to COVID-19 in Europe180 shows that 
five countries (Czechia, Denmark, Estonia, Spain and the Netherlands) reported no disruption 
to their public services. This appears to be in line with what was outlined in the 2020 edition of 
the  European  Commission  eGovernment  benchmark181  as  four  of  these  five  countries  are 
considered to have highly digitalised public services, scoring well above the European average. 
The delivery of public services in four of these five countries is also considered to be highly 
interoperable  with  scores  above  the  European  average  on  the  EIF.  This  is  also  further 
corroborated by the Danish, Estonian Spanish and Dutch national  CIOs, who stated that the 
already high level of digitalisation in public services delivery helped to ensure the continuity 
of these services throughout the first wave of the COVID-19 crisis. 

An example of an EU solution which puts these generic interoperability guidelines into practice 
is the EU interoperability gateway for COVID-19 contact-tracing and vaccines. A national/bi-
national  solution  is  Estonia  and  Finland’s  X-Road182  platform,  linking  all  public  digital 
services. 

Estonian experience demonstrates that the EIF aided in the development of a wide variety of 
digital and cross-border public services relating to company and property registration (for more 
details, see also the above section covering the impacts of external factors). Additionally, the 
EIF  contributes  to  the  free  movement  of  commodities,  services,  capital,  and  labour 
throughout  Member  States.  Experts  and  academia,  as  well  as  businesses  and  business 
organisations,  are  generally  more  optimistic  about  the  framework’s  potential  benefits  for 
research, development, and innovation. 

With regard to implementation of the EIF, IMAPS is an online survey developed under the ISA² 
programme,  helping  public  service  owners  evaluate,  consider  and  improve all  key 
interoperability  aspects of  their  digital  public  service  (legal,  semantic,  organisational,  or 
technical), in compliance with the EIF. Some public bodies have used the tool on a voluntary 
basis  to  make  their  administrative  procedures  fit  for  purpose  for  the  single  digital  gateway, 
assessing the level of digital maturity of a public service that needs to be provided online by 
2023  and  accessible  via  the  gateway.  Examples  include  online  submission  of  corporate  tax 
declarations  or  applications  for  a  European  health  insurance  card  (EHIC).  Some  countries 
(Belgium, Czechia and Slovenia) and the city of Valencia have used the survey to detect areas 
for improvement and to make digital services more mature from the EIF perspective. 

IMAPS  publishes  an  annual  report  on  the  interoperability  maturity  of  European  digital 
public  services  assessed  using  IMAPS.  The  latest  IMAPS  report  (2020)  reveals  that  all 
assessed  digital  public  services  are  on  average  below  the  essential  interoperability  level 
(IMAPS maturity level 3). 

However, this EIF evaluation shows the limitations on providing a deeper and wider analysis 
of  the  level  of  EIF  implementation  across  the  hundreds  and  thousands  of  bodies  in  the  27 
Member  States.  To  do  systematically,  sound  interoperability  governance  would  be  needed, 
involving the main national authorities dealing with digitalisation, in close coordination with 
the  regions,  local  authorities  and  the  European  Commission.  A  common  process  for  apply 
IMAPS  would  be  beneficial,  or  a  similar  tool  aligned  with  main  interoperability  and 
digitalisation goals stemming from the national and |EU strategies and policies. The aim would 
be to ensure that any interoperability maturity level assessment helped to monitor, rationalise 

180 

181 

182 

https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/report-
public-administrations-digital-response-covid-19-europe. 
The European average for the eGovernment benchmark 2020 is 72%. The four countries scoring above 
the European average are: the Netherlands (78%), Spain (78%) Denmark (84%) and Estonia (92%). 
https://x-road.global/. 

31 

 
                                                           
and steer all implementation plans and investments, and find gaps, overlaps, inefficiencies and 
ways to bridge them. 

The EIF can serve as a  genuine catalyst for public-government dialogue. It is possible to 
quantify both the impact of adopting an EIF-based NIF and the maturity level of digital, online 
public  services  to  gain  additional  insight  into  the  range  of  possible  EIF  impacts  and 
improvements in digital public services. 

An  economic  analysis  demonstrates  that  an  EIF-based  NIF,  or  an  NIF  with  EIF-based 
components,  is  more  conducive  to  public-government  interaction  than  no  EIF-based  NIF. 
Additionally, a 10% improvement in the quality of online public services would encourage an 
additional  4.3 million  to  5 million  EU  residents  to  connect  with  their  governments  via  the 
internet each year. 

Figure 9 and Figure 10 respectively report the results obtained under two different regressions: 

1.  a ‘complete’ EIF-based NIF compared to an ‘incomplete’ EIF-based NIF or no EIF-based 

NIF; and 

2.  a ‘complete’ or ‘incomplete’ EIF-based NIF compared no EIF-based NIF. 

Figure 9: Average effect of having a ‘complete’ NIF compared with an ‘incomplete’ NIF 
or  no  EIF-based  NIF,  on  the  level  of  people’s  online  interaction  with  public 
administrations 

Note: The results (i.e. when NIF ‘complete’=1 and NIF ‘incomplete’=0) compare countries having a fully EIF-
based  NIF  with  countries  having  a  partially  EIF-based  NIF  or  no  EIF-based  NIF  at  all.  The  results  assume 
linearity over time, i.e. the average effect of having an NIF on online interaction with public administrations is 
assumed to be constant over time. 
Source: Study supporting the evaluation of the implementation of the EIF 

32 

 
 
Figure 10: Average effect of having a ‘complete’ or ‘incomplete’ EIF-based NIF compared 
with  no  EIF-based  NIF,  on  the  level  of  people’s  online  interaction  with  public 
administrations 

Note:  The  results  (i.e.  when  NIF  ‘complete’=1  and  NIF  ‘incomplete’=1)  compare  countries  having  a  fully  or 
partially  EIF-based  NIF  with  countries  having  no  EIF-based  NIF  or  similar  strategies.  The  results  assume 
linearity over time, i.e. the average effect of having an NIF on online interaction is assumed to be constant over 
time. 
Source: Study supporting the evaluation of the implementation of the EIF. 

This  analysis  is  conducted  at  the  macroeconomic  level  and  is  based  on  binary  variables 
indicating whether or not the countries analysed have approved an NIF or comparable national 
strategic documents based entirely or partially on the EIF. As a result, the analysis  does not 
account for the effects of the practical implementation of specific interoperability standards, 
but rather for the effects of strategic advice in  general. The analysis determines the average 
impact throughout the sample examined. The magnitude of the effects may vary by country. 

Additionally, the findings confirm the association between increased service quality and the 
fact that improved eGovernment services enable people to interact more with their government 
via the internet. More precisely, a 10 % increase in the eGovernment index results in an increase 
of 1.5-1.7 % in people’s use of the internet to connect with public authorities while all other 
variables remain constant. 

On the efficiency front, the data indicate that benefits may outweigh costs in some cases. 
While public administrations bear the costs of implementing the EIF, streamlining procedures 
and  facilitating  data  exchange  are  seen  as  benefiting  a  broader  range  of  stakeholders  by 
improving interactions (i.e. saving time and money) between public administrations on the one 
hand and between public administrations and users of public services on the other. 

Most of the EIF’s benefits stem from two factors. 

1.  Affirming  the  importance  of  administrative  simplicity.  By  streamlining  and 
simplifying  the  digital  delivery  of  public  services,  EIF  implementation  costs  were 
negligible in comparison to the cost savings associated with interactions between public 
administrations, and between administrations and their constituents. 

2.  Encouraging data reuse. The EIF’s goal is to assist public administrations in avoiding 
duplication  of  efforts  and  effectively  leveraging  existing  resources  and  information, 
resulting in better service quality. 

33 

 
 
Concerning expenditure by public administrations in implementing the EIF, national and EU 
public  administrations  gave  consistent  responses.  Supporting  the  framework’s  roll-out  is 
regarded to be reasonably inexpensive, at both national and EU level. In the Netherlands, for 
example, an independent and external advisory body (ACTAL, the Dutch advisory board on 
regulatory  burden)  has  been  set  up  to  assist  the  government  and  parliament  on  minimising 
regulatory burdens. 

Additionally, short-term advantages must be weighed against long-term benefits. While initial 
investments can be costly, as procedures are optimised, operating costs tend to decrease over 
time. 

The deployment of NIFs inspired by the EIF clearly also resulted in various benefits. 

1.  The  EIF  facilitates  synergies  in  the  implementation  and  design  of  new  services, 
contributing to the reuse of a diverse range of software programmes collected in the 
Italian reuse catalogue, for example. 

2.  Interoperability has resulted in cost savings associated with various ICT projects. 
Centralised authorisation of ICT projects is estimated to have saved Czechia more than 
EUR 39 billion in public investment in 2019. Additionally, some ICT projects filed by 
public  agencies  were  rejected  due  to  their  failure  to  meet  interoperability  criteria. 
Notably,  public  administrations  were  able  to  rethink  and  adapt  their  project  ideas, 
resulting in further savings, by depending on the sharing and reuse of existing digital 
solutions and shared government services. 

While  voluntary  adoption  of  the  EIF  and  its  recommendations  has  undoubtedly  resulted  in 
benefits  recognised  by  public  administrations,  more  should  be  done  to  foster  an  EU-wide 
strategy that is really integrated. However, a comprehensive evaluation is impossible to make 
due to the framework’s broad scope and non-binding character. 

Cost-benefit analysis 

In measuring and comparing the expense of adopting the EIF and applying these costs to NIFs, 
many difficulties arise due to the voluntary nature of the framework. These difficulties are due 
to  the  varying  cost  of  digitalisation  across  different  public  sector  bodies  and  the  different 
approaches  taken  by  countries  in  implementing  the  EIF  and  the  varying  degrees  of 
digitalisation.  Comparisons  between  Member  States  are  complicated,  as  costs  may  vary  in 
terms of coverage and over time, even when cost differences are clear (see Box 1). 

Moreover, interoperability is an underlying support function for digital transformation; for this 
reason, many of the assessments conducted by Member States are based on digital government 
transformation  costs.  Defining  specific  interoperability  costs  requires  complex  research  and 
analysis. The consultation with the stakeholders nevertheless provided some reference points 
for  the  costs  linked  to  producing  a  strategy  or  an  NIF  document  and  bound  mainly  to 
consultation services. 

The  complexity  of  EIF  implementation  make  it  hard  for  Member  States  to  estimate  the 
associated costs. Many different authorities and public services are involved and it is difficult 
to isolate EIF implementation from the overall digital transformation processes. This is also the 
case for the implementation of EU policies. The team working on the single digital gateway 
conducted  a  study183  to  estimate  the  cost  of  implementing  the  once-only  principle  where 
interoperability plays a major role. Its conclusions, summarised below, show that the costs were 
hard to quantify. 

183 

Readiness  of  MS  to  connect  and  exchange  data  in  accordance  with  Article 14  of  the  Single  Digital 
Gateway Regulation and the once-only principle infrastructure. 

34 

 
                                                           
1.  The scarcity and low comparability of data makes it difficult to assess the costs and benefits 
of incorporating the EIF into national frameworks. The voluntary nature of the framework 
is reflected in its uneven and ad hoc uptake across the EU, limiting the availability of data 
and making comparison between countries difficult. Furthermore, implementation of the 
EIF  has  not  followed  the  same  timeline  in  the  different  countries.  To  overcome  this 
limitation,  the study team  contacted the national  public authorities that responded to  the 
consultation activities and indicated their availability to participate in a follow-up interview, 
to gain insight on the time spent (in terms of person-days) on the incorporating the EIF, or 
part of it, into their national framework. 

2.  The overarching scope of the EIF hinders the assessment of the direct costs and benefits 
stemming from its implementation. Costs and benefits deriving from the EIF do not relate 
exclusively to implementation of the framework; costs and benefits also depend Member 
States’  levels  of  digitalisation,  the  degree  of  centralisation  and  varying  organisational 
structures  involved  in  delivering  public  services.  The  costs  related  to  interoperability 
initiatives  are  borne  at  different  levels  of  administration  in  different  countries.  The  cost 
discrepancies are due to the following factors. 
  The extent to which information technology has been digitised. Member States that 
have already deployed interoperable solutions in line with the EIF can quickly build on 
pre-existing  solutions,  resulting  in  reduced  costs  and  responsibilities.  Additionally, 
governments that rely on open source solutions mitigate the risk of lock-in. As a result, 
solutions can be more easily transferred or altered, resulting in cost savings. 

  Catch-up effects. Because public services have differing degrees of centralisation and 
organisational  structures,  the  costs  of  interoperability  projects  are  borne  at  varying 
levels  of  administration  in  different  countries,  as  stated  in  the  expert  assessments 
accompanying  this  study.  In  Sweden,  for  example,  costs  are  borne  primarily  by 
individual  agencies.  In  comparison,  the  role  of  national  government  initiatives  and 
support  is  substantially  stronger  in  neighbouring  Norway  and  Denmark.  It  is  often 
difficult to assess the cost of ICT projects since they can contain significant hidden costs 
such  as  the  involvement  of  existing  workers  in  modifications  required  for  the  new 
operation design, such as training, or time spent on transitional operations. 

These constraints underline the need to identify common criteria so that Member States’ costs 
can be assessed consistently. 

Furthermore, incorporating EIF guidance into the national framework is likely to have delivered 
different benefits in the different Member States (see Box 2). 

Box 1: Member States’ feedback on EIF implementation costs 

Several  representatives  of  public  authorities  in  the  Member  States  provided  additional 
feedback on the number of person-days required to take up the EIF in the development of 
national  frameworks  or  strategies  for  interoperability.  The  number  of  person-days  is 
translated into a costing based on average labour costs in all sectors (based on NACE rev.2 
activity)184 in the respective countries. In line with the better regulation toolbox, an extra 
25% is added to labour costs to account for overheads (e.g. rent costs, utilities, insurance, 
office equipment and supplies, and travel costs)185.. 

On  that  basis,  time  and  costs  can  amount  to  150-205  person-days  (equivalent  to 
approximately EUR 24 000 –EUR 30 000) for creating updated drafts of existing national 

184 

185 

Eurostat  (reference  year:  2019),  Labour  cost  levels  by  NACE  Rev.  2  activity,  available  at: 
https://appsso.eurostat.ec.europa.eu/nui/show.do?dataset=lc_lci_lev&lang=en. 
European Commission (2017), Better Regulation Toolbox, p. 496. 

35 

 
 
                                                           
documents  and  strategies  based  on  the  EIF,  and  550  person-days  (equivalent  to 
approximately EUR 169 000) for designing EIF-based national documents and strategies 
from  scratch.  Costs  also  differ  according  to  how  fully  the  EIF  has  already  been 
implemented. Countries that have not implemented the EIF before may incur more up-front 
costs to ‘catch up’ with the other Member States. 

Box 2: Cost-saving benefits of interoperability 

Limited interoperability impedes the digitalisation of the public sector and makes it difficult 
to achieve benefits such as time and cost savings. Addressing this problem and enabling 
digital  transformation  in  public  services  could  save  result  in  significant  savings.  Several 
studies have carried out cost-savings analyses in different countries or geographical areas 
following  the  implementation  of  different  solutions.  Table  1  reports  the  findings  on 
estimated cost savings collected during the desk research. 

Table 1: Cost-savings estimations 

 ‘ 

Geographical 
coverage 

Implemented solutions 

Cost savings 

Belgium186 

Digital government solutions 

In 2016: 

UK187 

Netherlands 
and 
Belgium188 

Spain189 

1.  EUR 4.6 million 
businesses 

for 

2.  EUR 28 million  for  the 

public 

GBP 1.7 billion  to  GBP 1.8 billion 
every year 

In 2017: EUR 163 million  

EUR 100 million a year  

Digital 
implementation 
principles: 
public services 

strategies 
of 

and 
key 
digital-by-default 

Digital 
implementation 
principles: once-only principle 

strategies 
of 

and 
key 

strategies 
of 

Digital 
and 
implementation 
key 
e-administrations, 
principles: 
and 
principle, 
once-only 
interoperability platforms 

EUR 22 billion  between  2008  and 
2011, with: 

3.  60% 

linked 
administration 

to 

e-

186 

187 

188 

189 

see:  https://gds.blog.gov.uk/2015/10/23/how-digital-and-technology-

For further details please see: Digital Dashboard Belgium, Evolution of cumulative annual cost reduction 
per target group 2016, https://digitaldashboard.belgium.be/en. 
For  more  details  please 
transformation-saved-1-7 bn-last-year/. 
For further details please see: Cave J. et al (2017), EU-wide digital once-only principle for citizens and 
businesses, 
https://ec.europa.eu/digital-single-market/en/news/eu-wide-digital-once-only-principle-
citizens-and-businesses-policy-options-and-their-impacts. 
Gallo,  C.,  Giove,  M.,  Millard,  J.,  Thaarup,  R.  (2014),  Study  on  eGovernment  and  the  Reduction  of 
at: 
Administrative 
https://ec.europa.eu/futurium/en/system/files/ged/finalreportstudyonegovernmentandthereductionofadm
inistrativeburden.pdf. 

Commission, 

Available 

European 

Burden, 

p. 35. 

36 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                           
EU190 

strategies 
of 

and 
Digital 
implementation 
key 
principles:  once-only  principle 
and  digital-by-default  public 
services 

4.  20%  to  the  implementation 
of  the  once-only  principle 
and 
5.  20% 

interoperability 

to 
platforms 

In 2014: 

a 

1.  EUR 5 billion 

year 
linked  to  the  once-only-
principle; and 
2.  EUR 10 billion 

year 
linked  to  the  digital-by-
default principle 

a 

Source: Study supporting the evaluation of the implementation of the EIF based on official reports. 

The expense of translating the EIF, or parts of it, into national frameworks is difficult to assess 
and ranges from EUR 24 000 to EUR 169 000 (based on the estimates of time translated into 
costs provided by representatives of several public authorities in the Member States). 

The impacts of a public sector interoperability policy are more difficult to identify and quantify 
than those of many other EU initiatives. A report by the JRC191 analysed the economic impact 
of public sector interoperability overall. It found that, due to its very nature as an enabling factor 
across many policy fields, the impact of interoperability is hard to isolate from other aspects of 
a  policy.  Interoperability  in  its  many  forms  is  an  important  enabler  but  it  is  interoperable 
policies and public services that deliver benefits (and consume resources).  

Nevertheless, the evidence is quite conclusive. For instance, in terms of economic impact, the 
study by the JRC found that, at a conservative estimate, the public could save up to 24 million 
hours a year through improved interoperability. This would represent monetary savings in the 
order of EUR 543 million annually. For businesses, improved interoperability would lead to 
time  savings  of  30 billion  hours.  In  monetary  terms,  this  translates  into  savings  of 
EUR 568 billion annually. Improvements in public sector performance due to interoperability 
could lead to an increase of 0.4% in GDP. Overall, the potential positive benefits derived from 
increased  interoperability  for  the  public  sector  in  the  EU  are  considerable.  For  instance, 
converting  these  estimates  into  monetary  terms,  a  1 %  improvement  in  the  indicators  used, 
through greater interoperability, would boost EU GDP by EUR 21 billion - EUR 56 billion. 

Future  monitoring  should  focus  on  reusing  developed  technologies  and  providing  better 
estimates of costs. In this regard, a set of EIF-specific criteria could make it easier to compare 
costs across the EU (including, for instance, the daily cost of work by the team tasked  with 
implementing  the  EIF  at  national  level,  the  cost  of  partial  implementation  of  the  EIF,  the 
number of consultations between the European Commission and the Member States when the 
EIF was agreed on and introduced, the time spent by annual reporting under the NIFO initiative, 
etc.). 

The  present  evaluation  study  also  shows  the  current  limits  to  proper  correlational  analysis 
between digitalisation and interoperability, and the difficulties in decoupling the two effects in 

190 
191 

Ibid., p. VI. 
Ulrich,  P.,  Duch  Brown,  N.,  Kotsev,  A.,  Minghini,  M.,  Hernandez  Quiros,  L.,  Boguslawski,  R.  and 
Pignatelli,  F.  (2022),  Quantifying  the  Benefits  of  Location  Interoperability  in  the  European  Union, 
EUR 31004  EN,  Publications  Office  of  the  European  Union,  Luxembourg,  2022,  ISBN  978-92-76-
48846-0, doi: https://doi.org/10.2760/72064, JRC127330. 

37 

 
                                                           
the estimates. Future work is needed to come up with sound methodologies for estimating the 
cost of interoperability, as opposed to the costs of digital transformation in a broader setting. 

Coherence 

The  EIF  is  found  to  be  consistent  with  initiatives  such  as  the  Single  Digital  Gateway 
Regulation,  the  digital  single  market  strategy,  and  the  data  strategy.  Both  strategies 
highlight  the  EIF’s  role  in  digitalising  EU  public  services.  In  addition,  the  2017  EIF  and 
measures to monitor its implementation reflected measures outlined in the eGovernment action 
plan 2016-2020. Moreover, the 2015 DSM strategy192 called for revision of the EIF to better 
complement the strategy’s goals of digitalising public services in the EU. The 2017 EIF heeded 
this call. The interoperability action plan193 included key steps to be taken to further support 
the  vision  of  digital  and  interoperable  public  services.  The  ISA²  final  evaluation  study 
highlights the programme’s contributions (as the main EIF implementation instrument). 

Moreover, the adoption of a revised EIF Communication in 2017 helped with the successful 
completion of two actions in the e-government action plan for 2016-2020, namely actions 4 
and 6. It was developed as part of the NIFO initiative, which is funded by the ISA² programme 
to  monitor  the  framework’s  implementation  in  the  Member  States.  The  NIFO  action  also 
included an EIF toolbox. The toolbox maps possible interoperability solutions produced as part 
of  the  ISA²  and  CEF  programmes,  which  public  administrations  can  use  to  implement  EIF 
recommendations.  The  toolbox  also  links  the  EIF’s  suggestions  to  their  accompanying 
solutions. 

Many  of  the  EIF  principles  or  47  recommendations  have  been  targeted  by  other  provisions 
(examples given already include open data, SDG for once-only, privacy and data protection 
(GDPR),  accessibility,  etc.)  This  can  be  seen  from  a  first  mapping  of  digital-relevant  EU 
initiatives  onto  the  EIF  pillars  and  individual  recommendations,  part  of  the  EIF  toolbox 
resources (see Annex VIII). However, some parts of the EIF are not yet specifically targeted by 
the EU regulator, such as interoperability governance or reusability. 

As examples of coherence at EU level, the SDG Regulation mentions a number of ideas and 
fundamental concepts promoted by the EIF, such as the potential for reusing an ISA² solution 
and the core public services vocabulary application profile (CPSV-AP). The Regulation itself 
implements a pan-European technical system using EIF principles and some solutions from the 
interoperability action plan, such as e-delivery and common vocabularies.  

To implement the once-only principle, Article 14 of the Regulation obliges Member States to 
exchange evidence for 21 administrative procedures listed in Annex II to the Regulation. The 
EIF is fundamental to this endeavour. Implementation of harmonised common data models has 
already  started  for  eight  evidence  types,  to  enable  data  to  be  shared  according  to  semantic 
interoperability principles and fully in line with EIF guidance (semantic layer principles). These 
evidence  types  include  the  birth  certificate,  the  absence  of  criminal  records  and  academic 
qualifications. At Member State level, this involves firstly the central state digitalisation units. 
The last mile is taken care of by regional and local governments who host digital services in 
most  cases  and  are  strongly  impacted  by  EU  policies.  Solutions  for  eDelivery  and  for 

192 

193 

COM(2015) 192 final, Communication from the Commission to the European Parliament, the Council, 
the  European  Economic  and  Social  Committee  and  the  Committee  of  the  Regions,  A  Digital  Single 
Market Strategy for Europe, 
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52015DC0192. 
COM(2017) 134 final, Communication from the Commission to the European Parliament, the Council, 
the  European  Economic  and  Social  Committee  and  the  Committee  of  the  Regions,  European 
Interoperability Framework - Implementation Strategy, Annex 1. 

38 

 
 
                                                           
interconnecting authoritative sources of data on people and businesses are also designed in line 
with principles and solutions stemming from the EIF conceptual model for integrated digital 
public service delivery. 

The  data.europa.eu  portal  has  also  been  set-up  using  the  EIF  as  inspiration,  reusing  a  data 
model194  created under the ISA² programme, to catalogue data and link up the open data sets 
from EU institutions, agencies and bodies, as well as European countries national and regional 
catalogues in an interoperable manner, according to the principles of the EIF conceptual model 

Data portability is one of the main driving forces of the EIF. As another example, Article 28 of 
the  data  act  draft  proposal  provides  for  common  specifications  and  core  vocabularies  to 
underpin the interoperability of cross-sector and cross-border data spaces, fully in line with EIF 
principles and recommendations, to ensure interoperable data sharing services and flows. To 
ensure strong interoperability governance, the draft assigns a major role to the European Data 
Innovation  Board,  at  least  for  public  sector  services.  Where  there  are  different  approaches 
between sectors and even Member States on privacy, data protection, responsibilities, common 
use cases, data flows and data specifications, this can result in a lack of legal certainty for the 
sharing of data across domains,  policy areas and borders. The data act  proposal  is  meant  to 
address this lack of legal certainty, in conjunction with EU data policy and data governance. In 
this regard, it is aligned with the legal interoperability concept promoted in the EIF, ensuring 
that organisations operating under different legal frameworks, policies and strategies can work 
together. 

Health is another practical example. Patients typically have medical records that are hosted in 
a  hospital  or  medical  centre  as  a  digital  record.  If  the  person  moves  to  another  location  or 
country, transferring the file to another hospital digitally can be problematic since there are no, 
or few, common formats or solutions widely used in Europe. The solution is to agree on the 
pieces of information for the record, and then on common formats for the records.  

This will be part of the European Health Data Space195. It has to be done in a way that not only 
health  service  providers,  such  as  hospitals,  but  also  other  data  users,  for  instance  research 
institutions conducting medical research, can easily interpret the data and combine it with other 
data sources, such as mobility, socio-economic or environmental. This is one of the reasons 
why  data  spaces  should  be  interoperable  across  themselves,  and  minimum  technical 
requirements are needed. Cooperation and ongoing work in the area of digital health and health 
data use is applying EIF principles and actively seeking interoperability support. 

ISA² solutions also help to implement the EIF and more broadly interoperability initiatives at 
EU and Member State level. 

Other than those mentioned, the EIF aligns with Horizon 2020, the INSPIRE Directive, and 
the smart cities/communities projects. Moreover, the EIF’s overlaps and inconsistencies with 
other projects are restricted. 

More direct references to the EIF could help reinforce the idea that such an EU-level framework 
has broad consequences, and that interoperability is vital for projects wanting to leverage the 
benefits of digital transformation. 

194  https://joinup.ec.europa.eu/collection/semantic-interoperability-community-semic/solution/dcat-application-

profile-data-portals-europe 

195 https://health.ec.europa.eu/ehealth-digital-health-and-care/european-health-data-space_en  

39 

 
                                                           
An examination of rules, directives, decisions, and Commission communications from 2004 to 
2021  demonstrates  that  the  EIF’s  interconnections  with  other  EU  programmes  might  be 
strengthened. We discovered that: 

1.  the EIF has been considered in the preparation of some legislation in areas where 

interoperability is vital; 

2.  the EIF is mentioned in relevant communications, with a spike in 2020, indicating that 
public  sector  interoperability  is  becoming  a  more  important  policy  issue.  Most 
references are found after the 2010 EIF was adopted, although a few more references 
have been identified since the 2017 EIF was published. 

Figure 11 lists the EIF and related keywords in regulations, directives, and decisions from 
2004 to April 2021. 

Figure 11: EIF mentioned in EU policies 

Source: Study supporting the evaluation of the implementation of the EIF based on the EurLex dataset and the 
EurLex website 

Finally, more could be done to strengthen the position of the EIF as a guiding, strategic 
document.  It  is  important  to  ensure  that  alignment  with  the  EIF  is  not  limited  only  to  the 
declarative  level,  but  that  it  is  fully  taken  on  board  in  implementing  new  policies.  A  future 
interoperability strategy and policy could definitely contribute in this regard. 

4.2  How did EU intervention make a difference? 

National or sub-national initiatives would make a very limited contribution to the EIF’s aims. 
Consulted public authorities saw only limited potential for national and sub-national efforts 
to  contribute  to  the  EIF’s  existing  aims,  with  substantial  reliance  on  national  expertise  and 
improvements  in  digital  public  services.  Indeed,  there  are  countries  at  both  ends  of  the 
spectrum: those that are more advanced and experienced and those that are less advanced and 
experienced. This heterogeneity may have a detrimental effect on interoperability at EU level. 

Heterogeneity of solutions and methods may impede EU-wide interoperability. More crucially, 
the cross-border dimension cannot be addressed sole through the efforts of national or 
sub-national governments. Even if national, regional, and local public administrations strive 
to improve their approach to digital and interoperable public services, without coordination, the 
strategies and solutions selected may impede cross-border interoperability. Thus, the EU can 
serve both as a coordinator, ensuring a consistent approach to interoperability throughout the 
EU, and as a stimulant, offering direction and support. 

40 

 
 
The  Tallinn  Declaration  of  2017  and  the  Berlin  Declaration  of  December  2020196  both 
emphasised the importance of a unified strategy to interoperability. Additional evidence from 
desk research corroborates these conclusions. The list of Member States that have adopted an 
NIF or comparable strategic documents (Figure 3 and Annex VI.2) demonstrates that several 
countries have incorporated the EIF into their national frameworks, and in some cases, the EIF 
served as the starting point for such papers (as it is the case, for instance, for Croatia). 

Benefits at EU level 

By using economies of scale (via shared technologies) and economies of learning, the EIF is 
almost  certain  to  achieve  its  objectives  at  a  lesser  cost  than  equivalent  national  or  sub-
national projects (dialogue and shared solutions). 

The  EIF  decreases  costs  through  economies  of  scale  (via  common  tools)  and  economies  of 
learning,  as  confirmed  by  responses  to  the  evaluation  consultations  (dialogue  and  shared 
solutions). Initiatives at the national and/or local level would be substantially more expensive. 
Additionally, the EIF generates positive externalities as a result of stakeholder collaboration. 
Notably, the ISA² programme has aided in the development and sharing of tools and solutions 
by  serving  as  one  of  the  primary  instruments  for  executing  the  EIF,  namely  the 
interoperability action plan197. 

A uniform EU strategy eliminates the costs associated with each Member State assessing and 
devising  actions  to  address  challenges  on  an  individual  basis.  Additionally,  a  lack  of 
coordination  is  likely  to  result  in  divergent  paths  being  taken,  resulting  in  increased 
fragmentation rather than a shared vision. 

On average, the EIF has aided in the advancement of shared EU policy. However, more 
could  have  been  accomplished.  The  EIF  is  underutilised  across  sectors  and  by  Commission 
departments. Cooperation between other Commission departments should be strengthened to 
enable the EIF to produce stronger outcomes. A future interoperability governance, which is 
now being evaluated as part of the back-to-back study of a future interoperability policy, may 
help to close this gap. 

Although the EIF has given value to the EU, cross-border interoperability remains limited 
and driven by sectoral needs. The benefits of cross-border interoperability, as promoted by 
the EIF, have not been shared equitably throughout Member States. 

Without  a doubt, the framework might  make a  greater contribution  to  the development  of a 
European interoperability environment. Identifying and addressing significant cross-border 
demands would very certainly result in increased EIF take-up and consequently increased EU 
added value. 

The EIF’s primary contribution to cross-border interoperability is in raising awareness of the 
need for interoperability and in encouraging countries to implement NIFs based on a common 
model. The EIF contributes to cross-border interoperability in three ways. 

1.  It has increased awareness of the multi-tiered approach to interoperability that goes 
beyond  technical  considerations.  The  EIF  establishes  a  consistent  vocabulary  and 

196 

197 

Berlin  Declaration  on  digital  society  and  value-based  digital  government,  at  the  ministerial  meeting 
during  the  German  Presidency  of  the  Council  of  the  European  Union  on  8 December  2020, 
https://www.bmi.bund.de/SharedDocs/downloads/EN/eu-presidency/gemeinsame-erklaerungen/berlin-
declaration-digital-society.pdf?__blob=publicationFile&v=6. 
See SWD/2021/965 final (Staff working document), COM/2021/965 final (Report), CEPS (2021), Study 
supporting the final evaluation of the ISA2 programme. Available at: https://op.europa.eu/en/publication-
detail/-/publication/afa4297a-0acc-11ec-adb1-01aa75ed71a1/language-en/format-PDF/source-
229005953. 

41 

 
 
                                                           
conceptualisation  of  interoperability  (including  the  interoperability  layers  and 
conceptual model), which can serve as a common starting point for future projects in 
the field. 

2.  It encourages Member States to implement NIFs based on a common model, fostering 
an  EU-wide  approach  to  interoperability.  The  EIF  promotes  conversation  and 
cooperation.  eIDAS,  the  electronic  personal  identity  recognition  and  management 
system,  has  been  adopted  by  15  Member  States,  supporting  people  when  moving 
permanently or temporarily, or completing official procedures. E-invoices are another 
example  of  an  EU-developed  interoperable  solution  successfully  reused  at  national 
level. 

3.  Specific major digitalisation polices in the EU – such as the Single Digital Gateway 
Regulation  (SDGR)  or  the  Open  Data  Directive  –  have  been  designed  in  the 
implementation  phase  of  the  polices,  following  the  guidance  of  the  EIF  and  reusing 
some solutions developed under the EIF, such as eID, eDelivery and interoperable data 
models.  A  first  mapping  of  digital-relevant  EU  initiatives  onto  the  EIF  pillars  and 
individual recommendations shows this internal coherence. The SDGR is contributing 
to the interoperability of digital services and implementation of the once-only principle 
in many countries, such as Germany, Greece, Italy and Portugal. They have national 
plans – sometimes funded through the resilience and recovery plans (in Italy's case) – 
to develop interoperable and data sharing platforms, infrastructure and portals, common 
semantic  models,  and 
internal  coordination  and  cooperation  across  various 
administrative levels, in the spirit of implementing the once-only principle in parallel at 
home. With regards to the Open Data Directive, the national open data portals  – that 
have  a  set-up  following  the  Directive  in  Member  States  such  as  in  Belgium,  Spain, 
Ireland,  Italy,  the  Netherlands,  and  Sweden198  –  are  based  on  a  common  EU 
interoperable semantic model developed under EIF principles, guidance and solutions. 
Thanks to the semantic model, Member States can easily foster their national portals as 
interoperable  hubs  connecting  other  data  repositories  in  the  country,  mainly  from 
regions and big cities, so that data sets can be found easily and reused in an interoperable 
manner.  Likewise,  these  portals  are  interoperable  and  integrated  since  April  2021  in 
data.europa.eu, the official portal for European data, https://data.europa.eu, which relies 
on the same specification and their counter partners in the other countries in a cross-
border setting, allowing citizens and businesses in Europe to search for data sets hosted 
in any Member States of competent authority in a user centric way. 

Alignment at international level 

On a  global scale, the Organisation  for Economic Cooperation and Development  (OECD)’s 
work on digital government lends itself to comparison with the EIF. The ‘Recommendation on 
Digital Government199,’ approved in 2014 and consisting of high-level recommendations, and 
the ‘digital government toolkit’200 in particular, are relevant, with the toolkit giving assistance 
in implementing the high-level proposals. The OECD’s approach is, however, larger and less 
granular than what the EIF sought to do. 

198 

199 

200 

https://joinup.ec.europa.eu/collection/semantic-interoperability-community-semic/solution/dcat-
application-profile-data-portals-europe/about. 
OECD (2014), Recommendation of the Council on Digital Government Strategies, Adopted by the OECD 
Council  on  15 July  2014,  http://www.oecd.org/gov/digital-government/Recommendation-digital-
government-strategies.pdf. 
OECD Digital Government Toolkit, http://www.oecd.org/governance/digital-government/toolkit/home/. 

42 

 
 
                                                           
The  EIF  is  strongly  aligned  with  OECD  recommendations  on  interoperability;  nevertheless, 
when it comes to recommending the implementation of standards, it does not expressly refer to 
standards development organisations (SDOs). 

Most open data standardisation activities occur at the international level through the W3C and 
OASIS, as documented in the chapter on ‘Big data, open data, and public sector information’ 
in the 2022 rolling plan for ICT standardisation201.’ Additionally, it should be emphasised that 
the  rolling  plan  now  makes  a  reference  to  the  EIF.  Moreover,  the  Commission  states  in  its 
Communication on European data strategy202 that ‘the use of standard and shared compatible 
formats and protocols for collecting and processing data from diverse sources in a consistent 
and interoperable manner across sectors and vertical markets should be encouraged through the 
rolling plan for ICT standardisation and (in the case of public services) a strengthened European 
Interoperability Framework.’ 

To  summarise,  the  EIF  could  promote  further  adoption  of  standards  created  by 
standardisation bodies. The ICT standardisation rolling plan could go beyond just referencing 
the EIF and specify possible actions involving standards development organisations. Finally, 
given  the  EU’s  uniform  approach  to  interoperability,  the  EIF  may  help  ensure  that 
standardisation initiatives support the European data strategy in line with the notion of digital 
sovereignty for Europe. 

4.3  Is the EIF still relevant? 

The  EIF  mainly  aims  to  assist  public  administrations  in  providing  vital  interoperable,  user-
centric  digital  public  services  to  businesses  and  residents  at  EU,  national,  regional  and 
municipal level. 

Some  issues  with  EIF  design  persist.  Among  these  are  the  need  to  raise  awareness  about 
interoperability,  solve  skills  shortages  in  public  sector  IT  departments,  enhance  investment 
capacity, keep up with rapid technological change, and support public-private collaborations. 

Nonetheless, the EIF appears to have attained some of its key objectives based on input from 
the targeted consultation. 

The EIF’s early  goals  were to inspire and guide European public administrations at all 
levels to develop and provide seamless European public services. While the EIF has issued 
recommendations, it is insufficiently implemented. 

Public  administrations  would  still  benefit  from  a  common,  more  practical,  end-to-end 
approach  from  the  EIF,  from  policy  to  implementation.  This  would  increase  its  relevance. 
Stakeholders seem to agree that more direction and awareness-raising is required among public 
authorities, especially at sub-national level. More guidance goes hand in hand with promoting 
interoperability  at  all  levels.  More  ambitious  measures  are  expected  to  raise  public 
administrations’ knowledge of the EIF’s potential benefits, especially at sub-national levels203. 
Cultural characteristics, especially the diversity of the public sector and associated actors204, 
can play  a major role in implementing interoperability solutions. A silo-based strategy, for 
example,  produces  fragmentation  owing  to  user-unfriendly  systems,  which  hinders  the 

201 

202 
203 

204 

For further details please see: https://joinup.ec.europa.eu/collection/rolling-plan-ict-standardisation/big-
data-open-data-and-public-sector-information. 
https://ec.europa.eu/info/sites/default/files/communication-european-strategy-data-19feb2020_en.pdf 
See  Halmos  (2018),  Cross-border  digital  public  services.  Available  at:  http://institute.cesci-
net.eu/en/crossborder-review-2018. 
Pardo & Tayi (2007) identify the  multitude of actors and types of information as a barrier that could 
impede the development of interoperability. 

43 

 
                                                           
exploitation  of  synergies205,  according  to  the  impact  assessment  accompanying  the  SDG 
proposal.  Some  stakeholders  mentioned  the  linguistic  barrier.  Interoperability  is  thus  not 
primarily  a  technology  issue,  but  an  organisational  and  informational  one,  influenced  by 
conventions and values. Taking cultural issues into account, especially the diversity of public 
sector and associated actors, can play a vital role in developing interoperability solutions (see 
also Box 3). 

Box 3: The role of culture in enhancing interoperability 

Interoperability  plays  a  central  role  in  reducing  administrative  barriers,  which  are 
underpinned by a ‘complex context of different linguistic, cultural, legal, and administrative 
environments  in  the  EU.’206  Although  European  public  administrations  share  the  values 
associated with democracy and the rule of law, national administrative cultures show some 
differences207.  While  it  is  expected  that  cultural  differences  influence  the  adoption  of 
interoperability  solutions  in  various  ways,  disentangling  the  causal  effects  and  the  main 
factors involved is not straightforward. The literature contains no accurate assessment of the 
extent  to  which  different  national  administrative  cultures  influence  the  adoption  of 
interoperability solutions by European public administrations208. 

To create a culture of interoperability and secure common efforts to attain interoperability 
goals in the EU, raising awareness across all EU Member States and stakeholders is crucial. 
To this end a future interoperability strategy should consider ways to raise awareness of the 
benefits of interoperability across the EU among European public administrations, businesses 
and  the  public.  In  this  regard,  the  upcoming  Digital  Europe  programme  could  foster 
awareness of interoperability benefits, thus contributing to the achievement of the objectives 
of the future interoperability strategy.  

Relevance at EU level 

Digital  (and  interoperable)  transformation  of  the  EU  public  sector  requires  greater  EU 
cooperation,  common  rules  and  initiatives.  The  COVID-19  epidemic  has  undoubtedly 
hampered  cooperation  and  necessitated  stronger  governance  mechanisms.  The  crisis  has 
highlighted the significance of digitisation and interoperability in the public sector, not just 
to  improve  basic  public  services  but  also  to  build  resilience  and  capability  to  deal  with 
unexpected emergencies. Studies into the pandemic’s impact and the EU’s response provide 
evidence.  During  the  epidemic,  countries  with  more  digitalised  and  interconnected 
governmental  administrations  and  services  reported  less  service  disruption  (Charay  et.  al., 
2021). The pressing need for data transfer across countries has helped raise awareness of the 

205 

206 

207 

208 

p. 21. 

Commission staff working document impact assessment accompanying the ‘Proposal for a regulation of 
the  European  Parliament  and  of  the  Council  on  establishing  a  single  digital  gateway  to  provide 
information,  procedures,  assistance  and  problem-solving  services  and  amending  Regulation  (EU) 
No 1024/2012’, 
https://ec.europa.eu/transparency/documents-
at: 
Available 
register/detail?ref=SWD(2017)213&lang=fr. 
COM(2017) 134 final, Communication from the Commission to the European Parliament, the Council, 
the  European  Economic  and  Social  Committee  and  the  Committee  of  the  Regions,  European 
Interoperability Framework – Implementation Strategy, Annex 2, p. 26. 
Thijs  N.,  Hammerschmid  G.,  Palaric  E.  (2017),  A  comparative  overview  of  public  administration 
characteristics  and  performance 
in  EU28,  European  Commission,  pp.  34-37.  Available  at: 
https://op.europa.eu/en/publication-detail/-/publication/3e89d981-48fc-11e8-be1d-
01aa75ed71a1/language-en. 
Nauta B. (2019), The influence of national culture  on the  interoperability of cross-border IT systems, 
at: 
Dissertation 
https://www.researchgate.net/publication/337104072_The_influence_of_national_culture_on_the_inter
operability_of_cross-
border_IT_systems_A_mapping_of_dimensions_of_national_culture_and_layers_of_interoperability. 

submitted 

Available 

Institute. 

the 

IC 

to 

44 

 
                                                           
importance of interoperable systems, particularly for crisis management and response. The 
field literature supports targeted interoperability approaches. 

The  pandemic  also  highlighted  the  need  for  better  governance  of  pandemic  response 
mechanisms across the EU. Data exchanges, compatible systems and processes, and shared 
standards all play a role. The lack of data consistency hampered the EU’s capacity to respond 
cooperatively early in the outbreak (Renda and Castro, 2020). Interoperability by design was 
emphasised  as  vital  for  effective  cross-border  contact-tracing  apps  in  the  EU  (Ciucci  and 
Gouardères,  2020).  The  argument  around  the  EU  Digital  COVID  certificate  included 
discussions about standardisation and interoperability. 

To promote digitalisation initiatives and the creation of a cohesive interoperability environment 
across the EU, a more binding approach to interoperability requirements may be required 
in future. Interoperability by design and reuse of interoperability solutions could potentially be 
explored as ways to improve the cohesiveness of the interoperability environment across the 
EU. The usage of conditionality and the need for more interoperability collaboration are among 
the conclusions of the ISA² interim and final evaluations. 

Stakeholders emphasised the EIF’s potential benefits in other areas like mobility, environment, 
and smart cities. Encouraging public-private partnerships is also possible, as public services 
are not always generated and delivered solely by the public sector. 

5.  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNT? 

5.1  Conclusions 

Interoperability  makes  for  efficient  digitalisation  of  government  services,  and  enables  the 
seamless  and  trusted  exchange  of  data  and  connectivity  of  services  across  levels  of 
administration, borders and sectors. It is a foundational element for any modern digitalisation 
strategy. 

The 2017 EIF is a non-binding framework for interoperability in the EU public sector (a first 
version of the EIF was developed in 2004; a new version EIF was formally adopted in 2010 
through  a  Commission  Communication).  The  EIF  has  issued  suggestions  and  guidance  on 
interoperability to  enable public administrations to  establish digital  and interoperable public 
services. A whole set of interoperable and reusable solutions have been created as part of the 
interoperability action plan, and supported by the ISA² and CEF programmes to make the EIF 
more actionable and implementable. This report has assessed the framework’s achievements 
and faults, as well as lessons learnt, four years after its introduction. 

Successful elements of the 2017 EIF 

Because it represents an outstanding (but non-binding) reference approach to interoperability 
(see Section 4.2) the EIF brings EU added value. National or sub-national initiatives would 
contribute  very  little  to  the  EIF’s  goals.  Particularly,  national  or  sub-national  governments 
cannot  achieve  cross-border  interoperability  alone.  Economies  of  scale  (through  shared 
technologies) and experience enable the EIF to achieve its goals more cheaply than equivalent 
national or sub-national projects (dialogue and shared solutions). 

The framework has provided high-level advice through concepts and models to help guide the 
creation of interoperable digital public services. In this regard, the EIF monitoring mechanism 
shows that some of the EIF’s recommendations have been conceptually implemented by the 
Member States, such as openness, technological neutrality, data portability, reusability of IT 
solutions,  and the layered interoperability model  (except  for the interoperability  governance 
layer), including open data and base registries. 

45 

 
In terms of efficiency, the EIF’s benefits may outweigh the expenditures incurred by public 
agencies  to  implement  it  (see  Section 4.1).  Implementing  the  EIF  guideline,  or  parts  of  it, 
through national frameworks may cost between EUR 24 000 and EUR 169 000 per Member 
State (based on estimates of time translated into costs provided by representatives of several 
public authorities in the Member States). Also, short-term investments must be weighed against 
long-term  rewards.  While  initial  expenditure  can  be  costly,  implementing  EIF  advice  is 
projected to reduce costs over time as processes are optimised. 

The concepts, layered interoperability architecture, and conceptual model for integrated public 
service  delivery  defined  in  the  EIF  are  generally  synergistic,  and  the  suggestions  mutually 
reinforce  each  other  (see  Section 3.1.3).  In  terms  of  external  coherence,  the  examination 
indicates  that  the  EIF  has  been  considered  in  areas  where  interoperability  is  critical  (see 
Section 3.1.3). Public and targeted dialogues as well as desk research have identified synergies 
between  the  EIF  and  the  single  digital  gateway,  the  digital  single  market  strategy,  the  data 
strategy  and  the  upcoming  data  act.  The  EIF’s  role  in  digitalising  EU  public  services  is 
highlighted by the digital single market and data strategies. While the Single Digital Gateway 
Regulation does not expressly address the EIF, it does mention several of its major principles 
and concepts. However, these are more commonly mentioned in contexts other than the SDG 
and the two strategies mentioned, demonstrating that public sector interoperability is becoming 
more of a policy debate topic. 

Similar to the EIF, the OECD’s Digital Government initiative provides advice on setting up 
digital and interoperable public services. The OECD’s methodology is wider and less specific 
than the EIF’s. The EIF thus adds value in the EU. 

Less successful elements of the 2017 EIF and areas for improvement 

Whilst  the  EIF  as  an  EU-level  intervention  has  an  added  value  to  the  interoperability 
environment (see Section 3.2), its ultimate impact is dependent on national and sub-national 
implementation across all sectors with digital public services. The EU’s solution and strategy 
heterogeneity may impede interoperability. 

Given  the  EIF’s  low  overall  success  and  the  need  for  additional  practical  direction,  it  is 
judged just moderately useful. The EIF recommendations have only partially helped meet the 
goals set when the framework was adopted in 2017. This is because the EIF has wide aims and 
the  suggestions  are  often  high-level.  More  granularity  is  required  for  efficient  framework 
implementation,  especially  at  the  local  level.  A  few  concepts  (evaluation  of  efficacy  and 
efficiency, subsidiarity and proportionality, multilingualism, and information preservation) are 
too abstract or require additional elaboration to be effective. Also, the principles of openness, 
technological neutrality, and user-centricity require more clarification and practical direction. 
The EIF monitoring mechanism found that certain principles need to be improved at country 
level, indicating the need for further guidance (including the assessment of effectiveness and 
efficiency, multilingualism, and subsidiarity and proportionality). Further advice is needed to 
the make the layered interoperability and conceptual models more actionable, according 
to  the  study.  For  example,  the  layered  interoperability  model’s  advice  on  interoperability 
governance,  organisational,  and  legal  interoperability  should  be  improved  (as  emphasised 
throughout  the  consultation  activities  and  through  the  results  of  the  EIF  monitoring 
mechanism). 

Internal coherence could be improved by better organising the components. Assemblage of 
the three components should be improved. The EIF’s suggestions typically reinforce each other. 
Nonetheless, numerous EIF recommendations overlap, at least thematically (e.g. Nos 2, 42 and 
43; and Nos 21, 22, 23 and 24). The overlaps are not a problem, but better organising the advice 
would make the big ideas clearer and more actionable. In terms of external coherence, the 
EIF is openly mentioned in policy documents and less so in EU legal acts like directives and 

46 

 
regulations. On coherence, the EIF is found to be consistent with initiatives such as the Single 
Digital Gateway Regulation, the digital single market strategy, and the data strategy. To 
increase interoperability governance and the EIF’s role, more references in associated legal acts 
are needed. 

While, as an EU-level intervention, the EIF has added value to the interoperability landscape 
(see  Section 3.2  and  Annex VIII),  ultimately  its  impacts  depend  on  the  level  of  practical 
implementation  at  national  and  sub-national  level  and  across  all  sectors  with  digital  public 
services. Heterogeneity of solutions and strategies across the EU may hinder interoperability at 
EU level. 

Will the identified needs be met, and problems resolved, over time? 

The EIF has only partially addressed the demands and challenges indicated by the stakeholders 
(see  Section 3.3).  Public  administrations  emphasise  the  need  for  further  cooperation,  citing 
inadequate EIF implementation guidance. In this context, a more practical, end-to-end EIF 
approach should deliver additional benefits. The digital (and interoperable) transformation of 
the EU public sector requires unified norms and policies. Other public sector interoperability 
needs  include  raising  public  knowledge  of  the  EIF,  addressing  staffing  shortages  in  IT 
departments, increasing investment capacity, and encouraging private-public cooperation. 

In this respect, as stated before external factors such as technical improvements and national 
and cross-sectoral interoperability initiatives help the EIF to deliver. At the same time, negative 
aspects  such  as  limited  resources  and  institutional  complexity  must  be  mitigated  (see 
Section 4.1). Finally, the COVID-19 epidemic has heightened the need for greater coordination 
and cooperation to improve interoperability in the EU public sector. 

In summary, the needs and issues identified are expected to persist. To satisfy the needs and 
overcome the challenges outlined, more cooperation, common rules, and coordinated EU 
initiatives  are  required.  It  is  worth  exploring  the  possibility  of  putting  in  place  sound 
governance to enhance cooperation in a future interoperability policy. 

5.2  Lessons learnt 

The  EIF’s  actual  results  only  partially  match  the  predicted  outcomes  and  consequences 
mentioned  in  the  EIF’s  intervention  rationale  (see  Section 2).  As  a  model  for  national 
interoperability  policies  and  plans,  the  EIF  has  given  high-level  guidelines  promoting  the 
development of interoperable digital public services across the EU. Most Member States have 
adopted  NIFs  or  equivalent  national  strategic  papers  that  incorporate  EIF  suggestions, 
principles, and models. Over the last two  years, a small but a steady improvement has been 
observed in general in the level of implementation of the framework across the three main areas, 
namely the principles, the interoperability layers, and the conceptual model. 

However,  there  is  a  disconnect  between  high-level  guidance  and  actual  execution, 
particularly at regional and municipal level. So far, the EIF has helped build the digital single 
market  by  promoting  cross-border  and  cross-sector  interoperability,  but  more  can  be  done. 
Developing  a  European  public  services  ecosystem  and  making  interoperable,  user-centric 
public services widely available in the EU are key priorities. 

Improving interoperable digital public services would surely boost direct and indirect benefits 
for stakeholders. An economic analysis reveals that adopting an NIF based on the EIF, or only 
part of it, is preferable to not adopting it at all. A 10% improvement in online public service 
quality would also attract 4.3 million to 5 million more EU residents each year to interact with 
public authorities online. 

47 

 
In  connection  with  the  European  Commission’s  REFIT  project,  enhancements  to  the  EIF 
methodology  could  reduce  administrative  costs  by  streamlining  processes  and  assist  the 
production of more efficient digital public services through improved interoperability. 

Another difficulty is the scarcity of data, especially comparable data, for assessing the costs 
and  benefits  of  translating  the  EIF  into  national  frameworks.  The  voluntary  nature  of  the 
framework is reflected in its uneven, ad hoc uptake across the EU, limiting the availability of 
data and making comparison between countries difficult. Furthermore, implementation of the 
EIF has followed different timelines in different countries. 

Several lessons can be derived from the experience of EIF implementation in recent years. 
These lessons will be helpful in further enhancing interoperability activities at EU level. 

1.  Clarify  the  recommendations  that  are  generally  sound,  although  they  might  be 
enhanced with better structure, clarity, granularity, and implementation guidance. The 
recommendations  could  be  enhanced  by:  i)  arranging  them  by  target  stakeholders 
(national government, data owners, and local agencies); ii) distinguishing between basic 
and advanced recommendations; iii) bridging the conceptual and practical  gaps. The 
recommendations  could  be  refined  to  better  support  public  administrations  in 
implementing the EIF. 

2.  Give practical guidance by strengthening the links between EIF and ISA² solutions. 
The EIF Toolbox helps to connect the EIF’s suggestions to available solutions. Future 
EU  public  sector  interoperability  policies  should  provide  an  integrated  approach, 
integrating  strategic  initiatives  with  activities  to  increase  adoption  of  standard 
interoperability solutions.  To make the EIF more actionable in  the future, it must  be 
linked to specific solutions. 

3.  Encourage  best-practice  sharing.  Given  the  EIF’s  strategic  and  overall  high-level 
recommendations, best practices for implementation might be added. In this context, 
the NIFO collection on Joinup’s digital policy hub presents examples of trends across 
the Member States. The hub might be maintained in the future, and more examples of 
Member States’ specific implementing solutions could be collected, building on the 
EIF’s guidelines. 

4.  Foster  coordination  through  common  rules.  Common  rules  could  improve  digital 
public  services  across  the  EU  and  strengthen  the  public  sector’s  resilience  to 
unanticipated  shocks.  The  EIF’s  voluntary  framework  has  helped  promote 
interoperability in public service design, but an EU public sector interoperability policy 
might benefit from a compulsory approach and might consider the following. 

a.  Enhanced EU policy for interoperability with more binding requirements. 
Coordination is required to create an EU-wide digital public service landscape. 
Having EU-wide laws would help to ensure that cross-border aspects of digital 
public services are considered. 

b.  Use of conditional interoperability solutions and interoperability by design 
in the development of digital public services could help achieve interoperable 
cross-domain and cross-border services throughout the EU. Such a framework 
may combine, for example, EU financing for public sector transformation and 
digitisation with interoperability standards. 

5.  Strengthen the EIF’s role as a strategic guide. Enhancing synergies between the EIF 
and  EU  interoperability,  and  digitalisation  efforts,  is  critical  for  a  cohesive  EU 
interoperability environment. In this regard, more direct references to the EIF may assist 
raise awareness of its broad implications. It is also vital that other EU programmes are 
completely aligned with the EIF in terms of policy implementation. 

48 

 
6.  Educate and involve stakeholders. Interoperability has long been a technical topic, 
but its broader ramifications are becoming apparent, with the EIF’s help. To keep EU 
action in this subject relevant, it is necessary to: 

a.  promote  interoperability  as  a  fundamental  enabler  for  public  sector 

digitisation. 

b.  ensure  that  stakeholders  are  aware  of  the  benefits  stemming  from 
interoperability  and  thus  incentivise  public  administrations  to  carry  out 
coordinated  initiatives  for  digital  public  services  that  are  compatible,  so 
contributing  to  a  more  cohesive  public  sector  interoperability  environment 
across the EU. 

7.  Ensure monitoring of progress in the field of interoperability and the EIF’s effects, 
to keep building a solid understanding on the EIF’s effectiveness, especially at the local 
level.  Monitoring  initiatives  should  thus  continue  to  gather  data  on  interoperability 
guidance implementation and costs, and encourage reuse of developed technologies. In 
addition, the EIF monitoring mechanism might be improved to better capture practical 
acceptance of recommendations in developing digital public services. 

8.  The EIF could promote the adoption of open specifications and standards created 
by standardisation bodies. The EIF is already referenced in the ICT Standardisation 
rolling plan, and some semantic interoperability specifications are mentioned. Finally, 
given  the  EU’s  uniform  approach  to  interoperability,  the  EIF  may  help  ensure  that 
standardisation initiatives support the European data strategy and the data act proposal, 
in line with the notion of digital sovereignty for Europe. 

9.  Future monitoring should focus on reusing developed technologies and on providing 
better estimates of costs. In this regard, a set of EIF-specific criteria could make it easier 
to compare costs across the EU (including, for instance, daily work of the team tasked 
with implementing the EIF through official documents, the degree of implementation 
of  the  EIF,  the  number  of  consultations  between  the  European  Commission  and  the 
Member States when the EIF was agreed on and introduced, the time spent by annual 
reporting under the NIFO initiative, etc.). 

10. In  terms  of  efficiency  and  the  quantification  of  cost-benefits  stemming  from 
interoperability, future sound methodologies to estimate the costs of interoperability, as 
opposed to the costs of digital transformation in a broader setting, must be implemented 
and applied in the field. 

For all of it, a future enhanced EU policy for public administration interoperability is now being 
assessed  along  with  this  EIF  evaluation  (back-to-back  impact  assessment  for  a  future 
interoperable Europe policy). The policy will address binding requirements and a governance 
supporting  a  more  practical  and  actionable  EIF.  In  addition,  it  will  identify  common 
specifications and standards, reusable tools, and solutions to support at least digital checks for 
new EU policies. 

49 

 
 
 
ANNEXES 

Annex I. Procedural information 

Lead DG: Directorate General for Informatics 

Decide  Planning:  PLAN/2020/7507  -  ‘Evaluation  of  the  implementation  of  the  European 
Interoperability  Framework  (EIF)  and  proposal  for  an  EU  governments  interoperability 
strategy’. 

The Initiative is part of CWP 2022 - REFIT initiatives, point 12, and is referenced under the 
Policy objective ‘A Europe fit for the Digital Age’. The adoption is planned for Q2 (June 2022). 

Organisation and timing 

31/08/20 

15/09/20 

Decide Political Validation (Hahn, Vestager) 

1st ISSG meeting 

Participating  DGs:  CNECT,  DEFIS,  DIGIT,  ECFIN,  EMPL, 
ENER, ENV, ESTAT, FISMA, GROW, the JRC, JUST, MOVE, 
NEAR, OP, REFORM, REGIO, RTD, SANTE, SG and TAXUD 

15/10/20 

Publication  of  combined  Evaluation  Roadmap  and  Inception 
Impact Assessment 

01/02/21– 26/04/21  Public consultations 

12/10/21 

15/10/21 

ISSG approval for supporting studies 

Recommendations  of  the  Expert  Group  on  interoperability  of 
public services 

24/11/21 

9th ISSG meeting 

Participating  DGs: CNECT,  ENV,  ESTAT,  GROW,  JUST,  the 
JRC,  MOVE,  OP,  REGIO,  RTD,  SANTE,  SJ,  SG,  TAXUD, 
DIGIT 

19/01/22 

RSB meeting 

Consultation of the RSB 

An upstream meeting with the RSB took place on 12 March 2021, whose recommendations 
were duly taken into account.  

This  draft  EIF  evaluation  SWD  was  submitted  to  RSB  on  17  December  2021  for  the  RSB 
hearing that was held on the 19 December 2022, getting a positive opinion with some comments 
for improvement. 

The Table below summarises the changes introduced to this Staff Working Document (SWD) 
in response to the Board’s main comments 

Main RSB considerations 

Changes made to the SWD 

The scope of the EIF is unclear. The 
report  does  not  clearly  outline  the 

The  scope  of  the  EIF  and  the  definition  of  the 
concept  of  interoperability,  as  understood  in  this 

50 

 
Main RSB considerations 

Changes made to the SWD 

areas  covered  by 
interoperability 
principles and solutions, in particular 
whether the focus is  on  cross-border 
national 
or 
interoperability 
frameworks,  and  which  stakeholders 
the initiative targets 

on 

context,  have  been  made  clearer  and  presented  in 
more detail in the introduction and section 2.1 

to 

evaluation 

lacks 
illustrate 

concrete 
The 
examples 
the 
achievements of the EIF, whether in a 
sector,  policy  area  or 
specific 
Member State, or at EU level. 

The  evaluation  does  not  clearly 
identify  the  specific  contributions  of 
interoperability,  and  of 
the  EIF 
specifically,  as  opposed  to  broader 
benefits of the digitalisation of public 
services. 

The  analysis  of  costs  and  benefits 
does not sufficiently explain why cost 
data  are  difficult  to  quantify,  and 
whether this is the case for all aspects 
of interoperability. 

The evaluation illustrates now in sections 3.1 and 4 
interoperability  an  update  of  the  EIF  effects  at 
policy level as a guidance instrument in the 27 MS, 
and the evolution at macro level from 2021 to 2021. 
Also,  it  gives  examples,  drawing  on  achievements 
from  particular  sectors  or  from  Member  States, 
which can be directly  attributed to  interoperability 
and the implementation of the EIF. 

Addressed particularly in sections 3 and 4.1 and 4.2. 

Explicit  mentions  have  been  emphasised  to  the 
limitations 
of 
estimating 
interoperability,  as  opposed  to  the  costs  of  digital 
transformation in general. 

costs 

the 

of 

Added in section 4.1: 

 

  Expanding  the  cost  benefit  analysis  with  the 
findings of the economic impact of public sector 
interoperability in geolocation 
further  insights  on  the  current  difficulties  and 
limitations to quantify cost based on sound data 
  emphasising the tools that allow monitoring and 
measurement  of  the  level  of  implementation  of 
the EIF principles with some examples 

Evidence, sources and quality 

The Commission has been working for more than 25 years now on interoperability at EU level 
with  different  stakeholders  involved  in  the  digital  transformation  of  the  public  sector  and 
creation of EU-wide digital public services. 

The  evidence  used  in  the  Impact  Assessment  have  been  collected  from  the  following  main 
sources and has been used at different steps in the preparation of the initiative: 

  National Interoperability Framework Observatory (NIFO) data collection: 

o  The  Digital  Public  Administration  Factsheets  collected  and  published  yearly; 

available on the Joinup platform209; 

o  The European Interoperability Framework (EIF) Monitoring Mechanism210; 

209  
210  

The factsheets can be consulted on Joinup. 
An overview of the EIF Monitoring Mechanism. 

51 

 
                                                           
  Other databases including: Eurostat, the World Bank Worldwide Governance Indicators, 
United Nations e-Government survey, and the Organisation for Economic Co-operation and 
Development  (OECD)  Digital  Infrastructure  indicators;  the  datasets  related  to  the  e-
Government Benchmark reports211; 

  Expert Group on interoperability of public services inputs and recommendations 

o  The interoperability expert group was created on 14/02/2020 with representatives of 
the  national  administrations  CIO  offices,  members  appointed  for  all  27  Member 
States and observers from EFTA and Western Balkans countries 

o  Position papers from 18 Member States: In December 2020 the Commission kindly 
asked the members of the expert group to react on a short policy paper with some 
initial  ideas  for  the  next  interoperability  policy  (vision,  general  approach,  policy 
instruments, etc.); 

o  Co-creation workshops – five informal discussions and brainstorming sessions took 

place between January and December 2021 with more than 200 participants; 

o  Bilaterals – the Commission organised 26 bilateral calls with 26 different Member 

States between July and September 2021; 

o  Policy recommendations – they were formally endorsed by consensus of the whole 
group on the 5th of October 2021. Those recommendations served as a basis to build 
the draft legal text and the two dedicated workshops. 

o  Two workshops on legal draft – They took place in November 2021 gathering both 

around 40 external participants each. 

  CEPS external supporting studies - The Commission contracted the Centre for European 
Policy Studies (CEPS) to conduct three independent supporting studies between September 
2020 and October 2021. 

o  Supporting study for the evaluation of the ISA² programme 
o  Supporting study for the evaluation of the EIF 
o  Supporting study on the impact assessment for a future interoperability strategy 

  Joint Research centre (JRC) analysis – JRC has provided extensive support in preparation 
of the Impact Assessment, in particular with a study on quantifying the benefits of Location 
Interoperability and general costs of interoperability in the EU212 

  Input received to the Inception Impact Assessment feedback period213, 13 responses 

that are summarised in Annex V. 

  Targeted online survey 94 responses that informed the analysis and the conclusions of the 

study, summarised in Annex V. 

  In-depth interviews with 23 experts and stakeholders 
 

Input  from  Public  Consultation  launched  for  12  weeks,  from  01/02/21–  26/04/21, 
summarised in Annex V. 

  4 public workshops: 1 inception workshop and 2 validation workshops in the framework 

of the DIGIT all conference, 1 public expert panel on options for a legal instrument 

The Impact Assessment was based on certain assumptions, namely: 

 

It was estimated that the number of cross-border cases of usage of digital public services 
would be growing with the increase of the number of people living and working in a 

211  
212  

213  

The e-Government Benchmark datasets are available at: https://digital-agenda-ata.eu/datasets/e-gov 
Ulrich,  P.,  Duch  Brown,  N.,  Kotsev,  A.,  Minghini,  M.,  Hernandez  Quiros,  L.,  Boguslawski,  R.  and 
Pignatelli, F., Quantifying the Benefits of Location Interoperability in the European Union, EUR 31004 
EN,  Publications  Office  of  the  European  Union,  Luxembourg,  2022,  ISBN  978-92-76-48846-0, 
doi:10.2760/72064, JRC127330. 
See  relevant  web  page  on  Europa  Have  your  say,  available  at  https://ec.europa.eu/info/law/better-
regulation/have-your-say/initiatives/12579-Interoperable-digital-public-services-European-
Interoperability-Framework-evaluation-&-strategy_en 

52 

 
                                                           
Member State different from the one of their origins and with the increase of the number 
of people traveling for tourism purposes. 

The data limitations encountered in this Impact Assessment were the following:  

  Lack of data focusing solely on the benefits and costs of interoperability 
  Lack of comparable data on the costs and benefits of transposing the EIF into national 
frameworks, due to the voluntary nature and heterogeneous uptake of the framework 

  Relatively low number of respondents to the different surveys 

To mitigate the impact of the data limitations (to the extent possible), the external contractor 
followed up directly with some of the stakeholders to clarify certain aspects. The Joint Research 
Centre kindly offered their expertise and has developed a more specific study that took both a 
qualitative approach (20 use cases on location data interoperability) and quantitative (estimation 
of general costs of interoperability at EU level). In addition, where quantification of costs and 
benefits was not feasible, a qualitative approach was chosen instead (description of practices, 
processes and types of costs and benefits deriving from the options). To increase the number of 
stakeholders, involved an important number of public and targeted consultation activities were 
put in place. 

Annex II. Methodology and analytical models used 

This  annex  presents  the  methodology  employed  for  the  back-to-back  evaluation  and  impact 
assessment. It details the types of data collected and their validation, the methods used, and the 
main limitations of the analysis. 

Overview of information and data sources 

In the Data Collection Phase, a mix of data collection methods and desk research was employed 
to gather both primary and secondary information and data, as follows: 

  Primary information and data were collected via the following consultation activities: 
o  Work  on  recommendations  with  the  Expert  Group  on  Public  Sector 

interoperability 

o  Targeted in-depth interviews; 
o  Targeted online survey; 
o  Public consultation. 
o  feedback received on the Combined EIF Evaluation Roadmap and Inception 

Impact Assessment214 

  Secondary information and data were collected by reviewing: 

o  The Digital  Public Administration Factsheets,  developed as part of the  NIFO 

action and available on the Joinup platform215; 

o  The EIF Monitoring Mechanism216; 
o  Databases  including  Eurostat,  the  World  Bank  Worldwide  Governance 
Indicators, and the Organisation for Economic Co-operation and Development 
(OECD) Digital Infrastructure indicators; 

214  

215  

216  

The feedback of stakeholders can be consulted at:  https://ec.europa.eu/info/law/better-regulation/have-
your-say/initiatives/12579-European-Interoperability-Framework-EIF-evaluation-and-EU-
governments-interoperability-strategy/feedback?p_id=9804060&page=1 
The  factsheets  can  be  consulted  on  Joinup:  https://joinup.ec.europa.eu/collection/nifo-national-
interoperability-framework-observatory/digital-public-administration-factsheets 
EIF 
An 
https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-observatory/eif-
monitoring-mechanism 

Mechanism 

Monitoring 

overview 

available 

the 

of 

is 

at: 

53 

 
 
                                                           
o  The datasets related to the e-Government Benchmark reports217; 
o  Relevant literature; 
o  Official  EU-level  documents  such  as  communications,  regulations,  and 

directives; 

o  Official national-level documents such as national interoperability frameworks. 
The above-mentioned sources were complemented by expert assessments conducted by five 
independent experts who were part of the Team on the supporting studies conducted by CEPS, 
in  accordance  with  the  Technical  Proposal.  The  independent  experts  were  tasked  with 
completing  the  questionnaire  that  served  as  the  basis  for  the  in-depth  interviews  and  with 
providing additional expert input to support the evaluation and impact assessment. 

Primary data 

The consultation activities were launched in December 2020 (with a kick-off workshop to raise 
awareness on the evaluation process) and lasted until the end of April 2021 (marked by the end 
of the public consultation). The duration of each consultation activity was planned to ensure 
that stakeholders had enough time to provide their inputs. This was complemented with a co-
creation process with the Expert Group on interoperability of public services (see Annex V). 

The general consultation activities resulted in a total of 134 responses for the impact assessment 
and 112 responses for the EIF evaluation, which can be divided into six stakeholder groups as 
presented in Table 2. The stakeholder categories were defined on the basis of the assessment 
of relevant stakeholders conducted in the inception phase and by taking into account the need 
to ensure the comparability of results between the three strands of the consultation activities: 
in-depth  interviews,  online  survey,  and  public  consultation  (considering  in  particular  the 
mandatory template for background information used for public consultations). 

Table 2: Overview of responses to the consultation activities by stakeholder group for the 
impact assessment/evaluation 

Consultation 

Stakeholders  

In-depth 
interview 

Online survey  Public 

TOTAL 

consultation 

Civil society (all other 
stakeholders) 

Companies and business 
associations 

1/- 

-/- 

EU and non-EU citizens 

-/- 

EU public authorities 

3/5 

Experts and academia 
(including the 
independent expert 
assessments) 

6*/6* 

1/2 

5/3 

4/2 

10/8 

3/4 

8/7 

10/9 

13/14 

18/17 

31/18 

- 

5/4 

35/20 

13/13 

14/14 

National 
authorities 

public 

-/2 

27/25** 

16/12 

44/39 

217  

The e-Government Benchmark datasets are available at: https://digital-agenda-ata.eu/datasets/e-gov 

54 

 
                                                           
Consultation 

Stakeholders  

In-depth 
interview 

Online survey  Public 

TOTAL 

consultation 

TOTAL 

10 

50 

73 

134/112 

Note: x/y:x:Number of participants in impact assessment, y:number of participants for EIF evaluation. 
* In-depth interviews conducted with experts and academia include four independent expert assessments. 
**One follow-up interview to the targeted online survey was conducted. 
Source: Study supporting the impact assessment for a future interoperability strategy 
In  order  to  facilitate  the  comparison  of  stakeholders’  responses,  the  questionnaire  of  each 
consultation  activity  relied  on  Likert  scales.  Respondents  were  thus  asked  to  provide  their 
feedback  by  referring  to  a  scale  from  (1)  to  (5)  or  (--)  to  (++),  depending  on  the  type  of 
question218: 

1. 

2. 

(1) – not at all; (2) – to a limited extent; (3) – to some extent; (4) – to a great extent; or 
(5) – completely; 
(--) very negative; (-) negative; (0) neutral; (+) positive; or (++) very positive. 

The SWD presents the aggregate results of the consultation activities using bar charts showing 
the average scores of responses from each stakeholder group. The average scores do not account 
for ‘don’t know/no opinion’ answers. The data labels of the bar charts display the average score 
first, then the corresponding number of respondents in brackets. 

In order to assess the reliability of primary data, Figure 12 provides an overview of the level 
of knowledge in the field of digital public services, as well as the knowledge of the EIF among 
the participants who contributed to the consultation activities. Overall, stakeholders reflect a 
high level of knowledge, strengthening the reliability of the primary data collected. 

Overview of the level of knowledge of respondents 

On average, respondents have a good level of knowledge in the field, noting they are 
familiar to a great extent with digital public services and interoperability. EU and non-EU 
citizens report a relatively lower level of knowledge, but even in this case, the respondents 
are  familiar  at  least  to  some  extent  (3.50)  with  the  field  of  digital  public  services  and 
interoperability. 

Figure 12: Knowledge of digital public services and interoperability (breakdown by 
group of stakeholders; average score and number of respondents) 

Score:  (1)  not  at  all;  (2)  to  a  limited  extent;  (3)  to  some  extent;  (4)  to  a  great  extent;  or  (5)  completely.  
Note:  Averages  do  not  account 
for  respondents  answering  ‘don’t  know/no  opinion’  (DK/NO).  
Source: Study supporting the impact assessment for a future interoperability strategy 

218  

For each question, the respondent also had the possibility to select the answer ‘don’t know/no opinion’. 

55 

 
 
                                                           
Furthermore, respondents across almost all groups are, on average, familiar to a great extent 
with the EIF (see Figure 13). The only exception concerns respondents among EU and non-
EU citizens who have a relatively lower level of familiarity with the EIF, but they are still, 
on average, familiar to some extent with the EIF. 

Figure 13: Knowledge of the EIF (breakdown by group of stakeholders; average score 
and number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO). 
Source: Study supporting the impact assessment for a future interoperability strategy. 

Secondary data 

The impact assessment builds primarily on the evaluations of the ISA² programme and the 
EIF219. These evaluations were conducted simultaneously with the impact assessment as key 
sources to assess achievements accrued so far in the field of public sector interoperability in the 
EU as well as lessons learnt. 

The EIF Evaluation relies on an extensive review of available secondary data. One of the main 
sources  of  data  is  the  National  Interoperability  Framework  Observatory  (NIFO)  action 
supported by the ISA² programme and its outputs that are available in the Joinup collection 
with the same name220. The particular sources developed as part of NIFO and used as evidence 
feeding into the evaluation of the EIF include: 

  The  Digital  Public  Administration  Factsheets,  contributing  to  the  assessment  of  the 
effectiveness  of  the  EIF  by  providing  an  overview  of  the  development  of  national 
interoperability frameworks or similar strategies221; 

  The  2020  Report  on  the  State-of-Play  of  Digital  Public  Administrations  and 
Interoperability, detailing the progress made by the EU Member States in enhancing the 
provision of digital and interoperable public services (the report also covers nine additional 

219  

220  

221  

See SWD/2021/965 final (Staff working document), COM/2021/965 final (Report), CEPS (2021), Study 
supporting the final evaluation of the ISA² programme. Available at: https://op.europa.eu/en/publication-
detail/-/publication/afa4297a-0acc-11ec-adb1-01aa75ed71a1/language-en/format-PDF/source-
229005953 
CEPS  (2021),  Study  supporting  the  evaluation  of  the  implementation  of  the  EIF.  Available  at 
https://op.europa.eu/en/publication-detail/-/publication/29d694d4-4696-11ec-89db-
01aa75ed71a1/language-en/format-PDF/source-search 
For  further  details  please  see:  National  Interoperability  Framework  Observatory,  Knowledge  Centre, 
available  on  Joinup:  https://joinup.ec.europa.eu/collection/nifo-national-interoperability-framework-
observatory/knowledge-centre 
The  factsheets  can  be  consulted  on  Joinup:  https://joinup.ec.europa.eu/collection/nifo-national-
interoperability-framework-observatory/digital-public-administration-factsheets 

56 

 
 
                                                           
countries:  Iceland,  Liechtenstein,  Norway,  Switzerland,  the  United  Kingdom,  Ukraine, 
Montenegro, Turkey and the Republic of North Macedonia)222. This Report contributes to 
the assessment of the effectiveness and coherence of the EIF by providing an overview of 
key developments across the EU and the level of implementation of the EIF; 

  The 2019 EIF Monitoring Mechanism, tracking the implementation  of the EIF  and its 
principles, model and recommendations across the EU, thus contributing to the assessment 
of the effectiveness of the EIF223; 

  Other reports and documents developed as part of NIFO and available on Joinup, including 
the  Report  on  Public  Administrations’  Digital  Response  to  Covid-19  in  the  EU224, 
contributing in particular to assessing the relevance of the EIF. 

  The webpages of ISA² actions and the ISA² Rolling Work Programmes. 
Relevant indicators measuring digitalisation in the public sector were considered from a 
variety of databases, namely: Eurostat, the World Bank Worldwide Governance Indicators, and 
the Organisation for Economic Co-operation and Development (OECD) Digital Infrastructure 
indicators.  In  addition,  the  datasets  related  to  the  e-Government  Benchmark  reports225  were 
used, contributing in particular to the assessment of the effectiveness and efficiency of the EIF 
and for the impact assessment contributing in particular to quantify the existing problems and 
the assessment of the potential impacts stemming from the policy options considered for this 
impact assessment; 

Official  documents  such  as  communications,  regulations,  directives  and  decisions  were 
reviewed contributing in particular to the assessment of the coherence and EU added value of 
the  EIF.  In  addition,  official  national-level  documents  such  as  national  interoperability 
frameworks were also reviewed, primarily as part of the analysis of the effectiveness of the EIF, 
while also contributing to understanding the EU added value of the framework. For the impact 
assessment these data sources contributed across the study, from depicting the policy context 
to comparing the policy options considered for this impact assessment; 

In  building  the  evidence  base,  the  relevant  literature  in  the  field  of  interoperability  was 
reviewed and considered throughout the evaluation process. The key insights from literature 
contribute  particularly  to  the  assessment  of  the  relevance  of  the  EIF  and  for  the  impact 
assessment particular the existing problems in the field of public sector interoperability. 

Data validation 

To ensure the robustness of evidence, the collected data were validated via triangulation. Tool 
#4 of the Better Regulation Toolbox on evidence-based better regulation defines triangulation 
as ‘the application and combination of several research methodologies in the study of the same 
phenomenon’226. Validation requires checking ‘whether the findings of  a study  are true and 

222  

223  

224  

225  
226  

Bouhend,  A.,  et  al.  (2020),  ‘Report  on  the  State-of-Play  of  Digital  Public  Administrations  and 
Interoperability’, European Commission. Available at: 
https://joinup.ec.europa.eu/sites/default/files/news/2020-10/SC263_D04.02_State-of-
play%20report%20on%20digital%20public%20administration%20and%20interoperability%202020_v
FINAL.pdf 
The underlying data was shared with the Study Team by DIGIT.D2 to facilitate the analysis. An overview 
of  the  EIF  Monitoring  Mechanism  is  available  at:  https://joinup.ec.europa.eu/collection/nifo-national-
interoperability-framework-observatory/eif-monitoring-mechanism 
Charay  et.  al.  (2021),  Report  on  Public  Administrations’  Digital  Response  to  COVID-19  in  the  EU, 
Publications Office of the European Union. Available at: 
https://joinup.ec.europa.eu/collection/nifo/report-state-play-digital-public-administration-and-
interoperability 
The eGovernment Benchmark datasets are available at: https://digital-agenda-ata.eu/datasets/e-gov 
Better 
https://ec.europa.eu/info/sites/default/files/file_import/better-regulation-toolbox-4_en_0.pdf 

Evidence-based 

Regulation 

regulation, 

Toolbox 

(2017), 

better 

Tool 

#4 

57 

 
                                                           
certain: ‘true’ in the sense that research findings accurately reflect the situation; and ‘certain’ 
in the sense that research findings are supported by evidence’227. To ensure that the findings are 
supported  by  well-founded  evidence,  data  were  collected  from  multiple  sources  and  using 
different  tools  to  analyse  the  evaluation  criteria  and  questions.  This  approach  allows  for 
increasing confidence in collected data, comparing and contrasting findings, and providing a 
clearer  understanding  of  the  issues  at  stake.  The  analysis  relies  on  three  different  types  of 
triangulation to provide a solid basis for drawing robust conclusions: 

  Triangulation  of  data,  relying  throughout  the  impact  assessment  on  data  collected 
from multiple sources and stakeholders. 
  Triangulation  of  methods,  based  on  data  collected  via  at  least  two  different  data 
collection  methods  (e.g.,  interview,  targeted  questionnaire,  public  consultation,  desk 
research). 
  Triangulation of evaluators. Several members of the Core Study Team and Support 
Team were involved in data collection activities. In addition, each element of the impact 
assessment  was  addressed  by  at  least  two  members  of  the  Core  Team.  Hence, 
conclusions were agreed upon by at least two researchers. 

Methods 

The evaluation and impact  assessment combines  both  quantitative and qualitative data, thus 
relying  on  a  variety  of  methods  and  tools  to  analyse  the  collected  data.  In  particular,  the 
qualitative information was aggregated, compared and summarised to substantiate the evidence 
feeding into evaluation and impact assessment. To analyse the findings from the consultation 
activities, the Study Team employed descriptive statistics. To complement these methods, other 
specific quantitative methods were used to evaluate the effectiveness, efficiency and coherence 
of the framework: 
Cost-benefit analysis (CBA)228 

CBA  is  a  method  of  comparing  the  costs  and  benefits  associated  with  a  certain  initiative, 
typically a government policy, programme or project, in order to assess whether the initiative 
is expected to bring about a net improvement. A key feature of CBA is that all costs and all 
benefits are quantified and expressed in monetary terms, and are adjusted for the ‘time value of 
money’  (through  some  discounting  mechanism),  so  that  all  benefits  and  costs  that  occur  at 
different points in time are expressed on a common basis in terms of their ‘present value’. The 
main limitations of CBA concern the difficulties in attributing a monetary value to items for 
which no market exists and the fact that certain non-monetisable effects simply cannot be taken 
into  consideration.  CBA  was  used  to  assess  of  the  potential  impacts  stemming  from  the 
proposed policy options. 
The standard cost model (SCM)229 

SCM aims at assessing administrative costs imposed by rules or policies inter alia on businesses 
and public administrations. It is based on the identification of the basic components of a rule, 
the Information Obligations, whose costs for the addressees can be measured and quantified. 

227  

228  

229  

(2017),  Tool  #59  Methods 

Triangulation: Establishing the Validity of Qualitative Studies, Lisa A. Guion, David C. Diehl, and Debra 
McDonald, 2011. 
Better  Regulation  Toolbox 
https://ec.europa.eu/info/sites/default/files/file_import/better-regulation-toolbox-59_en_0.pdf 
European Commission (2017), ‘Tool #60. The standard cost model for estimating administrative costs’, 
in the Better Regulation ‘Toolbox’, available at https://ec.europa.eu/info/sites/info/files/better-regulation-
toolbox_2.pdf;  SCM  Network  (2005),  ‘The  International  SCM  Manual;  Measuring  and  Reducing 
Administrative Burdens for Businesses’, available at: 
http://www.administrativeburdens.com/filesystem/2005/11/international_scm_manual_final_178.doc 

to  assess  costs  and  benefits, 

58 

 
                                                           
An Information Obligation is a specific duty to gather, process or submit information to the 
public authority or a third party. The SCM is employed to measure the costs borne in terms of 
days spent by national public administrations to transpose the EIF into national frameworks as 
well  as  the  costs  that  would  be  borne  by  public  administrations  when  implementing  the 
proposed policy options. 
The Multi-criteria analysis (MCA)230 

MCA is a technique for comparing and ranking different courses of action that yield multiple 
outcomes expressed in different units of accounts (i.e. some are expressed in monetary terms, 
others in physical units of different nature, e.g. number of lives saved and quality of gaseous 
emissions). In some respects, it can be said that MCA goes one step further than CBA, as it 
gives  explicit  recognition  to  the  fact  that  a  variety  of  both  monetary  and  non-monetary 
objectives (i.e. social, environmental, technical, economic, and financial) may influence policy 
decisions.  At  the  same  time,  this  kind  of  analysis  inevitably  includes  a  higher  degree  of 
subjectivity  compared  to  CBA,  especially  regarding  the  weighting  and  ranking  of  different 
criteria.  Therefore,  unlike  CBA,  a  basic  feature  of  MCA  is  its  implementation  through 
participatory processes, involving policymakers, experts (e.g. through Delphi panel method), 
stakeholders, etc. This method was employed to compare the proposed policy options based on 
their effectiveness, efficiency, coherence and feasibility. 

Text mining 

Text  mining  is  a  technique  referring  to  a  set  of  data-processing  operations  that  extract 
knowledge according to a criterion of novelty or similarity in texts. In practice, this technique 
is based on an algorithm that counts the number of times a specific reference is made in a given 
text corpus. In this evaluation, text mining is used to evaluate the degree of external coherence 
of  the  EIF,  together  with  its  principles,  interoperability  layers  and  conceptual  model.  In 
particular,  this  approach  allows  checking,  throughout  the  corpus  of  EU  legislation  and 
Commission communications, the extent to which the EIF and its components are taken into 
consideration by other EU initiatives in the field of interoperability that were adopted since the 
first version of the EIF (2004). 

This analysis can be done in two steps: 

  Searching through the corpus of EU legislation and Commission communications since 
the adoption of the first version of the EIF (2004) to obtain an overview of the number 
of  times  the  EIF  and  keywords  related  to  the  EIF  are  mentioned  in  official  EU 
documents; 

  Taking a more in-depth look at a selection of relevant official documents and conducting 
a more granular analysis to understand which elements of the EIF are referenced. 
While  the  first  approach  provides  a  quantifiable  overview  of  the  references  to  the  EIF  in 
different  pieces of  legislation and communications over time, the second approach provides 
more insights into the take-up of the EIF and its specific elements. 

To quantify the number of mentions of the EIF and keywords related to the EIF in pieces of EU 
legislation,  the  Study  Team  relied  on  an  extensive  dataset,  the  ‘CEPS  EurLex  dataset’, 
customised to facilitate text mining231. The dataset contains 142,036 pieces of EU legislation 
adopted between January 1952 and August 2019, representing almost the entire corpus of the 
digitally available EU legal acts from this timeframe. The dataset covers three types of legally 

230  

231  

For an excellent review, please see Department for Communities and Local Government, Multi-criteria 
analysis: a manual, London, January 2009. 
Borrett, Camille; Laurer, Moritz, 2020, ‘The CEPS EurLex dataset: 142,036 EU laws from 1952-2019 
with full text and 22 variables’, https://doi.org/10.7910/DVN/0EGYWY, Harvard Dataverse. 

59 

 
                                                           
binding  acts  adopted  by  the  EU  institutions:  regulations,  directives,  decisions,  and 
implementing and delegated acts. When it comes to quantifying the number of references to the 
EIF and keywords related to the EIF in Commission communications, the analysis was run for 
documents including green and white papers, communications, reports (categorised as ‘DC’ on 
EurLex)232. Importantly, the analysis excluded self-references to the EIF Communications. 

The datasets were used to search for specific keywords in the timeframe 2004 (the year of the 
adoption of the first EIF) to August 2019. In addition, to cover the period from August 2019 
and April 2021, the EurLex website was used. The keywords, selected based on their connection 
to  the EIF and the extent  to  which they would be expected to  yield results  in  a text mining 
exercise, were grouped into three clusters as follows233: 

  Cluster 1: European Interoperability Framework. This cluster refers exclusively to the 
EIF and  contains  three keywords: ‘European  Interoperability Framework’, ‘EIF’ and 
‘interoperability framework’; 

  Cluster 2: Interoperability layers. This cluster gathers together the following keywords: 
‘semantic interoperability’, ‘technical interoperability’, ‘organisational interoperability’ 
and ‘legal interoperability’; 

  Cluster 3: Key interoperability concepts. This cluster contains the following keywords: 
‘once-only’ (keyword related to the ‘once-only’ principle), ‘digital by default’, ‘privacy 
by design’, ‘user-centricity’, ‘re-use of data’, ‘interoperability principle’, ‘interoperable 
e-Government service’. 

The datasets of EU legislation and communications were searched for the keywords, resulting 
in  a  list  of  pieces  of  legislation  and  communications  where  the  keywords  were  found.  The 
results were thus summarised by year and number of mentions per cluster of keywords. 

Econometric modelling 

The  econometric  model  statistically  tests  pre-determined  assumptions  regarding  the 
relationship  between  two  variables.  The  econometric  tool  is  more  powerful  than  a  simple 
correlation  because  it  unilaterally  determines  and  quantifies  the  impact  of  an  independent 
variable  on  a  dependent  variable.  By  isolating  the  effects  of  the  variables  of  interest  (i.e. 
independent variables) on the main variable (i.e. the dependent variable), the model specifies 
the  statistical  relationship  existing  (or  not)  between  two  variables.  In  order  to  test  for  the 
effectiveness of the EIF, the model is used to estimate the effects of adopting an EIF-based NIF 
and improving the quality of digital public services on the share of citizens using the Internet 
to interact with their public administrations. The model is estimated by relying on a regression 
with fixed effects: 

 𝐼𝑛𝑑𝑒𝑥_𝑃𝐴𝑖,𝑡 =  𝑐 + 𝛽1 × 𝐼𝑛𝑑𝑒𝑥_𝑒𝑔𝑜𝑣𝑖,𝑡 + 𝛽2 × 𝑁𝐼𝐹𝑖,𝑡 + 𝛽3 × 𝑅&𝐷𝑖,𝑡 + 𝛽4 × 𝐺𝐷𝑃_𝑐𝑎𝑝𝑖,𝑡 +
𝛽5 × 𝑔𝑜𝑣_𝑒𝑓𝑓𝑖,𝑡 + 𝛽6 × 𝑖𝑛𝑓𝑟𝑎𝑖,𝑡 + 𝑢𝑖 + 𝜀𝑖,𝑡,                                                                    (Eq. 1) 

where the indexes 𝑖 and 𝑡 denote the country and the year, respectively. In the model, 𝑐 is the 
intercept, ∑
 are the coefficients associated to each variable accounted for in the model, 

𝛽6
𝑗=1 𝑗

𝑢𝑖  defines  the  fixed  effects  per  country,  and  𝜀𝑖,𝑡  is  the  error  term.  This  model  controls  for 
additional country-level variables that are difficult to measure or observe and that do not vary 
over  time  (such  as  cultural  factors).  Fixed  effects  isolate  the  impacts  of  time-invariant 
characteristics so that we can assess the net effect of the predictors on the outcome variable. In 

232  

233  

Hradec,  Jiri,  2021,  the  database  is  part  of  the  ‘Trends  in  Global  Governance  and  Europe’s  Role’ 
(TRIGGER) project. 
An additional keyword, ‘conceptual model’ (related to the EIF conceptual model for integrated public 
services provision), was dropped from the analysis as it did not yield results related to the EIF. 

60 

 
                                                           
other words, this first model estimates an additional and single effect (𝑢𝑖) per country, but the 
latter is assumed to be common across time. 

Several  additional  variables  (control  variables),  which  vary  over  the  time,  are  taken  into 
consideration  in  order  to  isolate  the  effects  of  the  variables  of  interest  (i.e.  NIF  and  e-
Government  index).  These  control  variables  ensure  that  other  potential  factors  are  properly 
captured in the analysis:234 i) the real GDP per capita captures any differences that may stem 
from  the  economic  development  of  a  country;  ii)  the  level  of  infrastructure  development 
accounts for effects that may result from limited access to the Internet; iii) public spending on 
research and development related to general public services considers all the effects that can 
arise from discrepancies between countries’ expenditures in innovative public services; and iv) 
government effectiveness measures any effects that may arise from the perceived qualit13y of 
public services. All the variables used in the model are described in Table 3. Moreover, Table 
4 provides descriptive statistics of the analysed variables. 

Table 3: Description of variables used in the econometric analysis 

Variable 

Description 

Type 

Source 

Dependent variable 

Online  public 
services 
use 
(index_PA) 

Index  computed  as  an  average  of 
the 
percentage  of  individuals  (aged  between  16 
and  74)  using  the  Internet  to  interact  with 
public  authorities,  obtain  information  from 
public 
authorities’  websites,  download 
official  forms  and  submit  completed  forms 
(over the last 12 months). 

Independent variables 

Percentage 

Eurostat 

e-Government 
indicators 
(index_egov) 

Index  computed  as  an  average  of  User 
centricity,  Transparency,  Citizen  Mobility, 
Business Mobility and use of Key enablers. 

Percentage 

e-Government 
Benchmark 

National 
Interoperabili
ty Framework 
(NIF) 

Indicator  showing  whether  the  country  has 
implemented an EIF-based NIF. The value 1 
is assigned from the year in which the country 
implemented the NIF onward; otherwise, the 
value 0 is assigned in years where the country 
does/did  not  have  a  NIF.  However,  the 
analysis  differentiates  between  a  complete 
adoption  and  an  incomplete  adoption  of  the 
EIF. 
two  variables  are 
considered: 

In  particular, 

Dummy 0-1  Author’s 

elaboration 
based  on 
findings 
presented 
Annex VI.2 

the 

in 

234  

Several econometric studies conducted in the field of e-Government services and, more broadly, digital 
economy  rely  on  similar  variables.  See,  for  instance,  Lakka  et  al.  (2013),  What  drives  eGovernment 
growth? An econometric analysis on the impacting factors, Int. J. Electronic Governance Vol. 6, No. 1; 
Sakari Taipale (2013), The use of e-government services and the Internet: The role of socio-demographic, 
economic  and  geographical  predictors,  Telecommunications  Policy  37  (2013)  413–422;  Zaho  et  al. 
(2014), E-government development and the digital economy: a reciprocal relationship, Internet Research, 
Vol.  25  Iss  5  pp.  734  -  766;  and  Lakka  et  al.  (2015),  Cross-national  analysis  of  the  relation  of 
eGovernment maturity and OSS growth, Technological Forecasting & Social Change 99 132–147. 

61 

 
                                                           
Variable 

Description 

Type 

Source 

  NIF ‘complete’, which encompasses 
countries  that  have  adopted  an  EIF-
based NIF. 

  NIF 

which 
‘incomplete’, 
encompasses  countries 
that  have 
endorsed an EIF-based NIF, as well as 
a partial  adoption of the EIF in  their 
NIF. 

Control variables 

R&D 
government 
(R&D_gov) 

Government  spending  to  support  applied 
research  and  experimental  development 
related to general public services undertaken 
by  non-government  organisations  such  as 
research institutes and universities. 

Millions  of 
EUR 

Eurostat 

General 
public services 
(Gen_pub_ser
v) 

Government  spending  to  support  broader 
activities,  that  is  executive  and  legislative 
organs,  financial  and  fiscal  affairs,  external 
affairs; 
foreign  economic  aid;  general 
services;  basic  research;  R&D  related  to 
general  public  services;  general  public 
services; public debt transactions, transfers of 
a  general  character  between  different  levels 
of government. 

Millions  of 
EUR 

Eurostat 

Real  GDP per 
capita 
(GDP_cap) 

Government 
effectiveness 
(gov_eff) 

variable 

captures 

This 
economic 
discrepancies between countries. It is used as 
a  proxy  for  the  development  in  a  country’s 
material living standard. 

Effectiveness 

captures 
Government 
perceptions of the quality of public services, 
the quality of the civil service and the degree 
of its independence from political pressures, 
the  quality  of  policy 
formulation  and 
implementation,  and  the  credibility  of  the 
government’s commitment to such policies. 

Eurostat 

World Bank235 

Ratio of real 
GDP  to  the 
average 
population 

the 

Estimate 
gives 
country’s 
score  on  the 
aggregate 
indicator  in 
units  of  a 
standard 
normal 
distribution. 
The 
index 
ranges  from 
approximate

235  

Kaufmann Daniel, Aart Kraay and Massimo Mastruzzi (2010), ‘The Worldwide Governance Indicators: 
Methodology and Analytical Issues’, World Bank Policy Research Working Paper No. 5430. 

62 

 
                                                           
Variable 

Description 

Type 

Source 

Infrastructure 
(infra) 

the 

level  of 
captures 
This  variable 
infrastructure to access the Internet for each 
country.  Total  communication  refers  to  the 
total  access  to  telephone  lines,  total  fixed 
broadband, mobile subscribers. 

Source: Supporting study for the evaluation of the EIF 

Table 4: Descriptive statistics of the variables 

OECD 

-2.5 

to 

ly 
2.5. 

Total 
communicat
access 
ion 
per 
paths 
100 
inhabitants 

Variables 

Observations  Mean 

Standard 
deviation 

Min 

Max 

Online public services use 
(index_PA) 

e-Government indicators 
(index_egov) 

249 

256 

42.46 

17.25 

4 

80 

60.74 

16.53 

22 

95 

National 
Interopera
bility 
Framework 
(NIF) 

NIF 
‘incomplete’ 

256 

0.30 

0.46 

NIF 
‘complete’ 

256 

0.34 

0.47 

R&D government (R&D) 

205 

100.93 

162.25 

0 

0 

0 

1 

1 

762 

General public services 
(Gen_pub_serv) 

Real GDP per capita 
(GDP_cap) 

Government effectiveness 
(gov_eff) 

217 

256 

256 

31 289.51  47 558.62  532.8 

189 065 

28 071.72  18 220.32  5 350 

83 640 

1.14 

0.58 

-0.31 

2.22 

Infrastructure (infra) 

189 

187.72 

26.31 

119.65 

242.85 

Source: 
https://op.europa.eu/en/publication-detail/-/publication/29d694d4-4696-11ec-89db-01aa75ed71a1/language-
en/format-PDF/source-search. 
Before carrying out the analysis, it is important to ensure that the independent variables (i.e. the 
explanatory variables) are not highly correlated with each other. The simultaneous introduction 
of  two  highly  correlated  variables  in  the  model  (i.e.  collinearity  issue)  could  lead  to  erratic 
results.  For  this  purpose,  the  correlations  should  be  lower  than  0.8236.  Table  5  presents  the 
pairwise correlations between the variables used in the analysis. Beyond ensuring the soundness 

236  

This  criterion  is  suggested  in  Kennedy  P.A.  (2003),  A  Guide  to  Economics,  5th  ed.,  MIT  Press, 
Cambridge, MA. 

63 

 
                                                           
of the results, the table also shows how variables are linked to one another (whether they evolve 
in the same or opposite direction, or whether they are independent). In Table 5, three values 
illustrate important positive correlations: 

  The correlation between the online interaction between citizens and public administrations 
(index_pa)  and  government  effectiveness  (gov_eff).  However,  the  online  interaction 
between citizens and public administrations variable is not an independent variable, but a 
dependent  variable  (i.e.  the  explained  variable).  Therefore,  the  two  variables  are  not 
simultaneously introduced in the explanatory side of the model. 

  The correlation between the two binary variables (NIF ‘complete’ and NIF ‘incomplete’’). 
It is no surprise, since these two variables are built on similar criteria, one following stricter 
assumptions than the other. These two variables are introduced alternatively (see regressions 
(1), (2), (3) and (4)) in order to test for the soundness of the results. Furthermore, the low 
values  of  the  correlation  with  the  other  variables  can  be  explained  by  the  nature  of  the 
variable, which is binary. 

  The correlation between the government effectiveness (gov_eff) and the real GDP per capita 
(GDP_cap). Although the value is slightly below 0.8, additional regressions are performed 
to ensure the robustness of the results when the two variables are dropped from the model 
(see Table 5, regressions (1), (2), (3) and (4)). 

Table 5: Correlation matrix 

index
_pa 

index_
egov 

R&D_
gov 

Gen_
pub_s
erv 

GDP
_cap 

infra 

gov_eff  NIF 

‘co
mpl
ete’ 

NIF 
‘inc
om
plet
e’ 

index_pa 

1 

index_ego
v 

0.57* 

1 

R&D_gov 

-0.23* 

-0.29* 

1 

Gen_pub_ 

-0.03 

-0.03 

0.35* 

1 

serv 

GDP_cap 

0.65* 

0.24* 

-0.06 

0.13* 

1 

infra  

0.17* 

0.14 

0.20* 

0.30* 

0.40* 

1 

gov_eff 

0.78* 

0.48* 

-0.33 

0.15* 

0.77* 

0.37* 

1 

NIF 
‘complete’ 

NIF 
‘incomplet
e’ 

-0.05 

0.12 

0.15* 

-0.18* 

-0.16*  0.03 

-0.18* 

1 

-0.05 

0.10 

0.14* 

-0.01 

-0.22 

0.09 

-0.17* 

1 

0.77
* 

Notes: All significance levels are at the p>0.05, except where denoted by: *=p<0.05. 
Source: Supporting study for the evaluation of the EIF 
The analysis tests two hypotheses: 

64 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
  NIFs foster the use of online public services; 
  Better  e-Government  services  encourage  citizens  to  interact  with  their  public 

administration via the Internet. 

By testing these relationships using a fixed-effects model and ensuring that the independent 
variables meet the assumptions required to run the model, the results of the analysis support the 
above two hypotheses. Table 6 presents the results of the main model specification when the 
two variables, NIF ‘complete’ and NIF ‘incomplete’, are alternatively included as explanatory 
variables.  To  ensure  the  soundness  of  the  results,  additional  results  from  different  model 
specifications are provided in Table 6. 

Table 6: Results of fixed effects model 

Variables 

index_PA 

Comparison of NIF variables 

NIF ‘incomplete’ 

NIF ‘complete’ 

index_egov 

R&D_gov 

GDP_cap 

gov_eff 

infra 

Gen_ pub_serv 

Constant 

Number of countries 

5.110*** 

(1.438) 

0.146** 

(0.055) 

0.010** 

(0.005) 

4.15E-4 

(2.86E-4) 

-2.011 

(7.366) 

-0.133 

(0.129) 

48.98** 

(18.48) 

25 

2.367* 

(1.357) 

0.169** 

(0.066) 

0.013*** 

(0.005) 

5.06E-4 

(3.63E-4) 

-1.729 

(7.659) 

-0.104 

(0.134) 

40.15* 

(21.56) 

25 

Notes: Robust standard errors in parentheses 
*** p<0.01, ** p<0.05, * p<0.1 
Source: Supporting study for the evaluation of the EIF 
Table  7  also  presents  the  results  of  the  analysis  when  new  countries  are  introduced  (see 
regressions  (5)  and  (6)).  This  is  essential  in  supporting  the  robustness  of  the  results. 
Specifically,  regression  (5)  drops  the  variable  capturing  the  level  of  infrastructure,  which 

65 

 
 
 
 
 
 
constrains the model to 168 data observations. By removing this variable, four new countries 
can  be  introduced  into  the  analysis.  Finally,  regression  (6)  replaces  the  variable  capturing 
government  spending  on  research  and  development  to  support  the  development  of  general 
public services with a broader variable, namely total government spending on general public 
services, thus introducing an additional country into the analysis. 

Table 7: Robustness tests (fixed effects model) 

Variables 

index_PA 

Potential 
collinearity problem 
(i) 

Potential collinearity 
problem (ii) 

Introduction of new 
countries 

NIF ‘incomplete’  5.118*** 

(1.411) 

5.436*** 

(1.390) 

3.629** 

5.229*** 

(1.376) 

(1.431) 

NIF ‘complete’ 

2.472* 

(1.317) 

2.654* 

(1.296) 

index_egov 

0.145** 

0.169** 

0.184*** 

0.219*** 

0.118** 

0.143*** 

(0.055) 

(0.066) 

(0.058) 

(0.067) 

(0.046) 

(0.047) 

R&D_gov 

0.010** 

0.013*** 

0.010** 

0.014*** 

9.51e-05 

(0.005) 

(0.005) 

(0.005) 

(0.005) 

(0.010) 

GDP_cap 

4.31E-4 

33.05** 

(2.75E-4) 

(15.25) 

4.28E-4 

4.28** 

(3.2E-4) 

(2.91E-4) 

gov_eff 

-3.577 

-3.620 

-6.212 

-1.657 

(7.010) 

(7.331) 

(5.458) 

(6.244) 

infra 

-0.136 

-0.107 

-0.133 

-0.102 

(0.121) 

(0.127) 

(0.133) 

(0.139) 

Gen_ pub_serv 

-0.123 

(0.095) 

2.97E-5 

(9E-5) 

Constant 

46.41** 

-37.94* 

61.35*** 

54.72** 

29.56** 

48.31*** 

(21.33) 

(23.47) 

(18.35) 

(20.81) 

(12.82) 

(15.84) 

Number of 
countries 

25 

25 

25 

25 

29 

26 

Notes: Robust standard errors in parentheses 
*** p<0.01, ** p<0.05, * p<0.1 
Source: Supporting study for the evaluation of the EIF 

66 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Main limitations 

The  mix  of  primary  data  and  information,  obtained  through  consultation  activities,  and 
secondary data and information, mainly obtained through desk research, was crucial in allowing 
to  draw  conclusions  and,  more  importantly,  to  provide  estimates  of  costs  and  benefits. 
Nevertheless, the following limitations may affect the main findings: 

1.  The scarcity and difficult comparison of data for assessing the costs and benefits of 
transposing the EIF into national frameworks. The voluntary nature of the framework 
reflects a heterogeneous uptake of the framework across the EU, limiting the availability 
of  data  and  making  the  comparison  between  countries  difficult,  as  the  framework  is 
adopted  on  an  ad  hoc  basis.  Furthermore,  the  implementation  of  the  EIF  has  not 
followed the same timeline in the different countries. To overcome this limitation, the 
Study Team contacted the national public authorities that responded to the consultation 
activities and indicated their availability to participate in a follow-up interview in order 
to gain insight on the time spent (in terms of person-days) on the transposition of the 
EIF, or part of it, into their national framework. 

2.  The overarching scope of the EIF hinders the assessment of the direct costs and 
benefits stemming from its implementation. Costs and benefits deriving from the EIF 
do not just pertain to the implementation of the framework, as costs and benefits depend 
on  the  levels  of  digitalisation  of  the  Member  States  or  on  the  different  degrees  of 
centralisation  and  varying  organisational  structures  involved  in  delivering  public 
services. The costs related to interoperability initiatives are borne at different levels of 
administration in several countries. 

3.  The conclusions drawn in  the  Impact  Assessment  are built  upon the EIF  Evaluation, 
which evaluates the latest implementation of the EIF (i.e. from 2017 to 2020). However, 
the  framework  seeks  to  achieve  overarching  goals  and  impacts  that  require  time  to 
show results. 

Another limitation may be the potential ‘consultation fatigue’ of respondents. In particular, 
this Impact Assessment was carried out in a context in which the feedback of stakeholders may 
have been sought for several relevant developments: the end of the ISA² programme, the rollout 
of the new Digital Europe Programme, the 2030 Digital Compass and the new policy initiatives 
related to digitalisation in the EU. 

67 

 
Annex III. Evaluation framework for the EIF evaluation 

Evaluation 
questions 

Success/judgment 
criteria 

Evaluation criterion #1: Effectiveness 

Indicators 

Data sources 

Data collection / 
analysis methods 

  To what extent 
has the EIF 
reached its 
objectives? 

  What are the 
factors that 
have 
influenced 
positively and 
negatively the 
achievement 
observed? 
What obstacles 
remain? 

  Are there aspects 

or 
recommendations 
of the EIF that 
are more or less 
effective than 
others, and if so, 
what lesson can 

  Degree of alignment 
between actual and 
expected results and 
objectives of the 
EIF.237 

 

Impact of external 
factors on the 
achievements of the 
EIF. 

  Measurement of the 

indicators 
summarising the 
outputs of the EIF. 

 

Identification of 
direct and indirect 
benefits of the EIF 
and their distribution 
across stakeholders’ 
groups. 

  Share of stakeholders confirming 
the alignment between actual and 
expected results of the EIF. 

  Share of stakeholders confirming 

the alignment between the 
objectives and actual results of the 
EIF. 

  Share of stakeholders identifying 
external factors contributing 
to/jeopardising the achievements of 
the EIF. 

  Qualitative assessment of the 
alignment between objectives, 
expected and actual results of the 
EIF. 

  Qualitative assessment of factors 
contributing to/jeopardising the 
achievements of the EIF. 

  Quantitative assessment of 

performance indicators of outputs. 

  Primary 

information on 
actual results and 
contribution to the 
EIF’s objectives 
from the following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

 

 

ISA² action 
owners. 

ISA² solution 
users – European 
Commission. 

ISA² solution 
users – Member 
States. 

  Stakeholders 

responsible for 

  Desk research. 

 

Interviews with the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

ISA² action 
owners. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Online surveys/ 

written 
questionnaires 

237  

Results include outputs, outcomes, and impacts, with the caveat that impacts are longer term results that may only become noticeable after a more significant period of 
time. 

68 

 
                                                           
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

be drawn from 
this? 

  What are the 

benefits of the 
EIF 
implementation 
and how 
beneficial are 
they for the 
various 
stakeholders? 

  Qualitative assessment of direct 
and indirect benefits of the EIF. 

  Qualitative assessment of the 

distribution of EIF benefits across 
stakeholders’ groups. 

  Assessment of costs savings 
stemming from the EIF. 

linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Standardisation 
organisations. 

  Experts. 

 

Indirect 
beneficiaries and 
wider public. 

  Secondary 

information from 
operational 
documents and 
other official 
documents, such as: 

  EIF Monitoring 
Mechanism 

  Digital 

Government 
Factsheets. 

  Digital Economy 
and Society 
Index. 

Data collection / 
analysis methods 

targeted to the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

 

 

ISA² action 
owners. 

ISA² solution 
users – European 
Commission. 

ISA² solution 
users – Member 
States. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Standardisation 
organisations. 

  Public consultation. 

69 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Evaluation criterion #2: Efficiency 

70 

Data sources 

  Study on 

Member States’ 
Single Digital 
Gateway 
Regulation 
readiness. 

  Data from the 
IMAPS 
assessments 
carried out as 
part of ISA². 

  OECD Open 
Government 
Data. 

  UN e-

Government 
Development 
Index (EGDI). 

 

IMD World Digital 
Competitiveness 
ranking. 

Data collection / 
analysis methods 

  Case studies 
(success 
stories/lessons 
learnt). 

  Quantitative 

assessment of 
responses to 
interviews and 
surveys (Likert 
scale). 

  Qualitative 

assessment of 
responses to 
interviews and 
surveys and data 
and information 
collected via desk 
research. 

  Multicriteria 
analysis. 

  Cost of non-Europe 
approach for cost 
savings. 

  Expert assessment. 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

  Comparison between 
benefits and costs of 
the EIF. 

  Differences in 
benefits among 
Member States. 

  Differences in 

compliance costs and 
administrative 
burdens among 
Member States. 

  Compliance costs linked to the 
implementation of the EIF. 

  Administrative burdens linked to 
the implementation of the EIF. 

  Qualitative comparison between 
the benefits detected under the 
‘effectiveness’ criterion and costs 
stemming from the EIF. 

  Comparative assessment of 

benefits detected in different 
Member States. 

  Comparative assessment of the 

compliance costs and 
administrative burdens incurred in 
different Member States. 

  Have the benefits 
been achieved at 
a reasonable 
cost? 

 

Is there 
evidence that 
the 
implementatio
n of the EIF 
has caused 
unnecessary 
regulatory 
burdens? 

 

If there are 
significant 
differences in 
costs (or 
benefits) between 
Member States, 
what is causing 
them? 

  Primary 

  Desk research. 

information on 
costs from the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Secondary 

information from 
operational 
documents and 
other official 
documents, such as: 

  EIF Monitoring 
Mechanism. 

 

Interviews with the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Online surveys/ 

written 
questionnaires 
targeted to the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

71 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

  Digital 

  Stakeholders 

Government 
Factsheets. 

  Digital Economy 
and Society 
Index. 

  Study on 

Member States’ 
Single Digital 
Gateway 
Regulation 
readiness. 

  Data from the 
IMAPS 
assessments 
carried out as 
part of ISA². 

  OECD Open 
Government 
Data. 

  UN E-

Government 
Development 
Index (EGDI) 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Case studies 
(success 
stories/lessons 
learnt). 

  Quantitative 

assessment of 
responses to 
surveys (Likert 
Scale). 

  Qualitative 

assessment of 
responses to 
interviews and 
surveys and data 
and information 
collected via desk 
research. 

 

IMD World Digital 
Competitiveness 
ranking. 

  Standard cost 

model. 

72 

 
Evaluation 
questions 

Success/judgment 
criteria 

Evaluation criterion #3: Coherence 

Indicators 

Data sources 

Data collection / 
analysis methods 

  To what extent 
are the elements 
of the EIF 
coherent between 
themselves? 
(internal 
coherence) 

  To what extent is 
the EIF and its 
governance 
consistent with 
other EU 
legislation or 
initiatives 
(external 
coherence)? 

Are there any 
inconsistencies, 
overlaps or gaps? 

  Degree of coherence 
between the three 
main components of 
the EIF (principles, 
layered 
interoperability 
model, conceptual 
model for 
interoperable public 
services) 

  Degree of coherence 
between the EIF and 
relevant existing EU 
initiatives and 
policies. 

o  E.g.: DSM 

Strategy, European 
Cloud initiative, 
EU eGovernment 
Action Plan 2016-
2020, Single 
Digital Gateway, 
the revised 
Directive on the 
re-use of Public 
Sector 
Information, 

  Share of stakeholders identifying 

  Primary 

  Desk research. 

inconsistencies/synergies/overlaps 
between the main components of 
the EIF 

  Qualitative assessment of the 
inconsistencies/synergies/ 
overlaps between the main 
components of the EIF 

  Share of stakeholders identifying 

inconsistencies/synergies/overlaps 
between the EIF and relevant 
existing EU initiatives, policies, 
and programmes. 

  Qualitative assessment of 

inconsistencies/synergies/overlaps 
between the objectives of the EIF 
and relevant existing EU 
initiatives, policies, and 
programmes. 

  Qualitative assessment of 
inconsistencies/synergies/ 
overlaps between the objectives of 
the EIF and proposed EU 
initiatives and programmes. 

  Qualitative assessment of 

inconsistencies/synergies/overlaps 

73 

information on 
external coherence 
from the following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

 

 

ISA² action 
owners. 

ISA² solution 
users – European 
Commission. 

ISA² solution 
users – Member 
States. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

 

Interviews with the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

ISA² action 
owners. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Online surveys/ 

written 
questionnaires 
targeted to the 
following 
categories of 
stakeholders: 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

INSPIRE 
Directive, eIDAS 
Regulation, Data 
Strategy. 

between the objectives of the EIF 
and EU data initiatives in digital 
healthcare and connected and 
automated vehicles. 

  Degree of coherence 
between the EIF and 
relevant existing EU 
programmes. 

o  E.g.: CEF, SRSP, 
Horizon 2020, 
ESF, ERDF, 
Country-Specific 
Recommendations, 
National Reform 
Programmes. 

  Degree of coherence 
between the EIF and 
proposed EU 
programmes and 
initiatives. 

o  E.g.: Digital 
Europe 
Programme, 
Horizon Europe. 

  Degree of coherence 
between the EIF and 
selected EU data 
initiatives in specific 

74 

  Standardisation 
organisations. 

  Experts. 

 

Indirect 
beneficiaries and 
wider public. 

  Secondary 

information from 
operational 
documents and 
other official 
documents, such as: 

  EIF 

Communication 
(COM/2017/134) 
and 
accompanying 
documents 
(Annexes to the 
Communication 
and Staff 
Working 
Documents). 

  Governance of 
ISA² and the 
EIF. 

 

 

 

ISA² action 
owners. 

ISA² solution 
users – European 
Commission. 

ISA² solution 
users – Member 
States. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Standardisation 
organisations 

  Public consultation. 

  EIF Monitoring 
Mechanism and 
relevant outputs 
of the NIFO 
action providing 

  Quantitative 

assessment of 
responses to 
interviews and 

 
Data collection / 
analysis methods 

surveys (Likert 
scale). 

  Qualitative 

assessment of 
responses to 
interviews and 
surveys and data 
and information 
collected via desk 
research. 

  Expert assessment. 

Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

sectors (e.g. the 
degree of coherence 
with respect to the 
announced data 
spaces in the fields 
of healthcare and 
mobility). 

an overview of 
interoperability 
initiatives 

  eGovernment 

Action Plan 2016 
– 2020 

  Study on the role 
of eGovernment 
and 
interoperability 
in the European 
Semester. 

  The study ‘The 

nature of 
interoperability 
in digital public 
services. 
Deconstructing 
interoperability’. 

  Mid-term review 
of the Digital 
Single Market 
Strategy. 

  Legal texts 
establishing 
other relevant 
programmes 

75 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

accompanying 
documents. 

  Legal texts 

devising other 
relevant EU 
policies and 
accompanying 
documents. 

 

Interim 
Evaluation of 
ISA². 

  Mid-term 

Evaluation of 
CEF. 

  Other interim/final 
evaluations of EU 
programmes and 
policies. 

Evaluation criterion #4: EU added value 

  What is the 

additional value 
resulting from 
the 
implementation 
of the EIF, 
compared to 
what could 

  Achievement of 
objectives that 
could not be 
otherwise attained 
with national or 
sub-national 
interventions. 

  Share of stakeholders confirming 
the need for an EU intervention to 
achieve the objectives of the EIF. 

  Share of stakeholders confirming 
that an EU intervention is able to 
achieve the objectives of the EIF at 
cost lower than costs of 

  Primary 

  Desk research. 

information on 
cross-border 
interoperability and 
EU added value 
from the following 

 

Interviews with the 
following 
categories of 
stakeholders: 

76 

 
Evaluation 
questions 

reasonably have 
been expected 
from Member 
States acting at 
national, regional 
and/or local 
levels or 
activities on 
international 
level (e.g. 
standardisation 
activities)? 

Success/judgment 
criteria 

  Achievement of 

objectives at a cost 
lower than what 
could be attained 
via national or sub-
national 
interventions. 

  Achievement of 
objectives that 
could not be 
otherwise attained 
with international 
activities. 

  Achievement of 

objectives at a cost 
lower than what 
could be attained 
with international 
activities. 

  Achievement in 
terms of cross-
border 
interoperability. 

  Contribution to the 
advancement of 
common EU 
policies. 

Indicators 

Data sources 

national/sub-national or 
international interventions. 

  Share of stakeholders’ providing 

positive feedback on achievements 
in terms of cross-border 
interoperability. 

  Qualitative assessment of the 

contribution to the advancement of 
common EU policies. 

  Qualitative assessment of the value 
generated by the EIF compared to 
international activities in the field. 

categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

 

 

ISA² action 
owners. 

ISA² solution 
users – European 
Commission. 

ISA² solution 
users – Member 
States. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Standardisation 
organisations. 

  Experts. 

Data collection / 
analysis methods 

  Governance of 
ISA² and the 
EIF. 

 

ISA² action 
owners. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Online 

surveys/written 
questionnaires 
targeted to the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

ISA² action 
owners. 

77 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

 

Indirect 
beneficiaries and 
wider public. 

  Secondary 

information from 
operational 
documents and 
other official 
documents, such as: 

  EIF 

Communication 
(COM/2017/134) 
and 
accompanying 
documents 
(Annexes to the 
Communication 
and Staff 
Working 
Documents). 

  Digital 

Government 
Factsheets. 

  Digital Economy 
and Society 
Index. 

 

 

ISA² solution 
users – European 
Commission. 

ISA² solution 
users – Member 
States. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Standardisation 
organisations. 

  Public consultation. 

  Case studies 
(success 
stories/lessons 
learnt). 

  Quantitative 

assessment of 
responses to 
interviews and 
surveys (Likert 
scale). 

78 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

  eGovernment 

  Qualitative 

assessment of 
responses to 
interviews and 
surveys and of data 
and information 
collected via desk 
research. 

  Expert assessment. 

Action Plan 2016 
– 2020. 

  eGovernment 
Benchmark 
Report. 

  Study on the role 
of eGovernment 
and 
interoperability 
in the European 
Semester. 

  The study ‘The 

nature of 
interoperability 
in digital public 
services. 
Deconstructing 
interoperability’. 

  Mid-term review 
of the DSM 
Strategy. 

  National 

Interoperability 
Frameworks. 

79 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

 

Interim 
Evaluation of 
ISA². 

  OECD Open 
Government 
Data 

  UN E-

Government 
Development 
Index (EGDI) 

 

IMD World 
Digital 
Competitiveness 
ranking 

Evaluation criterion #5: Relevance 

  To what extent 
do the initial 
objectives of the 
EIF still 
correspond to the 
current needs of 
the stakeholders 
at the EU, 
national or sub-
national level? 

How has the demand 
for cooperation, 

  Degree of alignment 

between 
stakeholders’ 
perception of needs 
and problems at 
national and sub-
national levels and 
the objectives of the 
EIF. 

  Degree of alignment 

between 
stakeholders’ 

  Share of stakeholders confirming 

the alignment between the 
objectives of the EIF and current 
needs and problems at national and 
sub-national levels. 

  Share of stakeholders confirming 

the alignment between the 
objectives of the EIF and current 
needs and problems at EU level. 

  Share of stakeholders confirming 
the alignment between the needs 

  Primary 

information on 
needs and problems 
from the following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

ISA² action 
owners. 

  Desk research. 

 

Interviews with the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

ISA² action 
owners. 

80 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

common rules and 
EU actions around 
interoperability 
changed? 

perception of needs 
and problems at EU 
level and the 
objectives of the 
EIF. 

  Degree of alignment 
between the needs 
and problems 
originally addressed 
by the EIF and 
stakeholders’ 
perception of current 
needs and problems. 

Demand for cooperation, 
common rules and EU 
actions in the field of 
interoperability. 

and problems addressed by the EIF 
and current needs and problems. 

  Share of stakeholders demanding 
more cooperation, common rules 
and EU actions in the field of 
interoperability. 

 

 

ISA² solution 
users – European 
Commission. 

ISA² solution 
users – Member 
States. 

  Qualitative assessment of the 

  Stakeholders 

alignment between the objectives 
of the EIF and current needs and 
problems. 

  Qualitative assessment of the 
alignment between needs and 
problems addressed by the EIF and 
current needs and problems. 

  Qualitative assessment of changes 

in demand for cooperation, 
common rules and EU actions in 
the field of interoperability. 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Standardisation 
organisations. 

  Experts.238 

 

Indirect 
beneficiaries and 
wider public.239 

  Secondary 

information on 
needs and problems 
from operational 

Data collection / 
analysis methods 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Online surveys/ 

written 
questionnaires 
targeted to the 
following 
categories of 
stakeholders: 

  Governance of 
ISA² and the 
EIF. 

 

 

ISA² action 
owners. 

ISA² solution 
users – European 
Commission. 

238  
239  

The ‘expert’ category includes, inter alia, academia and open-source communities. 
The category ‘wider public’ includes, among others, IT representatives and representatives of the GovTech sector. 

81 

 
 
                                                           
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

documents, other 
official documents, 
and relevant 
literature, such as: 

  EIF 

Communication 
(COM/2017/134) 
and 
accompanying 
documents 
(Annexes to the 
Communication 
and Staff 
Working 
Documents). 

  EIF Monitoring 
Mechanism 

 

ISA² solution 
users – Member 
States. 

  Stakeholders 

responsible for 
linked EU 
policies / 
initiatives. 

  Key national 

interoperability 
actors. 

  Standardisation 
organisations. 

 

Indirect 
beneficiaries and 
wider public 

  Digital 

  Public consultation. 

Government 
Factsheets. 

  Digital Economy 
and Society 
Index. 

  eGovernment 

Action Plan 2016 
– 2020. 

  Quantitative 

assessment of 
responses to 
interviews and 
surveys (Likert 
scale). 

  Qualitative 

assessment of 
responses to 
interviews and 

82 

 
Data collection / 
analysis methods 

surveys and data 
and information 
collected via desk 
research. 

Expert assessment. 

Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

  eGovernment 
Benchmark 
Report. 

  Study on the role 
of eGovernment 
and 
interoperability 
in the European 
Semester. 

  The study ‘The 

nature of 
interoperability 
in digital public 
services. 
Deconstructing 
interoperability’. 

  Data on the 

digital economy 
from Eurostat. 

  Mid-term review 
of the DSM 
Strategy. 

 

Interim 
Evaluation of 
ISA². 

83 

 
Evaluation 
questions 

Success/judgment 
criteria 

Indicators 

Data sources 

Data collection / 
analysis methods 

Source: Study supporting the evaluation of the implementation of the EIF 

  Mid-term 

Evaluation of 
CEF. 

Publications and 
surveys on demand for 
interoperability. 

84 

 
 
Annex IV. Overview of benefits and costs  

Table 8: Overview of costs and benefits identified in the evaluation 

Citizens 

Businesses 

Public administrations 

Quantitative 

Comment 

Quantitative 

Comment 

Quantitative 

Comment 

Costs 

No direct 
compliance costs 
applicable. 

No direct 
compliance costs 
applicable. 

No direct 
compliance costs 
applicable. 

No direct 
compliance costs 
applicable. 

Direct 
compliance 
costs: 
Implementation 
of the EIF (one-
off costs) 

EUR 24 000 – 
EUR 169 000, 
corresponding 
respectively to 
150 and 550 
person-days. 

The costs of 
implementation of 
the EIF vary 
depending on 
whether the Member 
State updates drafts 
of existing national 
documents and 
strategies based on 
the EIF or designs 
national documents 
and strategies based 
on the EIF without 
relying on any other 
prior such 
documents. For 
further details, see 
Section 3.1.2 on 
Efficiency. 

General comment: The costs of implementing the EIF are borne by public administrations only. 

Benefits 

85 

 
 
 
 
 
 
Citizens 

Businesses 

Public administrations 

Quantitative 

Comment 

Quantitative 

Comment 

Quantitative 

Comment 

Direct benefits: 
additional EU 
users of the 
Internet to 
interact with 
their public 
administrations 
(recurrent 
benefits, but 
expected to 
decrease over 
the time) 

A 10% 
improvement in 
the quality of 
online public 
services would 
encourage 
approximately 4.3 
million to 5 
million more EU 
citizens per year 

This result is part 
of an econometric 
analysis run on 21 
EU Member 
States, Iceland, 
Norway, 
Switzerland and 
the UK. See 
Section 3.1.1 on 
Effectiveness for 
further 
information. 

General comment: It is important to note that although no benefits are quantified for businesses and public administrations, 
the evaluation supports qualitatively that benefits exceed the costs for public administrations and bring substantial benefits 
to both businesses and citizens.  

Source: Study supporting the evaluation of the implementation of the EIF 

86 

 
 
 
 
 
 
 
 
Annex V. Stakeholder consultation 

Summary of the stakeholder strategy 

A  continuous  and  active  stakeholder  consultation  strategy  was  designed  and  followed  in 
preparation for the Impact Assessment on the EU public sector interoperability policy. It was 
complemented  by  active  communication  activities  on  our  dedicated  social  media  channels 
(Twitter, LinkedIn, YouTube). 

A broad range of activities were put in place to ensure that all interested parties and stakeholders 
would  have  the  opportunity  to  provide  feedback  on  the  various  policy  options  that  the 
Commission has identified with regard to its initiative, and their likely impacts, as well as on 
the relevance, effectiveness, efficiency and the added value of the initiative. In that context, the 
Commission reached out  to  a broad range of stakeholders, including Member State national 
authorities, non-governmental organisations, professional associations, business organisations 
and individual citizens. 

To  involve  a  broader  range  of  stakeholders  the  following  consultation  activities  were 
conducted: 

  Public consultation on the inception impact assessment (15/10/21): The key ideas for 
the  review  of  the  European  Interoperability  Framework  and  a  future  policy  were 
outlined in an Inception Impact Assessment (IIA). The published IIA informed citizens 
and  stakeholders  about  the  Commissions’  plans  in  order  to  allow  them  to  provide 
feedback on the intended initiative. This fed into the subsequent consultation activities 
that  ensured  an  inclusive  process  with  all  interested  parties  being  actively  invited  to 
contribute. 

  Online kick-off workshop ‘How interoperability can achieve seamless data flows and 
services for the EU’s public sector’, conducted on 3 December 2020. It aimed at raising 
awareness about the EIF evaluation and the impact assessment process and engaging 
stakeholders in the process. 

  Online survey (19 January – 7 March 2021), targeting specific stakeholder groups. 

  In-depth  interviews  (1  February  –  8  March  2021)  with  19  selected  stakeholders  to 
collect  detailed  data  and  information  contributing  to  the  EIF  evaluation  (12 
stakeholders) and the impact assessment (7 stakeholders), respectively. The interviews 
are complemented by expert assessments conducted by independent experts who were 
tasked with, inter alia, completing the questionnaire that served as the basis for the in-
depth interviews271. 

  A 12-week long, Internet-based public consultation (1 February – 26 April 2021), open 

to the wider public and available in English, German, and French. 

  Validation workshop in the digitalALL conference272 on 22 April 2021 presenting and 
validating the outcome of the consultation activities with 126 registered participants. 

  An  innovative  co-creation,  co-design  process  has  been  put  in  place  with  the 
interoperability  expert  group  with  Member  States.  Workshops  and  roundtables 
following design-thinking methodologies and participatory practices were led with the 
experts  to  discuss  and  develop  the  needs  for  a  future  interoperability  policy:  5 
brainstorming  sessions  with  more  than  200  representatives  from  Member  States,  5 

271  

272  

Five  independent  experts  provided  their  assessment  for  the  EIF  evaluation  and  four  for  the  impact 
assessment. 
Further 
public/planning/UGxhbm5pbmdfMzk2Mzcw 

https://app.swapcard.com/event/digitall-

information 

found 

can 

be 

at 

87 

 
                                                           
formal expert group meetings with also more than 200 participants overall, 18 position 
papers  issued  by  the  Member  States,  26  Member  States  bilaterals  throughout  the 
summer of 2021. 

  The topic was also presented and discussed in two meetings of the Chief Information 
Officers network organised by the Presidencies of the European Council (Portugal - 
May  2021,  Germany  -  December  2020)  and  presented  to  the  digital  attachés  in  the 
Telecom Working Party of the Council in April and November 2021. Member States 
have provided us  with  positive and constructive feedback which has been taken into 
account in the construction of the proposal. 

 Summary of the feedback on the Inception impact assessment 

In  total  there  were  13  feedback  replies  on  the  IIA  from  8  different  EU  countries  (3  from 
Germany, 2 from Spain and Belgium, 1 from each Sweden, Italy, Greece, France and Finland) 
and one non-EU country (Norway) and different stakeholder groups (see Figure 14) 

Figure 14: Feedback on the IIA by category of respondent 

Source: Have Your Say Portal 

The input was relevant for the formulation of problem and objectives as well as for the policy 
options and the impacts. 

On the problems stakeholders highlighted the effects of competing and non-open standards. 
One stakeholder suggested that the EIF evaluation should include an analysis on the reasons 
that led to some recommendations to be better implemented than others. This suggestion was 
taken up in the EIF evaluation. 

On  the  objectives  several  stakeholders  draw  links  to  public  values  beyond  digitalisation  as 
such. They suggested to highlight the value of interoperability for the democratisation of public 
knowledge, civil rights, digital involvement of all citizens and diversity. 

One stakeholder suggested to strengthen the ambition of EIF beyond guaranteeing the necessary 
state tasks, public services should support the optimal processing and easy usage of digital data. 
In general, most stakeholders advocate to strengthen the principle of openness for the reuse of 
technical solutions as a core principle. One suggests including codified design principles and 
to encourage an API centric approach. Another stakeholder suggests that the EIF should more 
explicitly be also addressed to regions and sectors. 

Eight of the 13 contributions mention explicitly the need for agreed and open standards. Two 
stakeholders request to include specific reference to interoperability between blockchains into 
the EIF with a clear definition and open standards. One stakeholder highlights that this should 

88 

 
 
include  standards  around  metadata,  another  the  importance  of  unique  and  trusted  digital 
identifiers. Two stakeholders ask for more and coordinated involvement of the public sector 
in standardisation organisations. 

Five  stakeholders  bring  forward  the  idea  of  fostering  and  EU-wide  sharing  and  reuse  of 
mature,  reusable  and open  source  interoperable  solutions  for  public  administrations.  One 
stakeholder asks to facilitate the multiple use of the same privacy by design infrastructure at no 
extra  cost.  The  future  policy  should  incentivise  the  sharing  of  design  costs  around  usable, 
trusted and secure solutions also between public and private stakeholders. 

Three stakeholders mention the need for more guidance on the use of open standards and open 
source,  including  clear  definitions.  One  stakeholder  asks  for  a  dedicated  task  force  for 
immediate,  middle  term  and  long-term  technical  assistance  at  all  levels  to  support  any 
interoperability action, including relevant training resources in the Interoperability Academy. 
One stakeholder asks for a certification programme for interoperability trainers. 

Three stakeholders highlight the importance of making the interoperability policy consistent 
with other EU policies like the EU AI strategy. Two stakeholders brought forward the idea of 
a  reference  implementation  on  the  city  level  published  under  a  free  software  license  which 
would  act  as  formal  specification  and  can  be  implemented  directly.  This  idea  is  mentioned 
linked to EU policies but also to EIF standards, that should come with at least one Open-Source 
implementation to confirm its implementability. 

For the policy instrument two stakeholders suggest a more binding instrument, one of them a 
consolidated  EU  Regulation  while  another  sees  the  risk  that  a  strict  legal  framework  might 
hinder interoperability – but harmonisation of organisational procedures could be beneficiary. 

One stakeholder highlights the importance for the different sectors on the example of author’s 
rights. It asks not only for effective cooperation mechanism but also for effective safeguards to 
ensure  their  respect.  Another  stakeholder  has  a  similar  idea  in  developing  a  process  for 
assessing compliance with the EIF. 

Stakeholders  highlighted  the  potential  impact  of  a  holistic  transformation  of  (digital)  public 
services on the private ICT sector with the potential to create an ecosystem of related apps and 
services. 

Consultation activities for the EIF evaluation and Impact assessment 

The consultation activities led by the contractor targeted several groups of stakeholders. The 
following grouping is used to analyse the feedback to the consultation activities: 

o  Civil society; 

o  Companies and business associations; 

o  EU and non-EU citizens; 

o  EU public authorities; 

o  Experts and academia (including the independent expert assessments); 

o  National and sub-national public authorities in the Member States. 

The  questionnaires  used  throughout  the  consultation  activities  mainly  used  Likert  scale 
responses, referring to a scale from (1) to (5), (-2) to (2) or (--) to (++), depending on the type 
of question273: 

273  

For each question, the respondent also had the possibility to select the answer ‘don’t know/no opinion’. 

89 

 
                                                           
3. 

4. 

5. 

(1) – not at all; (2) – to a limited extent; (3) – to some extent; (4) – to a great extent; or 
(5) – completely; 

(-2)  –  definitely  would  not;  (-1)  –  probably  would  not;  (1)  –  probably  would;  (2)  – 
definitely would; 

(--) – very negative; (-) – negative; (+) – positive; (++) – very positive. 

With a total of 112 respondents for the EIF Evaluation and 134 respondents for the Impact 
Assessment  for  a  Future  Interoperability  Strategy,  the  consultation  activities  reached  all 
types of stakeholders described in Table 9. In what follows, the feedback received during the 
consultation  activities  are  presented  according  to  the  type  of  consultation,  i.e.  public  versus 
targeted  consultation274.  Feedback  received  during  in-depth  interviews  are  grouped  together 
with those from the targeted online survey since both consultation activities were based on the 
same questionnaire; only more qualitative feedback was sought in the in-depth interviews. 

Table 9: Overview of responses to the EIF Evaluation (EIF) and the Impact Assessment 
(IA) by stakeholder group 

Consultation 

In-depth 
interview 

Online survey  Public 

TOTAL 

consultation 

Stakeholder 

EIF 

IA 

EIF 

IA 

EIF 

IA 

EIF 

7 

8 

9 

IA 

10 

Civil  society  (all 
other 
stakeholders) 

Companies  and 
business 
associations 

EU  and  non-EU 
citizens 

EU 
authorities 

public 

- 

- 

- 

5 

1 

- 

- 

3 

and 

6* 

6* 

2 

3 

2 

8 

4 

1 

5 

4 

10 

3 

14 

13 

17 

18 

18 

31 

20 

35 

- 

4 

- 

5 

13 

13 

14 

14 

and 

2 

- 

25** 

28** 

12 

16 

39 

44 

Experts 
academia 

National 
sub-national 
public 
authorities 

TOTAL 

13 

10 

44 

50 

55 

73 

112 

134 

*The six in-depth interviews include the five expert assessments. 
**One follow-up interview to the targeted online survey was conducted in the scope of the EIF Evaluation and 
one was conducted in the scope of the Impact Assessment. 
Source: Study supporting the impact assessment for a future interoperability strategy 

274  

Averages do not account for respondents answering ‘don’t know/no opinion’. 

90 

 
 
                                                           
In the public consultations, feedback was received from 13 Member States, with the highest 
number of received answers from Spain (see Figure 15). In addition, six respondents are non-
EU citizens. For the work with the Expert Group all 27 Member States provided input. 

Figure 15: Geographical distribution per stakeholders’ category 

Source: Study supporting the impact assessment for a future interoperability strategy 

The group of respondents to the public consultations from public authorities includes a mix of 
administrations  exercising  at  the  international  (3  respondents),  national  (4  respondents), 
regional (6 respondents) and local (3 respondents) levels (see Figure 16). 

Figure 16: Scope of public authorities (share of respondents and number of respondents 
in brackets) 

Source: Study supporting the impact assessment for a future interoperability strategy 

On average, respondents have a good level of knowledge in the field, noting they are familiar 
to a great extent with digital public services and interoperability (see Figure 17). When it comes 
to specific knowledge about the EIF, the difference between the respondents to the targeted 
consultation  and  those  contributing  to  the  public  consultation  is  more  pronounced,  with 
respondents  to  the  public  consultation  reflecting  a  relatively  lower  level  of  knowledge  (see 
Figure 18). Nevertheless, the consulted stakeholders are on average familiar at least to some 
extent with the EIF. 

91 

 
 
 
Figure 17: Knowledge of digital public services and interoperability (breakdown by type 
of consultation; average score and number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Source: Study supporting the impact assessment for a future interoperability strategy 

Figure 18: Knowledge of the EIF (breakdown by type of consultation; average score and 
number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Source: Study supporting the impact assessment for a future interoperability strategy 

Results: EIF Evaluation 

The framework is deemed to be relevant to some extent. The needs and problems originally 
identified  by  the  EIF  continue  to  be  experienced  across  the  relevant  stakeholders  and  the 
framework  has  only  partially  addressed  them  (see  Figure  19).  Respondents  to  the  targeted 
consultations call for more guidance for public administrations to improve the governance of 
their  interoperability  activities  and  address  the  problems  linked  to  the  fragmentation  in  the 
delivery  of  digital  public  services  and  the  organisation  and  format  of  public  data.  Public 
administrations  particularly  point  to  the  need  for  more  cooperation,  especially  in  specific 
sectors  (e.g.,  health).  Several  other  needs  were  highlighted,  such  as  the  need  to  increase 
awareness, for staff with IT skills in public administrations and to build investment capacity to 
keep pace with rapid technological change and bridge technological barriers. 

Figure 19: Extent to which the following needs and problems are currently experienced 
by  European  public  administrations,  businesses  and/or  citizens  (breakdown by  type  of 
consultation; average score and number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Source: Study supporting the evaluation of the implementation of the EIF 

92 

 
 
 
 
Stakeholders participating in both consultations consider that the recommendations of the EIF 
have been effective to some extent in achieving the objectives (see Figure 20). However, the 
EIF  has  been  seeking  to  achieve  broad  objectives  and  the  recommendations  require  more 
granularity. The targeted consultation generally emphasises the contribution of the framework 
to raising awareness of the importance of cross-border interoperability, although more needs to 
be done at the sub-national level. The principles set out by the EIF have enhanced interoperable 
digital  public  services  but  are  difficult  to  assess  due  to  their  abstract  nature  or  still  limited 
achievements.  Stakeholders  call  for  more  clarification  when  it  comes  to  transparency, 
technological  neutrality  and  user-centricity.  The  layered  interoperability  and  the  conceptual 
model could be improved with more practical guidance for implementing the models. Overall, 
positive  impacts  have  been  experienced  across  stakeholders,  in  particular  in  enhancing  the 
quality of services provided by public administrations and in fostering the free movement of 
goods, services, capital and workers across the EU. Nevertheless, more could be achieved to 
increase the direct and indirect benefits gained by stakeholders. 

Figure 20: Extent to which the recommendations listed in the EIF have contributed so far 
to  the  achievement  of  the  following  objectives  (breakdown  by  type  of  consultation; 
average score and number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Source: Study supporting the evaluation of the implementation of the EIF 

Although  costs  and  benefits  are  difficult  to  assess  due  to  a  plethora  of  factors,  stakeholders 
participating  in  the  targeted  consultation  acknowledge  that,  in  the  long  run,  the  EIF  brings 
benefits  that  exceed  the  cost  of  implementation.  The  EIF  results  in  efficiency  gains  by 
supporting  data  re-use,  enabling  synergies  in  implementing  and  designing  new  services  and 
streamlining administrative procedures. A representative from a national public administration 

93 

 
 
noted that the costs of several ICT projects have been eliminated due to timely considerations 
of interoperability. The costs of implementing the framework are relatively moderate, both at 
the national and EU levels; initial investments can be challenging, but over time the benefits 
outweigh the costs and expand beyond a single administration. 

In terms of internal coherence, the respondents consider that the components of the EIF are 
generally synergetic (see Figure 21); the framework could be improved by better linking the 
conceptual  model  with  the  principles  and  the  layered  interoperability  model.  In  addition, 
clarification  of  some  thematically  overlapping  sets  of  recommendations  would  make  the 
framework more actionable. At the level of external coherence, synergies are identified with 
the Single Digital Gateway, the Digital Single Market Strategy and the Data Strategy; overlaps 
and inconsistencies remain limited, but challenges may arise within own frameworks that are 
developed as part of sectoral initiatives (see Figure 21). 

Figure 21: Extent to which there are synergies, overlaps between the EIF and other EU 
initiatives with similar objective (breakdown by type of consultation; average score and 
number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Source: Study supporting the evaluation of the implementation of the EIF 

Finally, respondents to both types of consultations generally agree that national or sub-national 
initiatives  would  bring  only  limited  contributions  to  the  objectives  pursued  by  the  EIF, 
confirming  its  EU  added  value  (see  Figure  22).  In  particular,  the  cross-border  dimension 
cannot  be  achieved  solely  by  national  or  sub-national  administrations.  The  EIF  is  not 
sufficiently  used  across  areas  and  by  Commission  services.  Furthermore,  cross-border 
interoperability remains limited and is driven by specific sectoral needs. 

94 

 
 
Figure 22: Extent to which national or sub-national interventions (in the absence of the 
EIF) would be able to achieve the following objectives (breakdown by type of consultation; 
average score and number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a great extent; or (5) completely. 
Source: Study supporting the evaluation of the implementation of the EIF 

Input of the Expert Group co-design process 

We  have  put  in  place  an  innovative  co-creation,  co-design  process  with  the  interoperability 
expert group with Member States that encourages open cooperation and transparency. This has 
led to a constant increase in the engagement and interaction in meetings. Moving all discussions 
online since more than 18 months now has also led to an increase in the number of participants. 
In physical meetings we would have had 1 or maximum 2 representatives per Member State for 
a maximum of 35 approximately, whereas online we regularly have more than 54 participants. 
Workshops and roundtables follow design-thinking methodologies and participatory practices. 
We organised since mid-2020:  5 brainstorming sessions with  more than 200 representatives 
from  Member  States,  5  formal  expert  group  meetings  with  also  more  than  200  participants 
overall. Member States issued 18 position papers and we met them throughout the summer of 
2021 in 26 bilaterals to discuss around the recommendations on the future policy. 

The  main  result  of  this  process  with  the  expert  group  are  the  recommendations  on  a  future 
interoperability policy. Other intermediate results can be interesting to understand the view of 
the stakeholders that are likely to be the most affected by the policy initiative. 

95 

 
 
Figure 23: Co-design process with Expert Group 

The co-design process with the expert group was structured around five work phases: 

  Position  papers  from  18  Member  States  –  CIOs  office  –  In  December  2020  the 
Commission kindly asked the members of the expert group to react on a short policy paper 
with some initial ideas for the next interoperability policy (vision, general approach, policy 
instruments, etc.). This was only on a voluntary basis, however two-third of the members 
replied positively and shared position papers explaining their views on the upcoming policy 
and the proposed instruments; 

  Co-creation workshops – five informal discussion sessions took place between January 
and December 2021 (see the list below), involving around 40-45 members of EU national 
administrations  each  time.  The  working  methodology  was  based  on  design-thinking 
principles,  including  ‘diverge’  and  ‘converge’  working  phases,  and  participants  were 
sometimes asked to fill polls (e.g. see below a poll from the workshop on the 30th of April). 
These meetings were moderated using Chatham House rules (no recording, participants do 
not  formally  engage  their  country),  in  order  to  foster  interactions  between  people  and 
ideation. Several working papers were circulated to stabilize ideas along the process and 
the  subjects  of  these  workshops  helped  build  the  structure  of  the  draft  policy 
recommendations of the group (see point d.). All papers were available to all the participants 
and we encouraged transparent feedbacks directly on a Teams space. 

o  04/02/2021 – General debate on the policy approach for the next interoperability 

policy following the position papers (e.g. vision, priorities, etc.) 

o  14/04/2021  –  Communities  of  Practitioners  (e.g.  topics,  organisation  and 

composition, etc.)  

o  30/04/2021  –  Synergies  and  Funding  (e.g.  links  with  other  EU  digital  policies, 

financial support to Member States for interoperability implementation) 

o  20/05/2021 – Digital Checks and digital-ready policy (e.g. experience from Member 

States, replicability at the EU institutional level)  

o  10/06/2021  –  Governance  (e.g.  composition,  structure  and  mandate  of  a 

strengthened EU interoperability governance) 

96 

 
 
Figure 24: Example of poll during the co-creation workshop on the 30th of April 2021 

  Bilaterals – the Commission organised 26 bilateral calls with 26 Member States between 
July  and  September  2021  (only  one  country  did  not  reply  because  of  a  lack  of  internal 
resources  at  that  time).  Each  of  these  informal  discussions  lasted  between  45  and  120 
minutes. They were very rich in terms of content and feedback received, while helping the 
Commission  to  identify  points  of  tension,  test  ideas  with  the  Member  States  and  build 
mutual trust; 

  Policy recommendations – this is the main outcome of the expert group since its creation 
in February 2020. They were formally endorsed by consensus of the whole group on the 5th 
of  October  2021.  The  document  includes  27  recommendations  structured  around  three 
chapters, each of them reflecting the common ideas expressed by the Member States in the 
position papers, the co-creation workshops and the bilaterals: 

o  Chapter 1 – A strengthened governance 

o  Chapter 2 – Interoperability for better EU policies 

o  Chapter 3 – Upgrade EU interoperable solutions 

97 

 
 
 
Those recommendations served as a basis to build the draft legal text and the two 
dedicated workshops; 

  Two workshops on the legal draft – They took place in October and November 2021. The 
first one gathered around 40 external participants who expressed their views on the legal 
concepts introduced by the Commission and discussed the composition and mandate of the 
future  strengthened  EU  interoperability  governance.  Following  the  first  workshop,  the 
Commission circulated a first draft of the legal text with the members of the expert group 
and organised a workshop to collect preliminary ideas and suggestions to be integrated in 
the impact assessment and the upcoming legal proposal. Member States provided written 
feedback by the 3rd of December 2021. 

98 

 
Annex VI. Supporting evidence from desk research 

This annex presents a series of sub-annexes containing supporting evidence for the assessment of the EIF. 

Annex VI.1. Scoreboards monitoring the implementation of the EIF 

Table 10: Scoreboard 1: Principles of interoperability 

Country 

Principl
e 01 

Principl
e 02 

Principl
e 03 

Principl
e 04 

Principl
e 05 

Principl
e 06 

Principl
e 07 

Principl
e 08 

Principl
e 09 

Principl
e 10 

Principl
e 11 

Principl
e 12 

Austria 

3 

Belgium  4 

Bulgaria  3 

Croatia 

Cyprus 

Czech 
Rep. 

Denmar
k 

Estonia 

Finland 

France 

German
y 

3 

4 

4 

4 

3 

3 

3 

4 

4 

4 

4 

4 

4 

4 

4 

2 

3 

3 

4 

4 

3 

3 

4 

4 

4 

No data 

#REF! 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

2 

4 

4 

2 

4 

4 

4 

4 

4 

3 

4 

3 

3 

4 

4 

4 

4 

4 

No data 

3 

3 

3 

3 

3 

3 

3 

2 

4 

4 

4 

4 

4 

4 

4 

4 

4 

2 

2 

2 

3 

3 

3 

3 

3 

4 

No data  No data  No data  No data   3 

4 

2 

3 

4 

3 

99 

4 

4 

3 

3 

4 

4 

4 

4 

4 

4 

4 

4 

No data 

4 

4 

4 

4 

4 

4 

4 

4 

3 

3 

3 

3 

3 

4 

2 

3 

 No data   No data 

4 

3 

 
  
Country 

Principl
e 01 

Principl
e 02 

Principl
e 03 

Principl
e 04 

Principl
e 05 

Principl
e 06 

Principl
e 07 

Principl
e 08 

Principl
e 09 

Principl
e 10 

Principl
e 11 

Principl
e 12 

Greece 

3 

Hungary  3 

Ireland 

Italy 

Latvia 

Lithuani
a 

Luxemb
ourg 

Malta 

Netherla
nds 

Poland 

4 

3 

4 

3 

4 

3 

4 

3 

Portugal  3 

Romani
a 

3 

Slovenia  3 

Slovakia  3 

4 

4 

4 

4 

4 

2 

4 

4 

4 

4 

4 

4 

4 

4 

3 

4 

4 

2 

3 

4 

4 

4 

3 

3 

3 

2 

3 

4 

4 

4 

3 

4 

4 

3 

4 

2 

4 

3 

2 

2 

4 

4 

4 

4 

3 

4 

4 

4 

4 

4 

4 

4 

3 

3 

4 

4 

1 

3 

3 

No data 

3 

3 

3 

3 

3 

3 

3 

3 

3 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

3 

3 

4 

2 

4 

4 

4 

4 

4 

3 

3 

2 

4 

3 

100 

2 

2 

3 

3 

4 

3 

3 

4 

3 

2 

3 

2 

3 

2 

3 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

2 

4 

4 

4 

4 

4 

4 

4 

4 

4 

2 

3 

1 

2 

4 

3 

3 

No data 

3 

4 

4 

No data 

1 

4 

4 

4 

3 

2 

2 

4 

3 

 
  
  
Country 

Principl
e 01 

Principl
e 02 

Principl
e 03 

Principl
e 04 

Principl
e 05 

Principl
e 06 

Principl
e 07 

Principl
e 08 

Principl
e 09 

Principl
e 10 

Principl
e 11 

Principl
e 12 

Spain 

Sweden 

4 

4 

4 

4 

4 

4 

4 

4 

4 

3 

4 

4 

3 

No data 

4 

4 

3 

3 

4 

4 

4 

4 

2 

1 

Note: The full description of the principles can be consulted in Annex 2 of the 2017 EIF Communication. 
Source: Authors’ elaboration based on the data available on Joinup concerning the EIF Monitoring Mechanism and the underlying database provided by DIGIT.D2. 

Table 11: Scoreboard 2: Interoperability layers 

Country 

Interoperability 
governance 

Integrated 
service governance 

public 

Legal 
interoperabili
ty 

Organisational 
interoperability 

Semantic 
interoperability 

Technical 
interoperability 

Austria 

Belgium 

Bulgaria 

Croatia 

Cyprus 

Czech 
Republic 

3 

3 

4 

3 

3 

4 

Denmark  4 

Estonia 

Finland 

2 

3 

4 

3 

4 

4 

4 

4 

4 

3 

3 

4 

2 

4 

4 

2 

4 

4 

4 

3 

4 

4 

4 

4 

4 

4 

4 

4 

4 

101 

3 

4 

4 

4 

3 

3 

4 

4 

4 

  4 

  4 

  4 

  4 

  4 

4 

  4 

  4 

  1 

 
 
 
 
Country 

Interoperability 
governance 

Integrated 
service governance 

public 

Legal 
interoperabili
ty 

Organisational 
interoperability 

Semantic 
interoperability 

Technical 
interoperability 

France 

No data 

No data 

No data 

No data 

No data 

  No data  

Germany  4 

Greece 

Hungary 

Ireland 

Italy 

Latvia 

3 

4 

3 

3 

3 

Lithuania  1 

Luxembo
urg 

Malta 

Netherlan
ds 

Poland 

Portugal 

Romania 

4 

4 

4 

3 

3 

2 

4 

4 

4 

4 

4 

4 

3 

4 

3 

4 

3 

3 

2 

4 

4 

4 

1 

2 

4 

4 

4 

3 

4 

3 

4 

3 

2 

2 

3 

2 

4 

4 

1 

4 

4 

4 

2 

2 

4 

102 

2 

2 

4 

4 

4 

3 

2 

4 

3 

4 

4 

2 

2 

  4 

  4 

  4 

  4 

  4 

  4 

  4 

4 

  4 

4 

  4 

  4 

  4 

 
 
 
 
Country 

Interoperability 
governance 

Integrated 
service governance 

public 

Legal 
interoperabili
ty 

Organisational 
interoperability 

Semantic 
interoperability 

Technical 
interoperability 

Slovenia 

Slovakia 

Spain 

Sweden 

3 

4 

4 

1 

4 

4 

4 

4 

4 

4 

4 

2 

4 

4 

4 

4 

4 

4 

4 

4 

  4 

  4 

  4 

  4 

Source: Authors’ elaboration based on the data available on Joinup concerning the EIF Monitoring Mechanism and the underlying database provided by DIGIT.D2. 

Table 12: Scoreboard 3: Conceptual model for integrated public services provision 

Country 

Austria 

Belgium 

Bulgaria 

Croatia 

Cyprus 

Czech 
Republic 

Denmark 

Estonia 

Conceptual 
model 

Internal information sources 
and services 

Base 
registries 

Open 
data 

Catalo
gues 

External 
sources and services  

information 

Security  and 
Privacy 

4 

4 

4 

3 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

3 

4 

4 

4 

4 

2 

3 

3 

2 

3 

3 

3 

2 

4 

2 

2 

3 

4 

3 

4 

4 

4 

4 

3 

4 

4 

4 

4 

4 

4 

4 

103 

3 

4 

3 

2 

3 

4 

3 

2 

 
 
 
Country 

Conceptual 
model 

Internal information sources 
and services 

Base 
registries 

Open 
data 

Catalo
gues 

External 
sources and services  

information 

Security  and 
Privacy 

Finland 

3 

4 

France 

No data 

No data  

4 

No data 

Germany 

Greece 

Hungary 

Ireland 

Italy 

Latvia 

Lithuania 

Luxembo
urg 

Malta 

Netherlan
ds 

Poland 

Portugal 

Romania 

3 

4 

4 

1 

4 

3 

4 

3 

3 

3 

4 

4 

2 

1 

4 

4 

1 

4 

4 

4 

4 

4 

4 

4 

4 

1 

3 

3 

3 

4 

4 

4 

3 

4 

4 

4 

3 

3 

3 

104 

4 

4 

3 

3 

4 

4 

4 

3 

3 

3 

3 

4 

4 

3 

3 

4 

3 

No data  No data 

3 

No data 

2 

3 

3 

2 

4 

3 

2 

4 

4 

4 

1 

3 

2 

1 

3 

3 

4 

3 

3 

3 

4 

2 

4 

3 

3 

3 

4 

3 

3 

3 

4 

2 

3 

3 

3 

4 

3 

4 

2 

 
Country 

Slovenia 

Slovakia 

Spain 

Sweden 

Conceptual 
model 

Internal information sources 
and services 

Base 
registries 

Open 
data 

Catalo
gues 

External 
sources and services  

information 

Security  and 
Privacy 

4 

4 

4 

4 

4 

4 

4 

4 

4 

3 

4 

3 

4 

3 

4 

4 

3 

4 

2 

1 

4 

3 

4 

4 

4 

4 

4 

2 

Source: Authors’ elaboration based on the data available on Joinup concerning the EIF Monitoring Mechanism and the underlying database provided by DIGIT.D2. 

105 

 
Annex VI.2. Overview of the adoption of NIFs 

Country  NIF  in  place 
or  EIF-based 
strategy 

Year 
of 
adoption142 

Remarks 

AT 

BE 

BG 

2015 

Austrian 
interoperability 
Framework 
(AIFv1.0) 

2005 

Belgian 
Interoperability 
Framework 
(BelgIF) 

2006 

Bulgarian 
National 
Interoperability 
Framework 
(BNIF) – Draft 

HR 

Croatian 
Interoperability 
Framework 

2010 

The  Austrian 
Interoperability  Framework 
(AIFv1.0)  was  found  to  be  almost  perfectly 
aligned with the EIF (with an alignment score of 
95%), improving the alignment compared to the 
the  national 
previous 
(75% 
interoperability 
alignment).143 

strategy  governing 
efforts 

in  Austria 

Interoperability  in  Belgian  public  services  was 
initially addressed in 2005 when the first NIF was 
adopted. In 2017, BelgIF was adapted to the new 
EIF.  The  current  Belgian 
Interoperability 
Framework relies on the 12 principles of the EIF 
as  the  basis  on  which  the  federal  and  regional 
public  authorities  define  interoperability.  The 
BelgIF  endorses  the  47  recommendations  put 
forward in the 2017 version of the EIF. 

The first version of the Bulgarian Interoperability 
Framework  was  adopted  in  2006.  The  current 
Bulgarian  National  Interoperability  Framework 
(BNIF,  v1.2)  is  available  in  draft  form  and 
provides  an  update  to  the  Bulgarian  National 
Interoperability  Framework  for  Governmental 
Information Systems adopted in 2006. The draft 
BNIF  builds  on  the  three  main  elements  of  the 
EIF 
levels,  a 
conceptual model for public services), providing 
recommendations 
the  development  of 
Bulgarian interoperable public services.  

interoperability 

(principles, 

for 

based 

The  Croatian  Interoperability  Framework  was 
developed 
European 
Interoperability  Framework  1.0  (published  in 
2004). A new interoperability framework has not 
been issued, but the subsequent revisions of the 

the 

on 

142  

143  

This column refers to the first adoption of the National Interoperability Frameworks in the Member State. 
Years of adoption show the date when interoperability of public services was first taken into account by 
adopting the National Interoperability Frameworks. It should be noted that the data accuracy depends on 
the data availability online. 
Further  details  on  the  alignment  score  is  available  here:  https://joinup.ec.europa.eu/collection/nifo-
national-interoperability-framework-observatory/news/austrias-renewed-interoperab 

106 

 
                                                           
Country  NIF  in  place 
or  EIF-based 
strategy 

Year 
of 
adoption142 

Remarks 

EIF  are  taken  into  account  in  the  actions 
undertaken in the field of interoperability.144 

CY 

2016 

Cyprus 
National 
Interoperability 
Framework 
(eGIF) 

CZ 

- 

- 

DK 

EE 

2017 

The  Common 
framework  for 
Public  Sector 
Digital 
architecture 

2011 

Estonian 
Interoperability 
Framework 
(EIF) 

FI 

- 

- 

The Cyprus National Interoperability Framework 
(eGIF)  v1.0  was  adopted  in  2016,  and  was 
updated by the Cyprus National Interoperability 
Framework (eGIF) v2.0. eGIF v2.0 was prepared 
in  order  to  align  the  already  existing  national 
framework with the new EIF published in 2017. 
The 
the  principles, 
interoperability  layers  and  conceptual  model  of 
public services presented in the EIF. 

eGIF  v2.0 

reflects 

national 

interoperability 

interoperability 

The 
framework 
currently  supports  legislative,  organisational, 
in 
technical  and  semantic 
compliance  with  the  European  Interoperability 
Framework.  It  takes  into  account  principles  of 
eGovernment  in  the description of its  ‘National 
Public 
Architecture 
eGovernment 
Administration’, which  was implemented based 
on  the  Information  Concept  Plan  (adopted  in 
2018).  

of 

The  Common  framework  for  Public  Sector 
Digital  architecture  (2017)  takes  into  account 
several 
layers 
interoperability 
interoperability principles of EIF. Also, it is built 
partly  on 
Interoperability 
Reference Architecture (EIRA).  

the  European 

and 

Estonia  has  addressed  the  interoperability  of 
public services for a long time. The third version 
of the Estonian Interoperability Framework was 
published  in  2011.  The  current  version  of  the 
Estonian  Interoperability  Framework  is  aligned 
with 
Interoperability 
Framework  on  the  basis  of  terminology  and 
general  principles.  It  includes  three  layers  of 
interoperability; 
interoperability, 
technical 
organisational and semantic interoperability.  

the  new  European 

Finland  does  not  have  a  NIF.  The  Information 
in  Public  Administration  Act 
Management 

144  

As  noted  in  the  following  strategic  document:  Ministry  of  Public  Administration  (2017),  e-Croatia 
Strategy 
http://digarhiv.gov.hr/arhiva/1231/168420/uprava.gov.hr/UserDocsImages/e-
Hrvatska/Strategija_e-Hrvatska_2020.pdf; https://uprava.gov.hr/UserDocsImages//Istaknute%20teme/e-
Hrvatska//e-Croatia%202020%20Strategy%20(20.01.2016.).pdf 

2020, 

107 

 
                                                           
Country  NIF  in  place 
or  EIF-based 
strategy 

Year 
of 
adoption142 

Remarks 

a 

includes 

requirement 

(906/2019) 
for 
government agencies to  utilise datasets of other 
government agencies whenever possible, if they 
by  law  have  access  to  such  data  via  electronic 
interfaces.  

FR 

- 

- 

DE 

- 

- 

EL 

2006 

Greek 
eGovernment 
Interoperability 
Framework 
(Greek eGIF) 

HU 

- 

IS 

- 

- 

- 

France  adopted  the  General  Interoperability 
Repository  (2016)  that  refers  to  technical  and 
semantic interoperability. The repository is a set 
of  recommendations  referencing  norms  and 
standards that promote interoperability within the 
public administration's information systems. The 
referencing  of  norms  and  standards  is  based  on 
criteria developed by the European Commission 
(the Common Assessment Method for Standards 
and Specifications). 

dedicated 

The Federal Republic of Germany does not have 
a 
Interoperability 
National 
Framework.  However,  it  does  take  EIF  into 
the  Architectural  Guideline  for 
account 
Federal 
to  help  public 
administrations  develop  new  public  services  by 
providing technical and semantic guidelines. The 
binding  Architecture  Guidelines  were  first 
adopted in 2017.  

in 
IT,  which  aims 

eGovernment 

Interoperability 
The  Greek 
Framework  has  been  developed 
in  2006, 
providing  standards  and  specifications  for  the 
and 
development  of  web-based 
guidelines 
for  public  administrations.  The 
framework  was  officially  adopted  through  state 
law in 2010.145  

services 

Hungary  does  not  have  a  dedicated  National 
Interoperability  Framework.  However  in  the  
‘National  Info  Communication  Strategy  2014  – 
2020’  several  principles  of  EIF  are  addressed 
such  as  openness,  technological  neutrality  and 
security. 

Iceland  does  not  have  a  dedicated  National 
Interoperability  Framework.  However,  National 
bodies  are  involved  in  the  development  of  the 

145  

Digital 

See: 
https://joinup.ec.europa.eu/sites/default/files/inline-
files/Digital_Public_Administration_Factsheets_Greece_vFINAL.pdf 

Administration 

Public 

Factsheet 

2020: 

Greece, 

108 

 
                                                           
Country  NIF  in  place 
or  EIF-based 
strategy 

Year 
of 
adoption142 

Remarks 

national  interoperability  framework  that  will 
public 
include 
administrations and private entities.  

government 

levels, 

all 

IE 

IT 

- 

- 

Ireland  does  not  have  a  dedicated  National 
Interoperability Framework.  

The 
New 
Interoperability 
Model  

2020 

In  Italy,  a  New  Interoperability  Model  is  being 
developed  in  Italy,  as  a  cornerstone  for  the  IT 
plan  for  public  administrations  between  2020  – 
2022.  It  provides  guidelines  and  technological 
specifications  and  standards  for  Italian  Public 
the  new 
Administrations.  The  design 
interoperability model is based on the principles 
of the European Interoperability Framework (the 
2010 version).  

for 

LV 

- 

- 

of 

of 

public 

Architecture 

a  dedicated  National 
Latvia  does  not 
the 
Interoperability  Framework.  However, 
Public 
Conceptual 
Administration  Information  Systems  (2015) 
provides  an  architectural  reference  covering  all 
aspects 
systems, 
organisation,  data,  systems  and  technology.  It 
provides  40  recommendations  on  the  long-term 
vision  for  development  of  public  services. 
Furthermore,  the  Latvian  Cabinet  of  Ministers 
suggested that the Conceptual Framework needs 
to  be  more  aligned  with  the  current  European 
Interoperability Framework.   

information 

LI 

LT 

LU 

MT 

- 

- 

2019 

2013 

- 

- 

The 
Luxembourg 
National 
Interoperability 
Framework 
(NIF) 

National 
ICT 
Interoperability 
Framework 
(NIF) 

Liechtenstein  has  not  adopted  a  National 
Interoperability Framework.  

The  Republic  of  Lithuania  does  not  have  a 
dedicated National Interoperability Framework.  

The  Luxembourg  National 
Interoperability 
framework  is  developed  on  the  basis  of  the 
European Interoperability Framework. It consists 
of  11  principles  and  48  recommendations, 
for  development  of 
providing  guidelines 
interoperable public services in Luxembourg.  

The  National  Interoperability  Framework  of 
Malta,  developed  by  the  Malta  Information 
Technology Agency (MITA), was first published 

109 

 
Country  NIF  in  place 
or  EIF-based 
strategy 

Year 
of 
adoption142 

Remarks 

administrations 

in  2013.146  It  provides  guiding  principles  for 
full 
public 
interoperability.  The  NIF  is  being  revised  to 
ensure 
the  European 
Interoperability Framework.  

full  alignment  with 

achieve 

to 

ME 

National 
Interoperability 
Framework  

2011 

NL 

2006 

Dutch 
Government 
Reference 
Architecture 
(NORA) 

NO 

Norwegian 
Interoperability 
Framework 

2018 

Montenegro  had  developed  the  first  version  of 
the  Montenegrin  Interoperability  Framework  in 
2011,  and  a  second  version  followed  in  2013, 
defining  rules  and  methods  for  enhancing  the 
interoperability  of  public  administrations.  The 
current  version  of  the  Montenegrin  National 
Interoperability  Framework  was  developed  in 
2019  and  is  built  on  the  three  main  elements 
(principles, 
and 
interoperability 
conceptual  model  for  public  services)  of  the 
current European Interoperability Framework.147  

layers 

The  Dutch  Government  Reference  Architecture 
the  role  of  a  National 
(NORA),  playing 
Interoperability Framework, was first adopted in 
2006.  It  provides  in  10  basic  principles  and  38 
derived principles for successful development of 
interoperable  public  services.  In  2016  NORA 
was  almost  perfectly  aligned  with  the  previous 
version  of 
Interoperability 
Framework.148  

the  European 

The  Norwegian  Interoperability  Framework  is 
considered  as  the  national  transposition  of  the 
European  Interoperability  Framework.  The  NIF 
is mandatory only for the national levels of public 
administrations, while for the local and regional 
levels it is strongly recommended. It is built on 
(principles, 
the 
interoperability layers and conceptual model for 
public services) of the European Interoperability 
Framework. Furthermore, it provides guidelines 
and  recommendations  on  how 
to  develop 
interoperable public service in Norway.  

three  main 

elements 

146  

147  

148  

Digital 

See: NIFO Factsheet 2015 – Malta, https://joinup.ec.europa.eu/sites/default/files/inline-files/NIFO%20-
%20Factsheet%20Malta_12_2015.pdf 
See: 
https://joinup.ec.europa.eu/sites/default/files/inline-
files/Digital_Public_Administration_Factsheets_Montenegro_vFINAL.pdf  
Further 
the 
https://joinup.ec.europa.eu/sites/default/files/custom-page/attachment/2017-
10/NIFO_Updated_Analytical%20Model_NETHERLANDS_2016_published.pdf  

Administration 

alignment 

available 

factsheet 

Montenegro, 

details 

Public 

2020: 

score 

here: 

are 

on 

110 

 
                                                           
Country  NIF  in  place 
or  EIF-based 
strategy 

Year 
of 
adoption142 

Remarks 

PL 

- 

- 

PT 

- 

- 

MK 

Macedonian 
Framework  for 
Interoperability 

2016 

RO 

2017 

Romanian 
Interoperability 
Framework 
(RIF) 

SK 

- 

- 

Poland  does  not  have  a  dedicated  National 
Interoperability  Framework.  However,  Poland 
set up the State Information Architecture (2018) 
which  aims  to  ensure  that  IT  activities  and 
processes taking place in public administrations 
are  consistent.  The  work  undertaken  in  Poland, 
starting from the State Information Architecture, 
is  carried  out  in  accordance  with  the  European 
Interoperability  Reference  Architecture  and  the 
European Interoperability Framework.149  

the 

Framework.  However, 

Portugal  does  not  have  a  dedicated  National 
Interoperability 
a 
platform  for 
interoperability  of  public 
administrations  is  available  online  (created  in 
2015).  The  platform  provides  tools  enabling 
public  administrations  to  provide  interoperable 
electronic services. 

for 
The  North  Macedonian  Framework 
Interoperability is developed on the basis of the 
previous  European  Interoperability  Framework 
it  consists  of  11  principles  and  4 
and 
interoperability 
semantic, 
organisational and technical). 

(legal, 

layers 

Interoperability  Framework, 
The  Romanian 
adopted  in  2017,  consists  of  the  three  main 
elements  (principles,  interoperability  layers  and 
conceptual  model  for  public  services)  of  the 
previous  European  Interoperability  Framework, 
providing  recommendations  and  guidelines  for 
public  administrations  to  develop  interoperable 
public services.  

Slovakia  does  not  have  a  dedicated  National 
Interoperability  Framework.  However,  the  New 
National  eGovernment  Concept,  approved  in 
December  2021  and  the  updated  one  build  on 
principles from the EIF. 

SL 

149  

The  Slovenian 
National 
Interoperability 

2012 

While  a  Slovenian  National  Interoperability 
Framework exists, this has been established as a 
of 
portal 

development 

coordinated 

for 

Digital 

See: 
https://joinup.ec.europa.eu/sites/default/files/inline-
files/Digital_Public_Administration_Factsheets_Poland_vFINAL.pdf  

Administration 

Public 

factsheet 

2020: 

Poland, 

111 

 
                                                           
Country  NIF  in  place 
or  EIF-based 
strategy 

Year 
of 
adoption142 

Remarks 

SP 

SE 

CH 

TR 

UK 

Framework 
(NIO)  

interoperable public services in Slovenia, rather 
than  as  a  strategic  document.  The  NIO  portal 
recommendations  and 
provides  guidelines, 
sharing  of  best  practices 
relevant 
for 
stakeholders.  Furthermore,  it  enables  exchange 
of 
the 
relevant  stakeholders  through  the  connection 
with  the  JoinUp  platform.150  Interoperability  is 
considered  in  strategic  documents  such  as  the 
‘Public  Administration  Development  Strategy 
2015–2020’. 

interoperability  with 

information 

in 

2010 

Spanish 
National 
Interoperability 
Framework 
(EIN)  

Spanish 

Interoperability 
National 
The 
Framework (EIN) was first published in 2010 and 
was  aligned  in  2018  with  the  new  European 
Interoperability Framework.  

- 

- 

- 

- 

- 

- 

- 

- 

The  National 
Strategy 
(Regeringens  strategi  för  standardisering)  was 
adopted in July 2018, being aligned to the EIF 

Standardisation 

Switzerland does not have a dedicated National 
Interoperability Framework. 

Turkey  does  not  have  a  dedicated  National 
Interoperability Framework 

the  current  European 

The United Kingdom does not have a dedicated 
National  Interoperability  Framework.  However, 
the United Kingdom has adopted the ‘The Digital 
Service Standard’ and the ‘Technology Code of 
Practice’.  Both  of  these  documents  are  aligned 
with 
Interoperability 
Framework.  The  UK  has  had  a  series  of  ICT 
strategies  to  improve  the  delivery  of  public 
services,  including  a  strategy  adopted  in  2011 
2012 
(Government 
Government  Digital  Strategy,  and  the  2017 
Government Transformation Strategy. 

Strategy), 

ICT 

the 

UA 

- 

- 

Ukraine  does  not  have  a  dedicated  National 
Interoperability Framework.  

Source:  Study  supporting  the  evaluation  of  the  implementation  of  the  EIF  based  on  the  Digital  Public 
Administration Factsheets and official national documents and portals. 

150  

Further 
https://nio.gov.si/nio/cms/page/purpose?lang=en  

details 

about 

the 

NIO 

platform 

are 

available 

here: 

112 

 
                                                           
Annex VI.3. Overview of the mentions of EIF in a sample of EU pieces of legislation and 
policy documents 

Does 
it 
mention 
the 
EIF? 

Yes.  

Yes.  

Initiative  /  Piece 
of legislation 

counter 

A 
Terrorism 
Agenda  for  the 
-  Proposal 
EU 
(Communication) 

Berlin 
on 
Declaration 
Digital  Society 
and  Value-based 
Digital 
Government 

Yes. 

Commission 
Implementing 
Regulation  (EU) 
No 821/2014151 

Digital  Services 
Act - Proposal  

No.  

eIDAS 
Regulation  

No.  

it  mention 
the 

Does 
any 
interoperability 
layers? 

of 

reference 
technical  and 

Indirect 
to 
semantic 
interoperability  (p. 
13 and p 17) 

reference 
technical  and 

Indirect 
to 
semantic 
interoperability  (p. 
12) 

it 

Does 
mention 
the 
EIF 
conceptual 
model? 

it  mention  the 
for 
public 

Does 
principles 
interoperable 
services? 

No.  

No. 

No.  

  Principle 

Transparency  

  Principle 

Reusability 

3) 

4) 

  Principle  6)  User 

centricity 

  Principle 7) Inclusion 
and accessibility 
  Principle  8)  Security 

and privacy 

No. 

No. 

reference 
technical  and 

Indirect 
to 
semantic 
interoperability. (p. 
8)  

No. 

reference 
Indirect 
to 
technical 
interoperability  (p. 
52) 

No.  

  Direct reference 
to 
technical 
Interoperability 

  Indirect 

to 

reference 
semantic 
interoperability 

  Principle 

Subsidiarity 
proportionality 

  Principle 

Transparency  

1) 
and 

3) 

  Principle  8)  Security 

and privacy 

  Principle 

subsidiarity 
proportionality 

1) 
and 

  Principle  8)  security 

and privacy 

  Principle 

preservation 
information 

11) 
of 

151  

Commission  Implementing  Regulation  (EU)  No  821/2014  of  28  July  2014  laying  down  rules  for  the 
application of Regulation (EU) No 1303/2013 of the European Parliament and of the Council as regards 
detailed  arrangements  for  the  transfer  and  management  of  programme  contributions,  the  reporting  on 
financial  instruments,  technical  characteristics  of  information  and  communication  measures  for 
operations and the system to record and store data 

113 

 
                                                           
Initiative  /  Piece 
of legislation 

Does 
it 
mention 
the 
EIF? 

it  mention 
the 

Does 
any 
interoperability 
layers? 

of 

it 

Does 
mention 
the 
EIF 
conceptual 
model? 

it  mention  the 
for 
public 

Does 
principles 
interoperable 
services? 

(in 
case 
Attributes) 

of 

Yes. 

No.  

No.  

  Principle 2) openness 
  Principle 
3) 

transparency 

  Principle  8)  security 

and privacy 

Yes. 

Direct  reference  to 
technical 
interoperability  (p. 
19) 

No.  

No. 

No. 

No.  

No.  

EU  Government 
plan 
Action 
(Communication) 

Single 

EU 
Window 
Environment  for 
– 
Customs 
Proposal 

European  Cloud 
Initiative 
(Communication) 

European  Data 
Governance 
– 
Proposal 

Yes. 

A 
European 
strategy for data  

Yes.  

(Communication) 

No. 

No. 

reference 
semantic 

Indirect 
to 
interoperability 
(p.31) 

Indirect:  Semantic 
interoperability and 
technical 
interoperability 
(p.12 and p.33) 

  Principle 

5) 

data 

portability  

  Generally  referenced 
to EIF principles  

  Principle 

proportionality  

  Principle 

transparency 

  Principle 

reusability 

1) 

3) 

4) 

No. 

Yes. 

INSPIRE 
Directive  

Source 

Open 
Software 
Strategy  2020  - 
2023 
(Communication 
to 
the 
Commission) 

Indirect:  Technical 
interoperability  (p. 
3) 

No. 

No. 

No. 

reference 
technical 

Indirect 
to 
interoperability 
(p.10).  

  Principle 2) Openness 
  Principle 
4) 

Reusability 

  Principle  8)  Security 

and privacy 

114 

 
 
 
 
 
 
Initiative  /  Piece 
of legislation 

Does 
it 
mention 
the 
EIF? 

it  mention 
the 

Does 
any 
interoperability 
layers? 

of 

it 

Does 
mention 
the 
EIF 
conceptual 
model? 

it  mention  the 
for 
public 

Does 
principles 
interoperable 
services? 

Yes. 

No. 

No. 

Regulation  (EU) 
2019/1149 
a 
establishing 
European  Labour 
Authority 

Digital 

Single 
Gateway 
Regulation 

No. 

No.  

reference 
Indirect 
to  semantic  and 
technical  layers  of 
interoperability  (p. 
21).  

  General  reference  to 
the  EIF  principles 
(p.25)  

  Principle 

Subsidiarity  

1) 

  Principle 2) Openness 
  Principle 
3) 

Transparency 

  Principle  6)  User 

centricity  

Source: Study supporting the evaluation of the implementation of the EIF 

Annex  VI.4.  Evolution  of  the  quality  of  digital  public  services  and  citizens’  use  of  the 
Internet to interact with public administrations between 2012 and 2019 

This annex presents the evolution of both the quality of digital public services, captured by the 
e-Government  index,  and  the  share  of  citizens  using  the  Internet  to  interact  with  public 
administrations between 2012 and 2019 depending on whether countries adopted the NIF. 

Figure 25: Evolution of the e-Government index and the use of the Internet to interact 
with public administration among citizens between 2012 and 2019 for Member States that 
have adopted the NIF before 2012 

Source: Study supporting the evaluation of the implementation of the EIF based on data on the maturity level of 
online  public  services  (e-Government  benchmark)  and  the  share  of  citizens  using  the  Internet  to  interact  with 
public administrations (Eurostat). 

115 

 
 
Figure 26: Evolution of the e-Government index and the use of the Internet to interact 
with public administration among citizens between 2012 and 2019 for Member States that 
have adopted the NIF after 2012, but before 2019 

Source: Study supporting the evaluation of the implementation of the EIF based on data on the maturity level of 
online  public  services  (e-Government  benchmark)  and  the  share  of  citizens  using  the  Internet  to  interact  with 
public administrations (Eurostat). 

Figure 27: Evolution of the e-Government index and the use of the Internet to interact 
with public administration among citizens between 2012 and 2019 for Member States that 
did not adopt the NIF over the studied period 

Source: Study supporting the evaluation of the implementation of the EIF based on data on the maturity level of 
online  public  services  (e-Government  benchmark)  and  the  share  of  citizens  using  the  Internet  to  interact  with 
public administrations (Eurostat). 

Annex VI.5. Academic and grey literature outlining the needs and problems in the field 
of interoperability 

This annex presents a synthetic overview of the academic and grey literature supporting the 
assessment of the needs and problems addressed by the EIF and experienced more generally in 

116 

 
 
 
the field of interoperability in the EU’s public sector, thus contributing to the evaluation of the 
relevance criterion. 

Table  13:  Literature  review:  needs  and  problems  related  to  the  EIF  in  the  field  of 
interoperability 

EIF identified needs and problems  Sources 

Public administrations  need  more 
specific  guidance  on  how 
to 
improve  the  governance  of  their 
interoperability activities 

  Halmos  (2018),  Cross-border  digital  public 
services,  Cross  Border  Review  2018  Central 
European Service for Cross-Border Initiatives; 
  Krimmer et. al. (2018), Contributing to a Digital 
Single Market for Europe: Barriers and Drivers 
of an EU-wide Once-Only Principle. Proceedings 
of  the  19th  Annual  International  Conference  on 
Digital Government Research; 

  Masciotta 

  Andrews  et.  al.  (2016),  Making  a  success  of 
digital government. Institute for Government UK  
the 
interoperability  issue  within  the  P.A.  from  the 
Italian  constitutional  perspective.  ITALIAN  J. 
PUB. L. 689 (2019); 

(2019),  A 

strategy 

on 

interoperability 

  Kourabali  and  Katehakis  (2019),  The  new 
European 
framework  as  a 
facilitator  of  digital  transformation  for  citizen 
empowerment. Journal of Biomedical Informatics 
94 

  Wimmer 

(2018), 

et.  al. 

Interoperability 
Governance: A Definition and Insights from Case 
Studies  in  Europe.  Proceedings  of  the  19th 
Annual  International  Conference  on  Digital 
Government Research; 

Fragmentation in the organisation 
and  format  of  public  data  in  the 
EU 

  Kalvet et. al (2018), Cross-border e-Government 
Services  in  Europe:  Expected  Benefits,  Barriers 
and  Drivers  of  the  Once-Only  Principle;  In 
Proceedings of the 11th International Conference 
on Theory and Practice of Electronic Governance 
(ICEGOV 18) 

  Capgemini et. al. (2013) Study on Analysis of the 
Needs for Cross-Border Services and Assessment 
of  the  Organisational,  Legal,  Technical  and 
Semantic  Barriers.  Publications  Office  of  the 
European Union; 

  Masciotta 

(2019),  A 

the 
interoperability  issue  within  the  P.A.  from  the 
Italian  constitutional  perspective.  ITALIAN  J. 
PUB. L. 689 (2019); 

strategy 

on 

  Sallamo  et.  al.  (2020),  Recommendations  for 
organising  and  governing 
integrated  public 
services.  Publications  Office  of  the  European 
Union; 

117 

 
EIF identified needs and problems  Sources 

  JRC  (2020),  Assessing  the  impacts  of  digital 
government 
the  EU. 
transformation 
Publications Office of the European Union; 

in 

  Kalogirou  and  Charalabidis 

(2019),  The 
European  Union  Landscape  on  Interoperability 
Standardisation:  Status  of  European  and 
National  Interoperability  Frameworks.  In  K. 
Popplewell 
Enterprise 
Interoperability  VIII,  Proceedings  of  the  I-ESA 
Conferences 9; 

(eds.), 

al. 

et 

The fragmented delivery of digital 
public services in the EU 

  Kalvet et. al (2018), Cross-border e-Government 
Services  in  Europe:  Expected  Benefits,  Barriers 
and  Drivers  of  the  Once-Only  Principle.  In 
Proceedings of the 11th International Conference 
on Theory and Practice of Electronic Governance 
(ICEGOV 18) 

  Sallamo  et.  al.  (2020),  Recommendations  for 
organising  and  governing 
integrated  public 
services.  Publications  Office  of  the  European 
Union; 

  Kalogirou  and  Charalabidis 

(2019),  The 
European  Union  Landscape  on  Interoperability 
Standardisation:  Status  of  European  and 
National  Interoperability  Frameworks.  In  K. 
Popplewell 
Enterprise 
Interoperability  VIII,  Proceedings  of  the  I-ESA 
Conferences 9;  

(eds.), 

al. 

et 

The 
need 
interoperability’  and 
awareness 

for 

‘cultural 
increased 

  Halmos  (2018),  Cross-border  digital  public 
services,  Cross  Border  Review  2018  Central 
European Service for Cross-Border Initiatives; 
  Capgemini et. al. (2013) Study on Analysis of the 
Needs for Cross-Border Services and Assessment 
of  the  Organisational,  Legal,  Technical  and 
Semantic  Barriers.  Publications  Office  of  the 
European Union; 

  Krimmer et. al. (2018), Contributing to a Digital 
Single Market for Europe: Barriers and Drivers 
of an EU-wide Once-Only Principle. Proceedings 
of  the  19th  Annual  International  Conference  on 
Digital Government Research; 

  Cave et. al.  (2017), EU-wide digital Once-Only 
Principle  for  citizens  and  businesses:  Policy 
options and their impacts. Publications Office of 
the European Union; 

  Nauta (2019), The influence of  national  culture 
interoperability  of  cross-border  IT 

the 

on 
systems. IC Institute 

118 

 
EIF identified needs and problems  Sources 

The  need  to  address  the  shortage 
in  public 
of  human  resources 
sector IT departments and to build 
investment  capacity  to  keep  pace 
with  rapid  technological  change 
and bridge technological barriers 

  Sallamo  et.  al.  (2020),  Recommendations  for 
organising  and  governing 
integrated  public 
services.  Publications  Office  of  the  European 
Union; 

  Misuraca  et.  al.  (2020),  Exploring  Digital 
Government Transformation. Publications Office 
of the European Union;  

  Krimmer et. al. (2018), Contributing to a Digital 
Single Market for Europe: Barriers and Drivers 
of an EU-wide Once-Only Principle; Proceedings 
of  the  19th  Annual  International  Conference  on 
Digital Government Research; 

  Cave et. al.  (2017), EU-wide digital Once-Only 
Principle  for  citizens  and  businesses:  Policy 
options and their impacts. Publications Office of 
the European Union; 

  Andrews  et.  al.  (2016),  Making  a  success  of 
digital government. Institute for Government UK 
  Chinn et. al. (2020) The future is now: Closing 
in  Europe’s  public  sector. 

the  skills  gap 
McKinsey & Company 

Source: Study supporting the evaluation of the implementation of the EIF 

119 

 
 
Annex VII. Solutions contributing to the EIF implementation 

Package 

Action name 

Contribution to the EIF and the IAP 

and 

Key 

1. 
generic 
interoperability 
enablers 

Trusted 
Platform (e-TrustEx) 

Exchange 

e-TrustEx  is  a  platform  offered  to  public  administrations  at  European,  national  and  regional 
levels to undertake secure exchange of natively digital documents or scanned documents from 
system to system via standardised interfaces.  

Contribution to the EIF: The action thus contributes particularly to Recommendation 15 of the 
revised EIF, through enabling the secure exchange of documents. 

Catalogue of Services 

and 

Key 

1. 
generic 
interoperability 
enablers 

to 

the 

conceptual  model 

The Catalogue of Services is one of the interoperability enablers for integrated public services 
EIF. 
according 
To that end, the action has defined a technical specification, the data model, CPSV AP 152adopted 
by several MS to catalogue their data and linked them up at cross-border level for example to 
support the implementation  of the Single Digital  Gateway Regulation. Also a set  of proof of 
concept  of  some  tools  to  facilitate  the  creation  of  catalogue  of  public  services  have  been 
produced. 

defined 

revised 

the 

by 

2. 
interoperability 

Semantic 

Promoting 

SEMIC: 
Semantic 
Interoperability Amongst 
the  European  Union  EU 
countries 

Contribution to the EIF: The action addresses Recommendation 44 of the revised version of the 
EIF on the catalogue of public services. 

The  Action  supports  the  implementation  of  the  EIF  by  promoting  semantic  interoperability, 
through  the  definition  and  use  of  common  specifications  such  as  the  Core  Vocabularies,  or 
DCAT AP used by the European Open Data Portal and other MS portals to catalogue their data 
and linked them up across borders.  

Contribution to the EIF: The action contributes primarily to Recommendation 16 of the revised 
EIF. In addition, the action covers the following underlying principles of the EIF: Reusability, 
Multilingualism,  Openness,  Semantic 
interoperability  and 
Standardisation. 

interoperability,  Technical 

152  

https://joinup.ec.europa.eu/collection/catalogue-services/open-specifications-cpsv-ap-and-sdg-services-model-describing-procedures 

120 

 
                                                           
Package 

Action name 

Contribution to the EIF and the IAP 

2. 
interoperability 

Semantic 

Public 
Multilingual 
Knowledge Management 
Infrastructure 
the 
Digital Single Market 

for 

The  creation  of  a  Public  Multilingual  Knowledge  Infrastructure  aims  to  support  EU  public 
administrations in creating services that can be accessible and shareable independently from the 
language actually used, as well as allowing SMEs to sell goods and service cross-border in a 
DSM. 

to 

3.  Access 
data/data 
sharing/open data 

Big  Data 
Administrations 

for  Public 

to 

3.  Access 
data/data 
sharing/open data 

Statistical 
Sharing 
and 
Production 
Dissemination  Services 
and  Solutions 
the 
European 
Statistical 
System 

in 

Contribution  to  the  EIF:  The  action  is  based  on  several  recommendations  (primarily 
Recommendation 16 on taking into account multilingualism in the setting up of European public 
services)  and  principles  of  the  new  EIF,  in  particular  those  concerning  multilingualism, 
accessibility, administrative simplification, transparency, and reusability of the solutions. 

This action will facilitate the sharing of open data between public administrations through the 
support to the execution of analytics projects on Big Data; increase the transparency of decision-
making in public administrations by supporting knowledge sharing on evidence-based policy-
making  practices;  support  the  re-use  of  open  source  data  analytics  tools  developed  by  EU 
countries  of  EU  Institutions;  and  provide  public  administrations  with  the  opportunity  to  test 
(open  source)  technologies  in  this  domain  before  making  a  decision  on  the  technical  way 
forward. 

Contribution to the EIF: This action contributes to several EIF principles including ‘openness’, 
‘transparency’, ‘reusability’ and ‘technological neutrality’. 

This action contributes to several areas: developing, maintaining and promoting interoperable 
solutions  for  the  production  and  dissemination  of  statistics  by  EU  public  administrations 
(including the EC) and 2) developing, maintaining and promoting a) a specification of the EIRA 
to  support  better  interoperability  and  cooperation  for  the  production  and  dissemination  of 
Official  Statistics  in  the  European  Statistical  System;  b)  a  common  infrastructure  for  the 
services. 
of 
exposure 
In  addition,  the  proposal  contributes  significantly  to  the  realisation  of  the  ESS  Vision  2020 
objectives in the domain of sharing tools and improving statistical dissemination. 

consumption 

statistical 

shared 

and 

Contribution  to  the  EIF:  Through  its  focus  on  aligning  infrastructures  for  shared  statistical 
services,  the  action  builds  on  the  principles  and  recommendations  of  the  EIF,  in  particular 
Recommendation 36. 

121 

 
Package 

Action name 

Contribution to the EIF and the IAP 

to 

3.  Access 
data/data 
sharing/open data 

Development of an Open 
Data  Service,  Support 
and  Training  Package  in 
the Area of Linked Open 
Data,  Data  Visualisation 
and 
Persistent 
Identification 

The action supports open data initiatives by facilitating data re-use and sharing and offering tools 
to visualise data effectively. 

Contribution  to  the  EIF:  The  action  contributes  to  the  new  EIF,  namely  the  interoperability 
principles:  openness, 
reusability,  user–centricity  and  multilingualism, 
accessibility. 

transparency, 

to 

3.  Access 
data/data 
sharing/open data 

Geospatial 

4. 
solutions 

of 

Provision 
services 
data.europa.eu, 
official 
European data.  

portal 

these 
via 
the 
for 

The action supports open data initiatives by facilitating data re-use and sharing and offering tools 
to visualise data effectively. 

Contribution  to  the  EIF:  The  action  contributes  to  the  new  EIF,  namely  the  interoperability 
principles:  openness, 
reusability,  user–centricity  and  multilingualism, 
accessibility. 

transparency, 

Location 

European 
Interoperability 
Solutions 
Government (ELISE) 

for 

ELISE has aimed to deepen the understanding of location interoperability enablers and barriers 
related to the transition towards digital government. 

e-

Contribution  to  the  EIF:  ELISE  builds  on  several  areas  of  the  EIF  including  openness, 
reusability,  technological  neutrality,  user-centricity,  multilingualism,  and  administrative 
simplification. 

5.  eProcurement/ 
eInvoicing 
- 
Supporting 
instruments 

European 
Procurement 
Interoperability Initiative 

Public 

This  action  supports  several  activities  designed  to  simplify  procurement  and  facilitate  the 
participation in online procurement as well the re-use of data in the field. 

Contribution to the EIF: The action builds in particular on Recommendations 28 and 30 of the 
EIF, as well as facilitating the implementation of the once-only principle in the area of public 
procurement. 

6. 
making 
legislation 

Decision 
and 
- 

Legal 
interoperability 
(former ICT Implications 
of EU Legislation) 

The ‘Legal Interoperability’ action supports policymaking across policy areas, bringing to the 
forefront the importance of considering potential digital impacts and the role of interoperability 
when developing new legislation. 

122 

 
 
Package 

Action name 

Contribution to the EIF and the IAP 

Supporting 
instruments 

Decision 
and 
- 

6. 
making 
legislation 
Supporting 
instruments 

Supporting 
for 

8. 
instruments 
public 
administrations 

Supporting 
for 

8. 
instruments 
public 
administrations 

Contribution to the EIF: The action implements Recommendation 27 on legal interoperability of 
the new EIF. 

Inter-Institutional 
Register  of  Delegated 
Acts (RegDel)  

This  action  focused  on  developing  an  IT  tool  setting  up  the  Inter-Institutional  Register  of 
Delegated Acts, increasing transparency around delegated acts and thus responding to the 2016 
commitment of the Commission in this sense. 

Contribution to the EIF: This action contributes primarily to the transparency principle of the 
EIF and to Recommendation 5 of the new EIF by providing a transparent overview of delegated 
acts. 

– 

European 
Joinup 
Collaborative  Platform 
and Catalogue  

The action facilitates the sharing and re-use of solutions for public administrations and provides 
the stakeholders with the means to collaborate via a collaborative platform. 

Contribution  to  the  EIF:  ‘Joinup’  builds  especially  on  the  reusability  principle  of  the  EIF, 
facilitating access and supporting the re-use of available interoperable solutions. 

National  Interoperability 
Framework Observatory 

The NIFO action has monitored interoperability initiatives in the Members and developed the 
Monitoring Mechanism to keep track of the implementation of the EIF Recommendations by 
EU  countries.  NIFO  has  also  developed  an  EIF  Toolbox  153to  support  EU  countries  in  the 
implementation of the EIF with concrete solutions addressing specific aspects of the EIF, and 
guidance on relevant best practice examples, pieces of legislation and open specifications. 

Contribution  to  the  EIF:  NIFO  provides  an  overarching  contribution  to  the  EIF,  by  helping 
monitor the alignment of national initiatives with the EIF and the implementation of the EIF in 
the EU countries. The action responds to the commitment from the 2017 EIF Communication 
that called for the development of a framework for monitoring the implementation of the EIF. 
This was achieved with the development of the EIF Monitoring Mechanism as part of the NIFO 
action. 

153  

https://joinup.ec.europa.eu/collection/national-interoperability-framework-observatory/eif-toolbox  

123 

 
                                                           
Package 

Action name 

Contribution to the EIF and the IAP 

Supporting 
for 

8. 
instruments 
public 
administrations 

European 
Interoperability 
Architecture (EIA) 

This action helps define the needs and shortcomings with relation to a common interoperability 
architecture for European public services and contribute to defining such an architecture as well 
as map reusable solutions and guidelines services as interoperability building blocks. 

Contribution to the EIF: The ‘EIA’ actions contribute in particular to Recommendation 23 of the 
EIF and to overall interoperability governance. 

EUSurvey 

Supporting 
for 

8. 
instruments 
public 
administrations 

Supporting 
for 

8. 
instruments 
public 
administrations 

IMAPS  

EUSurvey contributes primarily to the multilingualism principle of the EIF.  

This  action  supports  tools  for  the  assessment  of  the  interoperability  maturity  level  of  digital 
public services, helping to identify improvement priorities. Since 2019154, the focus of the work 
of  the  IMAPS  action  evolves  around  raising  awareness  on  IMAPS  and  assisting  public 
administrations in  EU Member States  in  the uptake of  IMAPS by their organisation, through 
capability-building and sharing of relevant knowledge, insights and good practices. 

Contribution to the EIF: This action contributes to the principles of reusability and user-centricity 
(in particular, Recommendation 12 of the EIF) by creating a mechanism for analysis, design, 
assessment and further development of the European Public Services. 

Supporting 
for 

8. 
instruments 
public 
administrations 

Standard-Based Archival 
Management, 
Data 
Exchange 
and 
Publication 

The  action  contributes  to  supporting  data  standards  in  the  field  of  archival  information 
management, studying among others how Open Data formats can be used in this area. 

Contribution to the EIF: The action builds on several principles of the EIF including openness, 
transparency, reusability, technological neutrality, preservation of information, user-centricity. 

8. 
instruments 

Supporting 
for 

Interoperability 
Academy 

This action was established in order to help increase awareness of interoperability, the EIF and 
the solutions developed under ISA2. The action facilitates access to information and learning 
material in this sense. 

154 https://joinup.ec.europa.eu/collection/imaps-interoperability-maturity-assessment-public-service/about  

124 

 
                                                           
Package 

Action name 

Contribution to the EIF and the IAP 

public 
administrations 

9.  Accompanying 
measures 

Raising  Interoperability 
Awareness 
– 
Communication 
Activities 

Contribution to the EIF: This action promotes the principle of reusability. In addition, it provides 
an overall contribution to the EIF, by promoting the principle, models, and recommendations of 
the Framework and facilitating access to information about implementing the EIF. 

Contribution to the EIF: This action contributes to the principles of inclusion, accessibility and 
transparency by disseminating information about interoperability and the work of ISA2. 

Annex VIII. Mapping of legal initiatives to the EIF 

Name of the legal 
initiative (with 
hyperlink) 

Short title: Open Data 
Directive 

Title: 
Directive 
2019/1024  on  open 
data  and  the  re-use  of 

Short description 

EIF Core 
Topics 

EIF pillar 

Open data (Art.1) Specific high-value datasets and arrangements for publication and 
re-use, formats of data and metadata, investments made by the Member States in 
open data approaches (Art.14) 

EIF 
Conceptual 
model 

Recommendation 
41 

Recommendation 
42 

Recommendation 
43 

125 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

public 
information (Recast)  

sector 

Short description 

Art. 1 - Open data (‘In order to promote the use of open data and stimulate innovation 
in products and services, this Directive establishes a set of minimum rules governing 
the re-use and the practical arrangements for facilitating the re-use of: (…)’) 

Art.  5:  ‘public sector bodies and public undertakings shall make their documents 
available  in  any  pre-existing  format  or  language  and,  where  possible  and 
appropriate,  by  electronic  means,  in  formats  that  are  open,  machine-readable, 
accessible, findable and re-usable, together with their metadata. Both the format and 
the metadata shall, where possible, comply with formal open standards.’ 

EIF Core 
Topics 

EIF pillar 

EIF Principles  Recommendation 
2 

Recommendation 
4 

Art. 7: Transparency 

Art. 7(3): ‘Public sector bodies shall ensure that applicants for re-use of documents 
are informed of available means of redress relating to decisions or practices affecting 
them.’ 

Art. 1: reuse of documents 

Art.  3:  ‘Subject  to  paragraph  2  of  this  Article,  Member  States  shall  ensure  that 
documents to which this Directive applies in accordance with Article 1 shall be re-
usable for commercial or non-commercial purposes in accordance with Chapters III 
and IV.’ 

EIF Principles  Recommendation 
5 

EIF Principles  Recommendation 
7 

Art. 9: ‘Member States shall, in cooperation with the Commission, continue efforts 
to simplify access to datasets, in particular by providing a single point of access’ 

EIF Principles  Recommendation 

11 

Art 9: ‘Where possible, Member States shall facilitate the cross-linguistic search for 
documents, in particular by enabling metadata aggregation at Union level.’ 

EIF Principles  Recommendation 

16 

126 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

Art. 9: ‘Member States shall also encourage public sector bodies to make practical 
arrangements facilitating the preservation of documents available for re-use.’ 

EIF Principles  Recommendation 

18 

Art.  5:  ‘public sector bodies and public undertakings shall make their documents 
available  in  any  pre-existing  format  or  language  and,  where  possible  and 
appropriate,  by  electronic  means  (…)  The  high-value  datasets,  as  listed  in 
accordance  with  Article  14(1)  shall  be  made  available  for  re-use  in  a  machine 
readable format, via suitable APIs and, where relevant, as a bulk download.’ 

Chapter V - High-value datasets 

Interoperability 
Layers 

Recommendation 
30 

Art. 5: ‘Both the format and the metadata shall, where possible, comply with formal 
open standards.’ 

Interoperability 
Layers 

Recommendation 
33 

Art. 14: ‘The arrangements may include terms applicable to re-use, formats of data 
and metadata and technical arrangements for dissemination.’ 

Art. 9: ‘Member States shall make practical arrangements facilitating the search for 
documents available for re-use, such as asset lists of main documents with relevant 
metadata, accessible where possible and appropriate online and in machine-readable 
format, and portal sites that are linked to the asset lists. [...] Member States shall, in 
cooperation with the Commission, continue efforts to simplify access to datasets, in 
particular by providing a single point of access.’ 

Interoperability 
Layers 

Recommendation 
36 

Short  title:  INSPIRE 
DIRECTIVE 

Art.  7(3)  -  Member  States  shall  ensure  that  all  newly  collected  and  extensively 
restructured spatial data sets and the corresponding spatial data services are available 

EIF Principles  Recommendation 
2 

Art. 14 - Data made available 

127 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Directive 
Title: 
on 
2007/2/EC 
an 
establishing 
Infrastructure 
for 
Spatial  Information  in 
European 
the 
Community   

Short description 

EIF Core 
Topics 

EIF pillar 

Art  17(5)  -  The  arrangements  for  the  sharing  of  spatial  data  sets  and  services 
provided for in paragraphs 1, 2 and 3 shall be open, on a reciprocal and equivalent 
basis 

Chapter IV, Article 11- Network services: ‘discovery services making it possible to 
search for spatial data sets and services’, ‘view services’, ‘download services’ 

EIF Principles  Recommendation 
5 

Sharing of data (Chapter V Data sharing, art. 17) 

Article  20  -  ‘The  implementing  rules  referred  to  in  this  Directive  shall  take  due 
account of standards adopted by European standardisation bodies’ 

EIF Principles  Recommendation 
7 

Article 5: These rules shall take account of relevant, existing international standards 
and user requirements, in particular with relation to validation metadata. 

Article  7:  users,  (...)  shall  be  given  the  opportunity  to  participate  in  preparatory 
discussions on the content of the implementing rules 

Article  11:  Those  services  shall  take  into  account  relevant  user  requirements  and 
shall be easy to use, available to the public and accessible via the Internet or any 
other appropriate means of telecommunication. 

Article 18 - identification of user needs 

EIF Principles  Recommendation 

10 
Recommendation 
12 

Article  11:  Those  services  shall  take  into  account  relevant  user  requirements  and 
shall be easy to use, available to the public and accessible via the Internet or any 
other appropriate means of telecommunication. 

EIF Principles  Recommendation 

14 

Article 20 - shall take due account of standards adopted by European standardisation 
bodies (…), as well as international standards. 

Interoperability 
Layers 

Recommendation 
20 

Chapter III - INTEROPERABILITY OF SPATIAL DATA SETS AND SERVICES, 
Article 7 

Recommendation 
21 

128 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

Chapter VI COORDINATION AND COMPLEMENTARY MEASURES ‘Member 
States shall ensure that  appropriate structures and mechanisms  are designated for 
coordinating, across the different levels of government, the contributions of all those 
with an interest in their infrastructures for spatial information.’ 

Chapter IV - Network services 

Chapter VI – Coordination 

Article 19 – Contact point: ‘Each Member State shall designate a contact point, 
usually a public authority, to be responsible for contacts with the Commission in 
relation to this Directive.’ 

EIF Core 
Topics 

EIF pillar 

Recommendation 
22 

Interoperability 
Layers 

Recommendation 
25 

Interoperability 
Layers 

Recommendation 
28 

Article 20 – shall take due account of standards adopted by European standardisation 
bodies (…), as well as international standards. 

Interoperability 
Layers 

Recommendation 
27 

Chapter II – Metadata 

Art.  1  –  for  the  purposes  of  Community  environmental  policies  and  policies  or 
activities which may have an impact on the environment. 

Art. 4 – evolving needs for spatial data in support of Community policies that affect 
the environment. 

Interoperability 
Layers 

Recommendation 
31 

Recommendation 
32  

Art. 7 – laying down technical arrangements for the interoperability 

Art. 10 - including data, codes and technical classifications 

Interoperability 
Layers 

Recommendation 
33 

Article  15  –  2.  Member  States  shall  provide  access  to  the  services  referred  to  in 
Article 11(1) through the Inspire geo-portal  

Conceptual 
model 

Recommendation 
36 

129 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short 
Data 
Regulation 

title:  General 
Protection 
(GDPR)  

of 

Title: Regulation (EU) 
2016/679 
the 
European  Parliament 
and  of  the  Council  of 
27  April  2016  on  the 
protection  of  natural 
persons with regard to 
the 
of 
processing 
personal  data  and  on 
the  free  movement  of 
such data.  

Short description 

EIF Core 
Topics 

EIF pillar 

Creation  of  a  common  model  for  the  unique  identification  of  spatial  objects, 
(Chapter Interoperability of Spatial Data Sets and Services, art. 8, 11) 

Conceptual 
model 

Recommendation 
35 

Metadata (Chapter II Metadata Art. 5, 1 and 3) 

Member State law which shall be proportionate to the aim pursued (Art. 9, g) 

Proportionality of the processing operations (Art. 35, b) 

Conceptual 
model 

Recommendation 
42 

EIF Principles  Recommendation 
1 

Processed lawfully, fairly and in a transparent manner in relation to the data subject 
(Art. 5, 1. a) 

EIF Principles  Recommendation 
5 

Section 1 - Transparency and modalities, Article 12 Transparent information 

Art. 20 - Right to data portability 

EIF Principles  Recommendation 
9 

Appropriate security of the personal data (Art. 5, 1. f) 

EIF Principles  Recommendation 

Processing  of  special  categories  of  personal  data:  social  security  and  social 
protection law (Art. 9, b) 

Security  of  personal  data  -  Security  of  processing  -  ensure  a  level  of  security 
appropriate to the risk (Art. 32, 1) 

15 

Art. 30 - Records of processing activities 

Standardised icons, machine-readable (Art. 12, 7) 

EIF Principles  Recommendation 

18 

Interoperability 
Layers 

Recommendation 
30 

130 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

Publicly accessible sources (Art. 14) 

Standardised icons, machine-readable (Art. 12, 7) 

Right to data portability - structured, commonly used and machine-readable format 
(Art. 20, 1) 

Data protection and protection of free movement of data (Art. 98) 

Processing of special categories of personal data (Art. 9, g) 

Data protection by design and by default (Art. 25) 

Data protection impact assessment and prior consultation (Section 3, Art. 35) 

Article 37 Designation of the data protection officer (Section 4, Art. 37) 

Appropriate security of the personal data (Art. 5, f) 

Security of personal data, security of processing (Section 2, Art. 32) 

EIF Core 
Topics 

EIF pillar 

Conceptual 
model 

Recommendation 
36 

Conceptual 
model 

Recommendation 
42 

Conceptual 
model 

Recommendation 
46 

Short 
title:  Web 
accessibility  directive 

of 

Title:  Directive  (EU) 
2016/2102 
the 
European  Parliament 
and  of  the  Council  of 
26  October  2016  on 
the  accessibility  of 
websites  and  mobile 
applications  of  public 

Article  5(1):  Member  States  shall  ensure  that  public  sector  bodies  apply  the 
accessibility requirements set out in Article 4 to the extent that those requirements 
do not impose a disproportionate burden on the public sector bodies for the purposes 
of that Article. 

EIF Principles  Recommendation 
1 

Art 7 – benefits to users and to owners of websites and mobile applications, and of 
the possibility of giving feedback in the case of any failure 

EIF Principles  Recommendation 

12 

Art  1  – (…) thereby enabling those websites and mobile applications to  be more 
accessible to users, in particular to persons with disabilities. 

EIF Principles  Recommendation 

14 

131 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

sector  bodies, 
and 
work  launched  by  the 
Commission 
a 
‘European 
Accessibility Act’  

on 

Short description 

EIF Core 
Topics 

EIF pillar 

Periodically monitor the compliance of websites and mobile applications of public 
sector bodies with the accessibility requirements - monitoring methodology (Art. 8) 

EIF Principles  Recommendation 
8 

Article  5  –  Member  States  shall  ensure  that  public  sector  bodies  apply  the 
accessibility requirements set out in Article 4 to the extent that those requirements 
do not impose a disproportionate burden on the public sector bodies for the purposes 
of that Article. 

EIF Principles  Recommendation 

17 

Article  6  –  Content  of  websites  and  mobile  applications  that  meets  harmonised 
standards  

Interoperability 
Layers 

Recommendation 
22 

Art  7  –  the  accessibility  statement  shall  be  provided  in  an  accessible  format,  
Provide data […] in a format which can be used by public sector bodies (Art. 8) 

Interoperability 
Layers 

Recommendation 
30 

Shor 
Governance 

title: 

Data 
Act 

The re-use of data (Art. 4, 1) 

Conditions for reuse (Art 5) 

EIF Principles  Recommendation 
7 

of 

Title:  Proposal  for  a 
the 
Regulation 
European  Parliament 
and of the Council on 
European 
data 
governance 
COM(2020) 767 final  

Conditions  for  re-use  [of  data]  shall  be  non-discriminatory,  proportionate  and 
objectively justified (Art. 5, 2) 

Public sector bodies (...) shall make publicly available the conditions for reuse (Art. 
5, 1) 

EIF Principles  Recommendation 
5 

The  award  of  an  exclusive  right  (…)  shall  be  transparent  and  be  made  publicly 
available online (Art 4.6) 

A single information point (Art. 8, 1) 

EIF Principles  Recommendation 

11 

132 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

Public sector bodies may impose obligations (a)to access and re-use the data within 
a secure processing environment (Art 5) 

EIF Principles  Recommendation 

15 

a European Data Innovation Board should be established, in the form of an expert 
group. 

Interoperability 
Layers 

Recommendation 
29  

Committee  (Art.  29)  Chapter  III  on  requirements  for  providers  of  data  sharing 
services 

Security measures (Art. 11) 

The  public  sector  body  (...)  shall  take  all  reasonable  technical,  legal  and 
organisational measures in order to prevent transfer or access to non-personal data 
held in the Union where such transfer or access would create a conflict with Union 
law or the law of the relevant Member State, (Art 30) 

Conceptual 
model 

Recommendation 
46 

availability of data (Art.1) Access to data by competent authorities (Art.5.1)  

EIF Principles  Recommendation 
2 

The  single  points  of  contact  shall  provide  users  with  general  information  on  this 
Regulation, including on the codes of conduct. (Art. 7) 

EIF Principles  Recommendation 

11 

Data  localisation  requirements  shall  be  prohibited,  unless  they  are  justified  on 
grounds of public security in compliance with the principle of proportionality.(Art 
4) 

Conceptual 
model 

Recommendation 
46 

Short title: Regulation 
on  the  free  flow  of 
data 
non-personal 

of 

Title: Regulation (EU) 
2018/1807 
the 
European  Parliament 
and  of  the  Council  of 
14 November 2018 on 
a  framework  for  the 
free 
flow  of  non-
personal  data  in  the 
European Union   

133 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

Short 
eProcurement 
Directive 

title: 

shall  not  restrict  economic  operators’  access  (Art.22.1)  completely  electronic 
process, and shall be open (Art.34) unrestricted and full direct access free of charge 
(Art.22.5, Art.54.1) 

EIF Principles  Recommendation 
3 

Directive 
Title: 
2014/24/EU  of 
the 
European  Parliament 
and  of  the  Council  of 
26 February  2014  on 
procurement 
public 
and 
repealing 
Directive 2004/18/EC  

transparency (Art.56.3, 76.1) 

non-discriminatory,  generally  available  (Art  22.1)  suitable  alternative  means  of 
access, alternative channel for electronic submission (Art. 22.5) principles of equal 
treatment (Art.56.3, 76.1) 

EIF Principles  Recommendation 
5 

EIF Principles  Recommendation 

14 

shall  specify  the  level  of  security  required  for  the  electronic  means  of 
communication  in  the  various  stages  of  the  specific  procurement  procedure; 
(Art.22.6) 

EIF Principles  Recommendation 

15 

electronic  tools,  building  information  electronic  modelling  tools  (Art.22.4) 
electronic auction (Art.35) format of an electronic catalogue (Art.36) 

Interoperability 
Layers 

Recommendation 
33 

Title  IV  -  Governance,  enforcement  (Art.83)  format  of  standard  forms  (Art.51, 
Annex VIII) 

Interoperability 
Layers 

Recommendation 
25 

level of security (Art.22.6) 

Electronic catalogues (Art.22, 7) 

Electronic catalogues (Art.36) 

title: 

Short 
Regulation 

eIDAS 

technology neutral and does not discriminate (Art.12.3a) 

134 

Conceptual 
model 

Recommendation 
46 

Conceptual 
model 

Recommendation 
44 

EIF Principles  Recommendation 
8 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Title: Regulation (EU) 
No  910/2014  of  the 
European  Parliament 
and  of  the  Council  of 
July  2014  on 
23 
electronic 
identification and trust 
services  for  electronic 
transactions 
the 
internal  market  and 
repealing  Directive 
1999/93/EC  

in 

Short description 

EIF Core 
Topics 

EIF pillar 

accessible for persons with disabilities (Art.15) 

EIF Principles  Recommendation 

processing  of  [only  those]  identification  data  that  are  adequate,  relevant  and  not 
excessive (clause 11) personal data breaches (clause 31) adequate level of security 
of electronic identification (Art.1) Processing of personal data (Art.5.1) breached or 
partly compromised (Art.10.1) principle of privacy by design (Art.12.3c) personal 
data is processed (Art.12.3d) 

14 

EIF Principles  Recommendation 

15 

Article  34  -  Qualified  preservation  service  for  qualified  electronic  signatures 
Article 40 - Validation and preservation of qualified electronic seals 

EIF Principles  Recommendation 

18 

shall  be  recognised  (Art.6.1)  interoperable  (Art.12.1)  interoperability  framework 
(Art.12.2;3) 

Interoperability 
Layers 

Recommendation 
20  

Art.  1c  ‘establishes  a  legal  framework  for  electronic  signatures,  electronic  seals, 
electronic time stamps, electronic documents, electronic registered delivery services 
and certificate services for website authentication.’ 

Interoperability 
Layers 

Recommendation 
27 

Art.  17  -  Supervisory  body:  ‘Member  States  shall  designate  a  supervisory  body 
established in their territory or, upon mutual agreement with another Member State, 
a supervisory body established in that other Member State. (…)’ 

Interoperability 
Layers 

Recommendation 
29 

Art  18.  -  Mutual  assistance:  ‘Supervisory  bodies  shall  cooperate  with  a  view  to 
exchanging good practice’ 

Minimum 
standards (Art.12.4g) 

technical  requirements  (Art.12.4a)  common  operational  security 

Interoperability 
Layers 

Recommendation 
33 

135 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

Adequate level of security of electronic identification means and trust services 
(Chapt 1, Art.1) 

Conceptual 
model 

Trust services (Art. 4) 

the technical and security specifications of the issued electronic identification 
means (Art. 8, 3, F) 

Recommendation 
46 

Recommendation 
47 

Security breach (Art. 10) 

Security standards (Art. 12) 

Supervisory bodies to inform about breaches of security (Art. 17) 

Security requirements applicable to trust service providers (Art. 19) 

security and personal data protection rules (Art. 24) 

electronic signature at a higher security level (Art. 27) 

security evaluation process carried out in accordance with one of the standards for 
the security assessment of 

information technology products (Art. 30, 3 (a)) 

The security of the duplicated datasets must be at the same level as for the original 
datasets (Annex II, 4. (a)) 

implementation of the principle of privacy by design (Art.12) 

Short  title  and  title: 
Commission 
Implementing 
Decision 
2015/296 
Regulation  

(EU) 
-  eIDAS 

Article  1  –  cooperation:  ‘procedural  arrangements  for  facilitating  cooperation 
between Member States, as is necessary in order to ensure the interoperability and 
security of electronic identification schemes’ 

Interoperability 
Layers 

Recommendation 
20  

Article 3 – Points of single contact 

EIF Principles  Recommendation 

11 

136 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short  title  and  title: 
Commission 
Implementing 
Regulation 
(EU) 
2015/1501  -  eIDAS 
Regulation  

Short  title:  European 
Directive  on  patients’ 
rights  

Directive 
Title: 
2011/24/EU  of 
the 
European  Parliament 
and  of  the  Council  of 
9 March  2011  on  the 
of 
application 
in 
patients’ 

rights 

Short description 

EIF Core 
Topics 

EIF pillar 

Article 3: Minimum technical requirements related to the assurance levels 

Article 12: Technical specifications 

Interoperability 
Layers 

Recommendation 
21 

Article 7: Data integrity and authenticity for the communication 

EIF Principles  Recommendation 

Article 9: Management of security information and metadata 

Article 10: Information assurance and security standards 

Article 8: Message format for the communication 

Article 9: Management of security information and metadata 

Article 10: Information assurance and security standards 

ICT  and  other  forms  of  cross-border  cooperation  (Art.10.2)  development  of 
European Reference Networks (Art.12.1) support and facilitate cooperation and the 
exchange  of  information  (Art.14.1;  15.1)  health  technology  assessment  network 
(Art.15) assisted by a Committee (Art.16)  

15 

Interoperability 
Layers 

Recommendation 
30 

Conceptual 
model 

Recommendation 
46 

Interoperability 
Layers 

Recommendation 
29 

Member States shall  facilitate cooperation in  cross-border healthcare provision at 
regional  and  local  level  as  well  as  through  ICT  and  other  forms  of  cross-border 
cooperation. (Art.10) 

Interoperability 
Layers 

Recommendation 
20 

137 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

cross-border 
healthcare  

Short description 

EIF Core 
Topics 

EIF pillar 

The objectives of the eHealth network shall be  to  (…) support Member States in 
developing  common  identification  and  authentication  measures  to  facilitate 
transferability of data in cross-border healthcare. (Article 14) 

EIF Principles  Recommendation 
9 

provision  of  objective,  reliable,  timely,  transparent,  comparable  and  transferable 
information (Article 15) 

National  contact  points  shall  provide  patients  on  request  with  contact  details  of 
national contact points in other Member States. (Art 6) 

EIF Principles  Recommendation 

11 

The  principle  of  non-discrimination  with  regard  to  nationality  shall  be  applied  to 
patients from other Member States. (Art. 4) 

EIF Principles  Recommendation 

14 

easily  accessible;  formats  accessible 
easily accessible; objectively and impartially (Art.9.2)  

to  people  with  disabilities  (Art.6.5)  

the fundamental right to privacy with respect to the processing of personal data is 
protected (Art. 4) 

EIF Principles  Recommendation 

15 

due respect of data protection (Art. 11.2a) 

principles of data protection (Art. 14.2)  

standards and guidelines on quality and safety (Chapter II, Art. 4, 1 (b)) 

guidelines  supporting  the  Member  States  in  developing  the  interoperability  of 
ePrescriptions (Art. 11, 2 (b)) 

produce good practice guidelines and to implement outcome measures and quality 
control (Art. 12, 4 (a)) 

Draw up guidelines on (...) (Art. 14, 2 (b)) 

Conceptual 
model 

Recommendation 
44 

138 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

achieving  a  high  level  of  trust  and  security,  enhancing  continuity  of  care  and 
ensuring access to safe and high-quality healthcare (Art. 14) 

Conceptual 
model 

Recommendation 
46 

privacy with respect to the processing of personal data (Chap II, Art. 4, 2 (e)) 

Short title: Regulation 
on  Interoperability  in 
the field of police and 
judicial  cooperation, 
asylum and migration. 

of 

Title: Regulation (EU) 
2019/818 
the 
European  Parliament 
and  of  the  Council  of 
20  May  2019  on 
a 
establishing 
for 
framework 
interoperability 
between 
EU 
information systems in 
the field of police and 
judicial  cooperation, 
asylum  and  migration 
amending 
and 

Art. 27 - The multiple-identity detection shall only be launched in order to compare 
data  available  in  one  EU  information  system  with  data  available  in  other  EU 
information systems 

EIF Principles  Recommendation 
6Recommendation 
7 

in  accordance  with  their  access  rights  (Art.6.1;  7;8;18.3)  Full  access  to  the  data 
(Art.22.3) access shall be granted to (Art. 26) not result in discrimination (Art.5)  

EIF Principles  Recommendation 

14 

 search  data  related  to  persons  or  their  travel  documents  (Art.7.5)  only  for  data 
protection monitoring (Art.10.2) minimum data quality standard (Art.13.3) only for 
as long as the corresponding biometric data are stored (Art.15; 35) data retention 
provisions  (Art.23.1)  regularly  verified  by  the  competent  supervisory  authority 
(Art.24.4)  security  of  processing  (Art.42)  compliance  of  data  processing  (Art.44) 
right of access to, rectification and erasure of personal data (Art.48) audit of personal 
data processing operations (Art.52) 

EIF Principles  Recommendation 

15 

clear and plain language, in a linguistic version the person concerned understands or 
is reasonably expected to understand (Art.47.2)  

EIF Principles  Recommendation 

16 

 keep logs (Art.10; Art.16; 24; 36) the data [..] shall be stored (Art.15; 35) 

EIF Principles  Recommendation 

18 

139 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

(EU) 
Regulations 
(EU) 
2018/1726, 
2018/1862  and  (EU) 
2019/816 

Short description 

EIF Core 
Topics 

EIF pillar 

Art 13 - 4. The storage of the data referred to in paragraph 1 shall meet the quality 
standards referred to in Article 37(2).  

Interoperability 
Layers 

Recommendation 
22 

data  quality  control  mechanisms  and  common  data  quality  indicators  (Art.37.1) 
interoperability components (Art. 1.2) data quality requirements (art. 1.3) improving 
data  quality  and  harmonising  the  quality  requirements  for  the  data  (Art.  2.2c) 
automated  quality  check  of  the  biometric  data  (Art.13)  technically  impossible 
because of a failure of the CIR (Art.17) implement mechanisms for evaluating the 
accuracy (37.2) 

universal message format (UMF) (Art.1.3; 38) 

Interoperability 
Layers 

Recommendation 
33 

Interoperability 
Layers 

Recommendation 
30 

140 

 
 
EIF Core 
Topics 

EIF pillar 

Conceptual 
model 

Recommendation 
46 

Name of the legal 
initiative (with 
hyperlink) 

Short description 

 area of freedom, security and justice (Chap I, Art. 1) 

 to  contribute to  a high level  of security within the area of  freedom,  security and 
justice (Art. 2 , 1(a)) 

strengthening,  simplifying  and  making  more  uniform  the  data  security  and  data 
protection conditions (Art. 2, 2 (e)) 

ensuring data security (Art. 16, 3) 

The CIR shall reply in such a way that the security of the data is not compromised. 
(Art.22, 2) 

security rules applicable to the repository (Art. 39, 5) 

Security of processing (Art. 42) 

Security incidents (Art. 43) 

with the conditions of security, availability, quality and performance (Art. 54, 2) 

 compliance with the rules of each EU information system regarding the security and 
integrity of personal data (Art. 56, 1(h)) 

security training - appropriate training programmes concerning data security, data 
quality, data protection rules, the procedures applicable to data processing and the 
obligations (Art. 72) 

Monitoring and evaluation - security and quality of service (Art. 74) 

an assessment of the security of the interoperability components (Art. 74, (e)) 

141 

 
 
Short description 

protection  of  personal  data,  privacy  (Art.  1.1;3)  security  of  its  services,  network 
security (Art.4.1) personal data, security policy (Art.4.1a) breach of the security of 
the  network 
(Art.4.3)  confidentiality  of 
communications and the related traffic data (Art.5) erased or made anonymous when 
it is no longer needed (Art.6.1) for the duration necessary (Art. 6.3; 9) anonymous, 
or with the consent (Art.9.1) 

(Art.4.1)  personal  data  breach 

transparent procedures (Art.10) 

Member States shall ensure that no mandatory requirements for specific technical 
features  are  imposed  on  terminal  or  other  electronic  communication  equipment 
which could impede the placing of equipment on the market and the free circulation 
of such equipment in and between Member States. (Art 14) 

EIF Core 
Topics 

EIF pillar 

EIF Principles  Recommendation 

15 

EIF Principles  Recommendation 
5 

EIF Principles  Recommendation 
8 

Name of the legal 
initiative (with 
hyperlink) 

title:  Privacy 
Electronic 

Short 
and 
Communications 
Directive 

Directive 
Title: 
2002/58/EC  of 
the 
European  Parliament 
and  of  the  Council  of 
2002 
12 
July 
concerning 
the 
processing of personal 
data and the protection 
of  privacy 
the 
electronic 
communications 
sector  (Directive  on 
privacy and electronic 
communications) 

in 

Member States shall ensure that subscribers are informed, free of charge and before 
they  are included in  the  directory,  about  the purpose(s) of a  printed or  electronic 
directory  of  subscribers  available  to  the  public  or  obtainable  through  directory 
enquiry services, in  which their personal  data  can be included and of any  further 
usage possibilities based on search functions embedded in electronic versions of the 
directory. (Art 12) 

Conceptual 
model 

Recommendation 
36 

142 

 
 
 
Short description 

Privacy (Art. 1) 

Privacy (Art. 7) 

EIF Core 
Topics 

EIF pillar 

Conceptual 
model 

Recommendation 
46 

Data protection and privacy (Chapter II, Art. 5, (5)c) 

Article 41: Protection of personal data 

Conceptual 
model 

A  European  cybersecurity  certification  scheme  shall  be  designed  to  achieve,  as 
applicable,  at  least  the  following  security  objectives:  (a)  to  protect  stored, 
transmitted or otherwise processed data against accidental or unauthorised storage, 
processing, access or disclosure during the entire life cycle of the ICT product, ICT 
service or ICT process; (Art 51) 

Name of the legal 
initiative (with 
hyperlink) 

Short 
Cybersecurity 

title: 
Act 

of 

Title: Regulation (EU) 
the 
2019/881 
European  Parliament 
and  of  the  Council  of 
17  April  2019  on 
ENISA  (the  European 
Union  Agency 
for 
Cybersecurity) and on 
information 
and 
communications 
technology 
cybersecurity 
and 
certification 
repealing  Regulation 
(EU) No 526/2013   

Short 
Directive 

title: 

NIS 

necessary  ;  relevant  and  proportionate  (Art.1.5)  appropriate  and  proportionate 
(Art.14.1; 16.1) 

EIF Principles  Recommendation 
1 

Title:  Directive  (EU) 

143 

 
 
 
  
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

of 

2016/1148 
the 
European  Parliament 
and of the Council of 6 
July  2016  concerning 
measures  for  a  high 
of 
common 
security  of  network 
information 
and 
the 
systems  across 
Union  

level 

without  imposing  or  discriminating  in  favour  of  the  use  of  a  particular  type  of 
technology (Art.19.1) 

EIF Principles  Recommendation 
8  

security of network and information systems (Art.1.7; 7.1; 14.1; 14.2; 16.1; 16.2; 
19.1)  adequate  protection  of  data  (Art.13)  confidentiality  of  the  information 
(Art.14.5; 16.6) personal data breaches (Art.15.4) 

EIF Principles  Recommendation 

15 

Art. 20 - Voluntary notifications shall only be processed where such processing does 
not constitute a disproportionate or undue burden on Member States concerned 

EIF Principles  Recommendation 

17 

Article  19  -  Standardisation:  encourage  the  use  of  European  or  internationally 
accepted  standards  and  specifications  relevant  to  the  security  of  network  and 
information systems.  

Interoperability 
Layers 

Recommendation 
22 

Cooperation  Group  (Art.5.6;  11.1)  designate  one  or  more  national  competent 
authorities  (Art.8.1)  CSIRTs  (Art.9)  cooperate  (Art.10.1)  network  of  the  national 
CSIRTs (Art.12.1) in  close cooperation with  (Art.15.4) Network and  Information 
Systems Security Committee (Art.22) 

Interoperability 
Layers 

Recommendation 
29 

Issuing guidelines in order to facilitate the convergence of operational practices (Art. 
12, 3, j) 

Conceptual 
model 

Recommendation 
44 

Guidelines (Art. 14.7) 

144 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

Security and privacy of data (Art. 1) 

National data protection authorities (Art. 8) 

Ensure adequate protection of data (Art. 12) 

EIF Core 
Topics 

EIF pillar 

Conceptual 
model 

Recommendation 
46 

The  competent  authority  shall  work  in  close  cooperation  with  data  protection 
authorities (Art.15) 

Short  title:  European 
Electronic 
Communications 
Code 

Member States shall ensure that in carrying out the regulatory tasks specified in this 
Directive, the national regulatory and other competent authorities take all reasonable 
measures which are necessary and proportionate for achieving the objectives set out 
in paragraph 2. (Art 3) 

EIF Principles  Recommendation 
1 

of 

Title:  Directive  (EU) 
2018/1972 
the 
European  Parliament 
and  of  the  Council  of 
11  December  2018 
establishing 
the 
European  Electronic 
Communications 
Code 

Member  States  shall  ensure  that,  where  national  regulatory  or  other  competent 
authorities  intend  to  take  measures  in  accordance  with  this  Directive,  they  give 
interested  parties  the  opportunity  to  comment  on  the  draft  measure  within  a 
reasonable period (Art 23 - Transparency mechanism) 

EIF Principles  Recommendation 
5 

Member States shall encourage the use of the standards or specifications referred to 
in  paragraph  1  for  the  provision  of  services,  technical  interfaces  or  network 
functions, to the extent strictly necessary to ensure interoperability of services, end-
to-end  connectivity,  facilitation  of  provider  switching  and  portability  of  numbers 
and  identifiers,  and  to  improve  freedom  of  choice  for  users.  (Art  19.2 
standardisation) 

Interoperability 
Layers 

Recommendation 
23 

Guidelines publication (Art. 12) 

Issue guidelines to assist national regulatory and/or other competent authorities on 
the consistent implementation (Art. 22, 7) 

Conceptual 
model 

Recommendation 
44 

Guidelines (Art. 32) 

145 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

Guidelines (Art. 63) 

security of networks (Art. 1, subject matter) 

by  maintaining  the  security  of  networks  and  services,  by  ensuring  a  high  and 
common level of protection for end-users (Art. 3, 2 (d)) 

Security of networks and services - SECURITY TITLE V (Art. 40) 

provide information needed to assess the security of their networks and services (Art. 
41, 2 (a)) 

submit to a security audit (Art. 41, 2 (b)) 

user-centric, user-friendly  ( Art. 9.1;  10.1;  Art. 25.1;  26.1 ) ‘once-only’ principle 
(Art.  1.1b;  14)  easy  online  access  through  different  channels  (Art.  7.1)  clear  and 
plain language (Art.9.1) easy to use (Art. 18.4a) not contain any unnecessary full or 
partial duplication and overlaps (Art. 19.6) common assistance service finder (Art. 
20) feedback tool (Art. 25)  

Conceptual 
model 

Recommendation 
46 

EIF Principles  Recommendation 

10 

Recommendation 
11 

Recommendation 
12 

Recommendation 
13 

provide citizens and businesses with easy access (Art.1.1a) easy, online access on 
(Art.  4.1;  4.2)  can  access  and  complete  any  of  the  procedures  (Art.  6.1)  in 
discrimination  against  (Art.  6.3)  accessed  and  completed  online  by  cross-border 
users, non-discriminatory way (Art. 13.1) without discrimination (Art. 16c) easily 
found (Art. 23) 

EIF Principles  Recommendation 

14 

146 

title:  Single 
Gateway 

Short 
Digital 
Regulation 

of 

Title: Regulation (EU) 
2018/1724 
the 
European  Parliament 
and of the Council of 2 
2018 
October 
establishing  a  single 
to 
digital  gateway 
to 
access 
provide 
to 
information, 
to 
procedures 
and 
assistance 

and 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

problem-solving 
services and amending 
Regulation  (EU)  No 
1024/2012  

Short description 

EIF Core 
Topics 

EIF pillar 

coordination group (clause 68; 29;30) national coordinator (Art. 28.1) Committee 
(Art. 37) 

Interoperability 
Layers 

Recommendation 
29 

accessible in all official languages of the Union (Art.2.3) additional languages (Art. 
11.1e) translations (Art. 12.1) 

EIF Principles  Recommendation 

16 

technical system (Art. 14) common user interface (Art.18)  

Interoperability 
Layers 

Recommendation 
33 

accessible  online  through  various  electronic  devices  (Art.  18.4b)  developed  and 
optimised for different web browsers (Art. 18.4c) 

EIF Principles  Recommendation 
9 

Those data shall be made available to the public in an open and commonly used, 
machine-readable format (Art. 24,3) 

Conceptual 
model 

Recommendation 
42 

it includes references, links to legal acts, technical specifications and guidelines (Art. 
9, 1 (c)) 

Conceptual 
model 

Recommendation 
44 

ensure a high level of security for the transmission and processing of evidence (Art. 
14, 3 (h)) 

Conceptual 
model 

Recommendation 
47 

security and hosting of the following ICT applications (Art. 21, 1) 

discuss the application of the principles of security by design and privacy by design 
in the context of this Regulation (Art. 30, 1 (l)) 

Art. 33 - Protection of personal data 

Art. 6 - Procedures to be offered fully online 

EIF Principles  Recommendation 
5 

147 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

Article 6(1) - digitisation: users can access and complete any of the procedures listed 
in Annex II fully online (…) 

EIF Principles  Recommendation 

17 

Chapter VII - Governance of the Gateway 

Articles 28, 29, 30, 31: governance bodies  

Article 10: the type and format of evidence to be submitted 

Short  title:  eInvoicing 
Directive 

Directive 
Title: 
2014/55/EU  of 
the 
European  Parliament 
and  of  the  Council  of 
16  April  2014  on 
electronic invoicing in 
public procurement  

Article 7 Receipt and processing of electronic invoices 

Member  States  shall  ensure  that  contracting  authorities  and  contracting  entities 
receive and process electronic invoices which comply with the European standard 
on electronic invoicing whose reference has been published pursuant to Article 3(2) 
and with any of the syntaxes on the list published pursuant to Article 3(2). 

Article 4 Formal objections to the European standard 

1. Where a Member State or the European Parliament considers that the European 
standard on electronic invoicing and the list of syntaxes do not entirely satisfy the 
requirements set out in Article 3(1) 

Interoperability 
Layers 

Recommendation 
25 

Interoperability 
Layers 

Recommendation 
30 

Interoperability 
Layers 

Recommendation 
22 

Interoperability 
Layers 

Recommendation 
30 

data protection, principle of the protection of privacy (Art. 8) 

EIF Principles  Recommendation 

15 

special security measures in accordance with the laws, regulations or administrative 
provisions In force and transposed in a Member State (Art. 1) 

Conceptual 
model 

Recommendation 
46 

Data protection and the principle of the protection of privacy (Art. 8, 3) 

148 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

title:  Services 

Short 
Directive 

Directive 
Title: 
the 
2006/123/EC  of 
European  Parliament 
and  of  the  Council  of 
12 December 2006 on 
services in the internal 
market  

Short description 

EIF Core 
Topics 

EIF pillar 

Article 6 1. a) including applications for inclusion 

EIF Principles  Recommendation 

14 

Article 5 1. Member States shall examine the procedures and formalities applicable 
to access to a service activity and to the exercise thereof. 

3. that they are easily accessible at a distance and by electronic means and that they 
are kept up to date. 

Article 8 1. Member States shall ensure that all procedures and formalities relating 
to access to a service activity and to the exercise thereof may be easily completed, 

Article 9 1. Member States shall not make access to a service activity or the exercise 
thereof  subject  to  an  authorisation  scheme  unless  the  following  conditions  are 
satisfied: 

Article 10 2. g) transparent and accessible. and 4. The authorisation shall enable the 
provider to have access to the service activity. 

Article 13 2. . They shall be easily accessible and any charges which the applicants 
may incur from their application 

Article 14 Prohibited requirements. Member States shall not make access to, or the 
exercise of, a service activity in their territory subject to compliance with any of the 
following. 

Article 16 1. The Member State in which the service is provided shall ensure free 
access to and free exercise of a service activity within its territory 

Article 20 2. Member States shall ensure that the general conditions of access to a 
service, which are made available to the public at large by the provider 

Article 21 1. (a) general information on the requirements applicable in other Member 
States  relating  to  access  to,  and  exercise  of,  service  activities,  in  particular  those 
relating to consumer protection; 

149 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

Article 22 2. (b) is easily accessible to the recipient at the place where the service is 
provided or the contract concluded; and (c) can be easily accessed by the recipient 
electronically by means of an address supplied by the provider; 

Article  26  2.  Member  States  shall  ensure  that  information  on  the  significance  of 
certain labels and the criteria for applying labels and other quality marks relating to 
services can be easily accessed by providers and recipients. 

Article  32  3.  Any  information  in  question  which  is  public  shall  be  accessible  to 
consumers. 

Article 37 2. Member States shall ensure that the codes of  conduct  referred to  in 
paragraph 1 are accessible at a distance, by electronic means² 

150 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

- Article 13 (2) Authorisation procedures and formalities shall not be dissuasive and 
shall not unduly complicate or delay the provision of the service. They shall be easily 
accessible and any charges which the applicants may incur from their application 
shall be reasonable and proportionate to the cost of the authorisation procedures in 
question and shall not exceed the cost of the procedures. 

- Article 10 2. The criteria referred to in paragraph 1 shall be: (c) proportionate to 
that public interest objective; 

- Article 15 about legal system of Member States 2 c) proportionality: requirements 
must be suitable for securing the attainment of the objective pursued; they must not 
go beyond what is necessary to attain that objective and it must not be possible to 
replace  those  requirements  with  other,  less  restrictive  measures  which  attain  the 
same result. 

- Article 16 1. c) proportionality: the requirement must be suitable for attaining the 
objective pursued, and must not go beyond what is necessary to attain that objective. 

- Article 18 1. d) the measures are proportionate. 

-  Article  24,  2.  Professional  rules  on  commercial  communications  shall  be  non-
discriminatory, justified by an overriding reason relating to the public interest and 
proportionate. 

- Article 31 4. are not motivated by the fact that the provider is established in another 
Member State and are proportionate. 

Article 10 2. (g) transparent and accessible. 

Article  12  1.  which  provides  full  guarantees  of  impartiality  and  transparency, 
including, in particular, adequate publicity about the launch, conduct and completion 
of the procedure. 

151 

EIF Core 
Topics 

EIF pillar 

EIF Principles  Recommendation 
1 

EIF Principles  Recommendation 
5 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

CHAPTER II is on ADMINISTRATIVE SIMPLIFICATION 

EIF Principles  Recommendation 

Article  5  Simplification  of  procedures.  1.  Member  States  shall  examine  the 
procedures  and  formalities  applicable  to  access  to  a  service  activity  and  to  the 
exercise thereof. Where procedures and formalities examined under this paragraph 
are not sufficiently simple, Member States shall simplify them. 

Article  26  1.Member  States  shall  (...)  take  accompanying  measures  to  encourage 
providers to take action on a voluntary (...) (a) certification or assessment of their 
activities  by  independent  or  accredited  bodies;  and  4.  4.  Member  States  shall,  in 
cooperation with the Commission, take accompanying measures to encourage the 
development of independent assessments. 

17 

EIF Principles  Recommendation 

19 

Article  7  1.  Member  States  shall  ensure  that  the  following  information  is  easily 
accessible to providers and recipients through the points of single contact 

EIF Principles  Recommendation 

11 

Art. 6: Points of single contact 

152 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short 
title: 
Enforcement 
Directive 

Law 

of 

Title:  Directive  (EU) 
2016/680 
the 
European  Parliament 
and  of  the  Council  of 
27  April  2016  on  the 
protection  of  natural 
persons with regard to 
of 
processing 
the 
personal 
by 
data 
competent  authorities 
for the purposes of the 
prevention, 
investigation, 
or 
detection 
prosecution 
of 
criminal  offences  or 
the 
of 
execution 
criminal penalties, and 
on the free movement 
of  such  data,  and 
repealing 
Council 
Framework  Decision 
2008/977/JHA  

Short description 

processing  of  personal  data  (Art.1.1;  1.2;  2.1;  2.2;  4;  9.1;  9.2;  10)  protection  of 
personal data (Art. 1.2) erasure of personal data (Art. 5; 16) data subjects (Art. 6; 
14.1; 17.1; 20.1; 22.1) processing to be lawful (Art. 8.1) data protection principles, 
data  minimisation  (Art.  20)  by  default  (Art.  20.2)  level  of  security  (Art.  29.1) 
personal data breach (Art. 30;31) appropriate safeguards (Art 36; 370 

EIF Core 
Topics 

EIF pillar 

EIF Principles  Recommendation 

15 

Article  21  1.  They  shall,  in  a  transparent  manner,  determine  their  respective 
responsibilities for compliance with this Directive 

EIF Principles  Recommendation 
5 

Article  43  1.  Member  States  shall  provide  for  each  member  of  their  supervisory 
authorities to be appointed by means of a transparent procedure 

Chapter 2, Article 4 2. (b) processing is necessary and proportionate to that other 
purpose in accordance with Union or Member State law 

EIF Principles  Recommendation 
1 

Article  13  3.  constitutes  a  necessary  and  proportionate  measure  in  a  democratic 
society. 

Article 57 The penalties provided for shall be effective, proportionate and dissuasive 

concise, intelligible and easily accessible form, using clear and plain language (Art. 
12.1; 31.2) 

EIF Principles  Recommendation 

14 

153 

 
 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

Article 5 Time-limits for storage and review 

Member  States  shall  provide  for  appropriate  time  limits  to  be  established  for  the 
erasure  of  personal  data  or  for  a  periodic  review  of  the  need  for  the  storage  of 
personal data. Procedural measures shall ensure that those time limits are observed. 

logs to be kept (Art. 25) 

cooperate with the supervisory authority (Art. 26)  

standardised format (Art. 50.6) 

Information to be made available or given to the data subject (Art. 13) 

That information shall be made available to the supervisory authorities (Art. 15) 

makes available to the controller all information necessary (Art. 22, 3, (e)) 

EIF Core 
Topics 

EIF pillar 

EIF Principles  Recommendation 

18 

Interoperability 
Layers 

Recommendation 
29 

Interoperability 
Layers 

Recommendation 
33 

Conceptual 
model 

Recommendation 
37 

processed in a manner that ensures appropriate security of the personal data (Art. 4, 
1 (f)) general description of the technical and organisational security measures (Art. 
24, 1 (i)) ensuring the integrity and security of the personal data (Art. 25) Section 2 
Security of personal data  

Conceptual 
model 

Recommendation 
46 

Article 29 Security of processing 

Short title: Regulation 
on  Interoperability  in 
the  field  of  justice, 
freedom  and  security 

Art. 27 - The multiple-identity detection shall only be launched in order to compare 
data  available  in  one  EU  information  system  with  data  available  in  other  EU 
information systems 

EIF Principles  Recommendation 
6 

Recommendation 
7 

154 

 
 
Name of the legal 
initiative (with 
hyperlink) 

Short description 

EIF Core 
Topics 

EIF pillar 

of 

Title: Regulation (EU) 
2019/817 
the 
European  Parliament 
and  of  the  Council  of 
20  May  2019  on 
a 
establishing 
framework 
for 
interoperability 
between 
EU 
information systems in 
the  field  of  borders 
and visa and amending 
Regulations  (EC)  No 
(EU) 
767/2008, 
(EU) 
2016/399, 
(EU) 
2017/2226, 
2018/1240, 
(EU) 
2018/1726  and  (EU) 
the 
2018/1861 
European  Parliament 
and of the Council and 
Decisions 
Council 
and 
2004/512/EC 
2008/633/JHA 

of 

 in  accordance  with  their  access  rights  (Art.6.1;  7;8;18.3)  Full  access  to  the  data 
(Art.22.3) access shall be granted to (Art. 26) not result in discrimination (Art.5)  

EIF Principles  Recommendation 

14 

 search  data  related  to  persons  or  their  travel  documents  (Art.7.5)  only  for  data 
protection monitoring (Art.10.2) minimum data quality standard (Art.13.3) only for 
as long as the corresponding biometric data are stored (Art.15; 35) data retention 
provisions  (Art.23.1)  regularly  verified  by  the  competent  supervisory  authority 
(Art.24.4)  security  of  processing  (Art.42)  compliance  of  data  processing  (Art.44) 
right of access to, rectification and erasure of personal data (Art.48) audit of personal 
data processing operations (Art.52) 

EIF Principles  Recommendation 

15 

clear and plain language, in a linguistic version the person concerned understands or 
is reasonably expected to understand (Art.47.2)  

EIF Principles  Recommendation 

16 

keep logs (Art.10; Art.16; 24; 36) the data [..] shall be stored (Art.15; 35) 

EIF Principles  Recommendation 

18 

Art 13 - 4. The storage of the data referred to in paragraph 1 shall meet the quality 
standards referred to in Article 37(2).  

Interoperability 
Layers 

Recommendation 
22 

data  quality  control  mechanisms  and  common  data  quality  indicators  (Art.37.1) 
interoperability components (Art. 1.2) data quality requirements (art. 1.3) improving 
data  quality  and  harmonising  the  quality  requirements  for  the  data  (Art.  2.2c) 
automated  quality  check  of  the  biometric  data  (Art.13)  technically  impossible 
because of a failure of the CIR (Art.17) implement mechanisms for evaluating the 
accuracy (37.2) 

universal message format (UMF) (Art.1.3; 38) 

Interoperability 
Layers 

Recommendation 
33 

Interoperability 
Layers 

Recommendation 
30 

155 

 
 
 
EIF Core 
Topics 

EIF pillar 

Conceptual 
model 

Recommendation 
46 

Name of the legal 
initiative (with 
hyperlink) 

Short description 

 area of freedom, security and justice (Chap I, Art. 1) 

 to  contribute to  a high level  of security within the area of  freedom,  security and 
justice (Art. 2 , 1(a)) 

strengthening,  simplifying  and  making  more  uniform  the  data  security  and  data 
protection conditions (Art. 2, 2 (e)) 

ensuring data security (Art. 16, 3) 

The CIR shall reply in such a way that the security of the data is not compromised. 
(Art.22, 2) 

 security rules applicable to the repository (Art. 39, 5) 

Security of processing (Art. 42) 

Security incidents (Art. 43) 

with the conditions of security, availability, quality and performance (Art. 54, 2) 

 compliance with the rules of each EU information system regarding the security and 
integrity of personal data (Art. 56, 1(h)) 

security training - appropriate training programmes concerning data security, data 
quality, data protection rules, the procedures applicable to data processing and the 
obligations (Art. 72) 

Monitoring and evaluation - security and quality of service (Art. 74) 

an assessment of the security of the interoperability components (Art. 74, (e)) 

156