COMMISSION STAFF WORKING DOCUMENT Interim evaluation of the ISA² programme Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Results of the interim evaluation of the ISA² programme
Date: 2019-09-23
Institutions: Directorate-General for Digital Services, European Commission
Subjects: EU national, EU programme, administrative cooperation, cooperation policy, digitisation, exchange of information, information technology applications, public administration, public authorities, type of business

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 23.9.2019  SWD(2019) 1615 final  COMMISSION STAFF WORKING DOCUMENT Interim evaluation of the ISA² programme Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Results of the interim evaluation of the ISA² programme {COM(2019) 615 final}  
 
 
Table of contents 

1. 

INTRODUCTION ................................................................................................................... 5 

1.1. 

Purpose and scope ........................................................................................................... 5 

2.  BACKGROUND TO THE INTERVENTION ....................................................................... 7 

2.1.  Description of the intervention and its objectives ........................................................... 7 

2.2.  Baseline and points of comparison ................................................................................ 11 

3. 

IMPLEMENTATION / STATE OF PLAY .......................................................................... 12 

3.1.  Description of the current situation ............................................................................... 12 

4.  METHOD .............................................................................................................................. 15 

4.1. 

Short description of methodology ................................................................................. 15 

4.2. 

Limitations and robustness of findings ......................................................................... 18 

5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS............................. 20 

5.1.  Relevance ...................................................................................................................... 20 

5.2. 

Effectiveness ................................................................................................................. 25 

5.3. 

Efficiency ...................................................................................................................... 39 

5.4.  Coherence ...................................................................................................................... 48 

5.5. 

EU added value ............................................................................................................. 57 

5.6.  Utility ............................................................................................................................ 64 

5.7. 

Sustainability ................................................................................................................. 70 

6.  CONCLUSIONS ................................................................................................................... 75 

6.1.  Context .......................................................................................................................... 75 

6.2.  Method .......................................................................................................................... 75 

6.3. 

Findings ......................................................................................................................... 76 

6.4.  Next steps ...................................................................................................................... 78 

ANNEX 1: PROCEDURAL INFORMATION ............................................................................. 79 

ANNEX 2: SYNOPSIS REPORT OF THE CONSULTATION ACTIVITIES ............................ 82 

ANNEX 3: INTERVENTION LOGIC ........................................................................................ 111 

ANNEX 4: EVALUATION FRAMEWORK .............................................................................. 121 

ANNEX 5: SAMPLE OF ACTIONS ........................................................................................... 129 

ANNEX 6: SUPPORTING EVIDENCE FROM DESK RESEARCH ........................................ 135 

1 

 
Glossary 

Term or acronym 

Meaning or definition 

ADMS 

CEA 

CEF 

CEN 

CENELEC 

CEPS 

CPSV-AP 

DCAT-AP 

DG 

DG DIGIT 

DIGIT.D2 

DSM 

EEA 

EFTA 

EIF 

EIRA 

ELISE 

EQ 

ERDF 

ESF 

ESIF 

Asset Description Metadata Schema 

cost-effectiveness analysis 

Connecting Europe Facility 

European Committee for Standardisation 

European Committee for Electrotechnical 
Standardisation 

Centre for European Policy Studies 

Core Public Service Vocabulary Application Profile 
(CPSV-AP) 

Data Catalogue Vocabulary Application Profile for 
Data Portals in Europe 

Directorate-General 

Directorate-General for Informatics 

Directorate-General for Informatics, Directorate D 
Digital Services, Unit D2 Interoperability 

Digital Single Market 

European Economic Area 

European Free Trade Association 

European Interoperability Framework 

European Interoperability Reference Architecture 

European Location Interoperability Solutions for e-
Government 

evaluation question 

European Regional Development Fund 

European Social Fund 

European Structural and Investment Funds 

2 

 
 
ESPD 

EU 

EVM 

Horizon 2020 

IAP 

ICT 

IDA 

IDABC 

IMAPS 

Interoperability 

ISA 

ISA2  

European Single Procurement Document 

European Union 

earned value management 

EU framework programme for research and innovation 
(2014-2020) 

Interoperability Action Plan (Annex I to the 
Communication from the Commission to the European 
Parliament, the Council, the European Economic and 
Social Committee and the Committee of the Regions: 
European Interoperability Framework — 
Implementation Strategy. Interoperability Action Plan, 
Brussels, 23.3.2017, COM(2017) 134 final) 

information and communication technology 

Programme on Interchange of Data between 
Administrations 

Programme on Interoperable Delivery of Pan-European 
eGovernment Services to public administrations, 
Business and Citizens 

Interoperability Maturity Assessment of a Public 
Service 

As explained in Article 2(1) of the ISA2 Decision, 
‘interoperability’ means the ability of diverse 
organisations to interact towards mutually beneficial 
and agreed common goals. It involves the sharing of 
information and knowledge between the organisations, 
through their business processes and by means of the 
exchange of data between their respective ICT systems. 

Programme on interoperability solutions for European 
public administrations 

Programme on interoperability solutions and common 
frameworks for European public administrations, 
businesses and citizens 

ISA2 actions webpage 

https://ec.europa.eu/isa2/actions_en  

ISA2 dashboard 

ISA2 Decision 

https://ec.europa.eu/isa2/dashboard/  

L 318/1 Decision (EU) 2015/2240 of the European 
Parliament and of the Council of 25 November 2015 
establishing a programme on interoperability solutions 
and common frameworks for European public 

3 

 
administrations, businesses and citizens (ISA2 
programme) as a means for modernising the public 
sector, Brussels 4.12.2015. 

ISA2 solutions webpage 

https://ec.europa.eu/isa2/solutions_en  

ISA2 website 

https://ec.europa.eu/isa2/isa2_en  

ISSG 

IT 

JC 

Joinup 

NIFO 

OECD 

PMKI 

REFIT 

RegDel 

SCM 

SEMIC 

SPI 

SRSP 

TOOP 

UNPAN 

VIES 

W3C 

Inter-Service Steering Group 

information technology 

judgment criterion 

collaborative platform facilitating the sharing and reuse 
of IT solutions developed for public administrations 

National Interoperability Framework Observatory 

Organisation for Economic Cooperation and 
Development 

Public Multilingual Knowledge Management 
Infrastructure for the Digital Single Market 

the European Commission's regulatory fitness and 
performance programme 

Inter-institutional Register of Delegated Acts 

standard cost model 

The ISA2 programme action that promotes semantic 
interoperability among EU Member States  

schedule performance index 

Structural Reform Support Programme 

the ‘once-only’ principle project, http://www.toop.eu/ 

United Nations Public Administrations Network 

Value Added Tax Information Exchange System 

World Wide Web Consortium 

4 

 
1. 

INTRODUCTION 

1.1.  Purpose and scope 

ISA2  (interoperability  solutions  for  European  public  administrations,  businesses  and 
citizens)  is  an  EU  programme  that  supports  the  development  of  digital  solutions  and 
common  frameworks  that  enable  public  administrations,  businesses  and  citizens  in 
Europe to benefit from interoperable cross-border and cross-sector public services. 

The  programme’s  primary  beneficiaries  are  European  public  administrations  at  all 
levels  -  EU,  national,  regional  and  local  levels  -  by  reusing  the  solutions  offered  by 
ISA2  they  can  provide  better  –  more  interoperable,  user-centric  and  digital  –  public 
services. However, the programme also benefits a wider group of stakeholders, including 
EU businesses and citizens. 

ISA2 is open to EU Member States, other countries of the European Economic Area and 
candidate countries. In addition to the 28 EU Member States, three other countries take 
part in the programme: Iceland, Norway (since 2016) and Montenegro (since 2018). The 
programme encourages  cooperation with  other non-EU countries and  with international 
organisations and bodies. 

The  ISA2  programme  —  established  by  the  ISA2  Decision1  —  has  a  budget  of  €130.9 
million  and  runs  for  five  years  from  1 January  2016  until  31 December  2020.  In  line 
with  Article 13(3)  of  the  ISA2  decision,  the  Commission  must  carry  out  an  interim 
evaluation of the ISA2 programme by 30 September 2019. Besides fulfilling this legal 
obligation,  the  evaluation  aims  to  improve  the  implementation  of  ISA2  and  any  of  its 
successor programmes. 

The evaluation covers ISA2 activities from the programme’s start until the third quarter 
of  2018  in  all  participating  countries.  The  evaluation  framework  builds  on  the  five 
evaluation  criteria  stemming  from  the  Commission’s  better  regulation  requirements: 
relevance, effectiveness, efficiency, EU added value and coherence. As shown in Box 1, 
it also addresses utility and sustainability in line with the ISA2 Decision. 

1  Decision  (EU)  2015/2240  of  the  European  Parliament  and  of  the  Council  of  25 November  2015 
establishing a programme on interoperability solutions and common frameworks for European public 
administrations,  businesses  and  citizens  (ISA2  programme)  as  a  means  for  modernising  the  public 
sector, Brussels 4.12.2015. 

5 

 
 
 
                                                 
Box 1: Evaluation framework 

The evaluation was based on an evaluation framework composed of seven evaluation criteria and 
10 evaluation questions (EQs). The evaluation criteria and the EQs are summarised below: 

Relevance  refers to  the  alignment  between the  programme's  objectives  and the  evolving  needs 
and problems experienced by stakeholders. 

EQ1: To what extent are the objectives of the ISA² programme still pertinent in relation 
to the evolving needs and problems at both national and EU levels? 

Effectiveness focuses on the extent to which the ISA² programme has achieved the objectives it 
intended to achieve, and generated the results it intended to produce. 

EQ2:  How  far  are  the  ISA²  programme’s  results  in  the  process  of  achieving  the 
programme’s objectives? 

EQ3:  Are  there  aspects  that  are  more  or  less  effective  than  others  are,  and  if  so,  what 
lessons can be drawn from this? 

Efficiency  concerns  the  minimisation  of  costs  borne  by  various  stakeholders  in  achieving  the 
objectives/results identified under the ‘effectiveness’ criterion. 

EQ.4: To what extent has the programme been cost-effective? 

EQ.5: Which aspects of the programme are the most efficient or inefficient, especially in 
terms of resources mobilised? 

EQ5.1:  How  is  the  programme  performing  relative  to  the  planned  work  and 
budget? 

Coherence is a measure of the degree to which the actions supported by the ISA² programme are 
consistent with each other (internal coherence) and with the EU policy framework at large and 
relevant global initiatives (external coherence). 

EQ.6: To what extent do the ISA² actions form part of a ‘holistic’ approach within the 
framework of the programme? (Internal coherence) 

EQ.7:  To  what  extent  is  the  ISA²  programme  coherent  with  other  EU  interventions, 
which  have  similar  objectives  and  with  global  initiatives  in  the  same  field?  (External 
coherence) 

EU added value captures the programme's impacts additional to those that would be achieved if 
the issues addressed by ISA2 were left solely in the hands of national and sub-national authorities. 

EQ.8:  What  is  the  additional  value  resulting  from  the  ISA²  programme,  compared  to 
what  could  reasonably  have  been  expected  from  Member  States  acting  at  national, 
regional and/or local levels? 

Utility refers to (i) the extent to which the results of ISA2 meet stakeholders’ needs and (ii) the 
level of stakeholder satisfaction with the solutions offered by ISA2. 

EQ.9:  How  do  the  ISA²  programme’s  actions  and  results,  achieved  and  anticipated, 
compare with the needs they are supposed to address? 

Sustainability  measures  the  likelihood  of  the  ISA²  programme's  results  lasting  beyond  its 
completion. 

EQ.10:  To  what  extent  is  the  financial,  technical  and  operational  sustainability  of  the 
developed  solutions  —  maintained  and  operated  through  the  ISA²  programme  — 
ensured? 

For further details, please see Annex 4: Evaluation Framework. 

6 

 
2.  BACKGROUND TO THE INTERVENTION 

2.1.  Description of the intervention and its objectives 

The ISA2 programme’s ultimate objective is to promote the ICT-based modernisation of 
the  public  sector  in  Europe  and  to  facilitate  addressing  the  needs  of  businesses  and 
citizens, via improved interoperability of European public administrations. 

More specifically the programme aims to do the following: 

  Facilitate efficient and effective electronic cross-border or cross-sector interaction 

between European public administrations, businesses and citizens. 

  Contribute to the development of a more effective, simplified and user-friendly e-

administration at the national, regional and local administration levels.  

  Promote a holistic approach to interoperability in the EU, by identifying, creating 
and  operating  interoperability  solutions  and  facilitating  their  reuse  by  European 
public  administrations.  This  will  support  the  implementation  of  various  EU 
policies and activities. 

By  working  towards  achieving  the  above  objectives,  the  programme  intends  to  address 
the  problem  of  existing  or  emerging  ‘electronic  barriers  that  impede  the  proper 
functioning of the internal market’2. 

It  is  important  to  note  that  ISA2  is  part  of  a  wider  policy  framework  related  to  the 
digitalisation of public administrations in the EU. In cooperation with the Member States 
and  the  Commission,  the  ISA2  programme  promotes  and  maintains  the  European 
Interoperability  Framework  (EIF)3,  which  has  been  in  place  since  2010  and  was 
revised  in  2017.  The  revision  of  the  EIF  was  called  for  in  the  Communication  on  a 
Digital  Single  Market  Strategy  for  Europe4,  which  recognised 
the  role  of 
interoperability  in  cross-border  and  cross-sector  connections  between  communities, 
public services and public administrations. Other initiatives related to the digitalisation of 
public administrations to which ISA2 contributes include: 

  The eGovernment Action Plans (the current version of which runs from 2016 to 
2020), which aim to make ‘public administrations and institutions in the EU open, 

2  Recital (21) of the ISA2 Decision (see footnote Error! Bookmark not defined.). 
3  Communication  from  the  Commission  to  the  European  Parliament,  the  Council,  the  European 
Economic  and  Social  Committee  and  the  Committee  of  the  Regions,  European  Interoperability 
Framework — Implementation Strategy, Brussels, 23.3.2017, COM(2017) 134 final. 

4  Communication  from  the  Commission  to  the  European  Parliament,  the  Council,  the  European 
Economic and Social Committee and the Committee of the Regions, A Digital Single Market Strategy 
for Europe, Brussels, 6.5.2015, COM(2015) 192 final. 

7 

 
                                                 
efficient  and inclusive,  providing borderless, personalised, user-friendly,  end-to-
end digital public services to all citizens and businesses in the EU’5. 

  The  2017  Tallinn  Declaration  on  eGovernment6,  a  declaration  made  by 
ministers  in  charge  of  eGovernment  policy  across  the  EU  to  spell  out  their 
commitment  to  a  number  of  principles,  including  ‘interoperability  by  default’, 
which aims to ensure that public services are designed to work seamlessly within 
the single market and across organisational silos, relying on the free movement of 
data and digital services in the EU. 

  The  Connecting  Europe  Facility  (CEF)7  established  in  2013  as  a  funding 
programme  providing  support  to  cross-border  interaction  between  the  digital 
services  infrastructures  of  Member  States.  Based  on  interoperability  agreements 
between  Member  States,  CEF  Digital  provides  building  blocks  that  ensure  that 
citizens, businesses and administrations can benefit from seamless digital public 
services across the EU. 

  The  Single  Digital  Gateway8,  to  provide  citizens  and  businesses  with  online 
access  to  information  and  procedures  and  to  assistance  and  problem-solving 
services. 

  The Structural Reform Support Programme (SRSP)9, which provides support 
to  Member  States  in  five  main  reform  areas,  including  IT  capacity  building  in 
public administrations. 

  Several  other  EU  funding  programmes,  like  Horizon  202010  (dedicated  to 
research and innovation), the European Social Fund (ESF)11 and the European 
Regional Development Fund (ERDF)12. 

5  Communication  from  the  Commission  to  the  European  Parliament,  the  Council,  the  European 
Economic  and  Social  Committee  and  the  Committee  of  the  Regions  EU,  eGovernment  Action  Plan 
2016-2020. Accelerating the digital transformation of government, COM/2016/0179 final, p. 3. 

6  Tallinn Declaration on eGovernment, made at a ministerial meeting during the Estonian Presidency of 

the Council of the EU on 6 October 2017. 

7  Regulation (EU) No 1316/2013 of the European Parliament and of the Council of 11 December 2013 
establishing  the  Connecting  Europe  Facility,  amending  Regulation  (EU)  No 913/2010  and  repealing 
Regulations (EC) No 680/2007 and (EC) No 67/2010. 

8  Regulation  (EU)  2018/1724  of  the  European  Parliament  and  of  the  Council  of  2 October  2018 
establishing a single digital gateway to provide access to information, to procedures and to assistance 
and  problem-solving  services  and  amending  Regulation  (EU)  No 1024/2012,  OJ  L  295,  21.11.2018, 
p. 1-38. 

9  Regulation  (EU)  2017/825  of  the  European  Parliament  and  of  the  Council  of  17 May  2017  on  the 
establishment of the Structural Reform Support Programme for the period 2017 to 2020 and amending 
Regulations (EU) No 1303/2013 and (EU) No 1305/2013, OJ L 129, 19.5.2017, p. 1-16. 

10  More information about Horizon 2020 and its support for public sector innovation can be found here: 
in  Horizon  2020’,  https://ec.europa.eu/digital-single-

‘ICT-enabled  public  sector 
market/en/ict-enabled-public-sector-innovation-horizon-2020. 

innovation 

11  More information about activities related to the modernisation of public institutions can be found here: 
Institutions’, 
— 

Effective 

More 

Fund 

‘European 
http://ec.europa.eu/esf/main.jsp?catId=526&langId=en. 

Social 

8 

 
                                                 
Chapter  5  provides  further  details  of  the  links  between  ISA2  and  the  above-mentioned 
programmes and initiatives, under the coherence evaluation criterion (section 5.4). 

The  intervention  logic  aims  to  clarify  the  reasoning  followed  by  EU  decision  makers 
when establishing the ISA2 programme. It includes a detailed description of: 

 
 

 
 
 

the needs, problems and drivers that the programme intended to address;  
the  objectives  set  out  for  the  programme  (following  a  three-level  hierarchy: 
global objective, specific objectives and operational objectives);  
the inputs or activities of the programme; 
the programme's expected results; and  
the logical links between these components.  

The intervention logic elements and links provide a benchmark against which the interim 
evaluation  can  assess  the  programme's  achievements  in  the  first  half  of  its  period  of 
operation. Figure 1 provides an overview of the elements of the intervention logic and the 
links between them, and further details are provided in Annex 3: Intervention Logic. 

12  More information about the ERDF and its support for the ICT modernisation of public administrations 
technologies’, 

‘Regional  Policy  —  Information  and  communication 

can  be 
https://ec.europa.eu/regional_policy/en/policy/themes/ict/. 

found  here: 

9 

 
                                                                                                                                                 
Figure 1: Intervention logic of the ISA2 programme 

10 

 
 
 
2.2.  Baseline and points of comparison 

ISA2  is  the  fifth  in  a  series  of  EU  programmes13  providing  and  promoting 
interoperability solutions for public administrations in the EU. ISA2 succeeded the ISA 
programme, which implemented 51 actions during its 6-year period of operation (from 
2010 to 2015) with a total budget of around €164 million14. The results of ISA represent 
the main baseline used for the purpose of this interim evaluation. 

The end of the ISA programme was marked by a final evaluation15, which took stock of 
the programme’s achievements and put forward three recommendations for the future 
edition of the programme, focusing on the below areas: 

  communication  with  public  administrations  and  raising  their  awareness  of 

interoperability; 

  a holistic approach to and cross-cutting view of interoperability; and 

  cooperation with other EU policies and initiatives. 

Although the present  interim evaluation  focuses  on results achieved under  ISA2, it also 
addresses  the  above-listed  recommendations.  It  is  also  worth  noting  that  ISA2  aims  to 
achieve its objectives through a number of actions that continue from the previous edition 
of the programme, as well as through new actions. Where relevant, the impact of ISA on 
ISA2  was  assessed  by  comparing  the  performance  of  actions  continued  from  ISA 
with that of actions started during ISA2. 

13  1) Community contribution for telematics interchange of Data between Administrations (IDA), 1995-
1997; 2) Second phase of the IDA programme (IDA II), 1999-2004; 3) Interoperable Delivery of Pan-
European  eGovernment  Services  to  public  Administrations,  Business  and  Citizens  (IDABC),  2005-
2009;  4)  Interoperability  Solutions  for  European  Public  Administrations  (ISA),  2010-2015;  5) 
Interoperability  solutions  and  common  frameworks  for  European  public  administrations,  businesses 
and citizens (ISA2), 2016-2020. 

14  Further details can be found at http://ec.europa.eu/archives/isa/. 
15  Report  from  the  Commission  to  the  European  Parliament  and  the  Council,  Results  of  the  final 

evaluation of the ISA programme, Brussels, 1.9.2016, COM(2016) 550 final. 

11 

 
 
 
                                                 
3. 

IMPLEMENTATION / STATE OF PLAY 

3.1.  Description of the current situation 

ISA²  has  been  operational  from  1 January  2016  and  will  continue  to  run  until 
31 December 2020.  It  has  a total  budget  of around €131 million.  ISA2  consists of  nine 
groups  of  actions,  called  packages,  which  are  defined  on  a  yearly  basis  through  an 
annual revision of the rolling work programme (rolling work programme): 

  key and generic interoperability enablers; 

  semantic interoperability16; 

  access to data / data sharing / open data; 

  geospatial solutions; 

  eProcurement / eInvoicing; 

  decision-making and legislation; 

  EU policies — supporting instruments; 

  supporting instruments for public administrations; and 

  accompanying measures. 

The  2018  rolling  work  programme  consists  of  53  actions,  categorised  into  the  nine 
packages  mentioned  above.  ISA2  managed  39  actions  in  its  first  year  of  operation,  43 
actions  in  2017,  53  actions  in  2018,  and  is  currently  managing  54  actions  under  the 
2019 rolling work programme. Table 1 provides a breakdown of actions and allocated 
budget per package and per year. As ISA2 is scheduled to end on 31 December 2020, one 
additional  rolling  work  programme  remains  to  be  adopted.  The  present  evaluation  is 
based on the rolling work programme implemented in 2016, 2017 and 2018, as the 2019 
rolling work programme had not yet been adopted when the data collection took place. 

Table 1: Overview of allocated budget and number of actions per package from 
2016 to 2018 

Package 

Key and generic 
interoperability 
enablers 
Semantic 
interoperability 
Access to data / 
data sharing / 

Allocated 
budget 2016 
(thousand €) 
4 900 

2 008 

2 800 

Actions 
2016 

6 

3 

5 

Allocated 
budget 2017 
(thousand €) 
5 407 

1 831 

3 548 

Actions 
2017 

6 

3 

5 

Allocated 
budget 2018 
(thousand €) 
4 250 

1 503 

3 925 

Actions 
2018 

7 

4 

7 

16 

In the 2016 and 2017 rolling work programmes, the semantic interoperability package is referred to as 
the ‘information interoperability’ package. 

12 

 
                                                 
Package 

Allocated 
budget 2016 
(thousand €) 

Actions 
2016 

Allocated 
budget 2017 
(thousand €) 

Actions 
2017 

Allocated 
budget 2018 
(thousand €) 

Actions 
2018 

1 

1 

6 

1 

1 

983 

2 260 

2 400 

2 608 

1 063 

1 445 

1 900 

2 240 

open data 
package 
Geospatial 
solutions  
eProcurement / 
eInvoicing  
Decision-making 
and legislation 
EU policies — 
supporting 
instruments 
Supporting 
instruments for 
public 
administrations 
Accompanying 
measures 
Total 
Source: ISA2 rolling work programmes (2016, 2017, 2018), indicative planning and financial overview. 

23 991 

25 545 

26 301 

2 235 

3 580 

4 820 

3 533 

5 315 

1 290 

2 935 

4 425 

1 280 

730 

39 

43 

12 

13 

3 

2 

9 

3 

2 

1 

1 

10 

5 

16 

2 

53 

The programme in  its entirety is  overseen by the  Interoperability Unit of DG DIGIT 
(DIGIT.D2), while individual actions are implemented by different Commission services, 
depending  on  the  thematic  scope.  The  Member  States  are  also  involved  in  programme 
governance  through  two  channels:  the  ISA²  Committee,  the  programme's  high-level 
governing  body,  and  the  ISA²  Coordination  Group,  a  technical  body  mandated  to 
ensure  coherence  between  the  programme’s  actions17.  All  28  EU  Member  States 
participate  in  the  programme.  Beyond  the  EU,  ISA2  has  three  additional  countries  — 
Iceland,  Montenegro  and  Norway  —  participate  in  ISA2.  In  addition,  an  administrative 
agreement between DG DIGIT and the Agency for the Development of the Government 
of Electronic Management and the Information and Knowledge Society of Uruguay has 
been in place since March 201818. 

ISA2  actions  are  selected  and  implemented  through  a  process  consisting  of  four  steps 
taken each year: 

  Submission:  Commission  services,  Member  States,  and  other  countries 
participating in ISA2 are invited to submit proposals for actions to be included in 
the rolling work programme via a call for proposals. 

17 

In addition, working groups on specific topics are organised as part of specific actions. For instance, in 
the field of geospatial interoperability solutions, the ‘European Location Interoperability Solutions for 
e-Government’  (ELISE)  action  set  up  the  ISA2  Working  Group  on  Geospatial  Solutions,  bringing 
together  representatives  from  Member  States  active  in  the  field  in  order  to  set  priorities  and 
disseminate results. See: https://ec.europa.eu/isa2/actions/elise_en. 

18  For  more  details,  please  see:  https://ec.europa.eu/isa2/news/european-commission-reinforces-

cooperation-uruguay-interoperability_en. 

13 

 
                                                 
  Evaluation and consultation of the ISA2 Committee: The submitted proposals 
are analysed by DIGIT.D2, which compiles a list of proposals that qualify to be 
included in the rolling work programme, based on the selection and prioritisation 
criteria as defined in the ISA2 Decision. The ISA2 Committee gives an opinion on 
the proposed action list. 

  Adoption:  The  annual  rolling  work  programme  is  adopted  by  the  Commission 

and the budget is released. 

  Implementation: The actions in the adopted annual rolling work programme are 

implemented by the Commission services in charge. 

The  programme's  overall  performance  and  the  progress  of  each  action  are  frequently 
recorded through the quarterly and annual monitoring and evaluation reports, which feed 
into the ISA2 dashboard19. The dashboard is an online interactive tool that facilitates the 
dissemination  of  information  about  action  activities  and  achievements,  and  their 
efficiency, effectiveness and coherence. The efficiency of actions is measured using the 
earned value management  (EVM)  analysis (see  section 5.3  Efficiency),  effectiveness  is 
presented  in  terms  of  performance  indicators  (see  section  5.2  Effectiveness),  and  the 
coherence  of  actions  is  mapped  using  network  analysis  (see  section  5.4  Coherence). 
Figure 2 illustrates the performance of the  ISA2  programme until  Q1 2019, as  captured 
by the ISA2 dashboard based on data from the EVM mechanism. 

Figure 2: Earned value analysis at programme level 

Note: The cumulated planned value is the sum of the planned values of the programme's different actions 
for which the EVM is used. The cumulated earned value is the sum of the earned values of the programme's 
different actions for which the EVM is used. The budget at completion (BAC) is the sum of all undergoing 
and completed specific contracts financed by the ISA2 budget. 
Source: Monitoring team of the ISA2 programme (see also the Efficiency view of the ISA2 Programme). 

19  The dashboard is available at https://ec.europa.eu/isa2/dashboard/. 

14 

 
 
 
 
                                                 
4.  METHOD 

4.1.  Short description of methodology 

As  a  first  step,  the  Commission  outlined  and  agreed  on  the  draft  evaluation  design 
(including  the  intervention  logic  and  the  evaluation  questions).  It  then  reached  out  to 
external  experts  and  asked  them  to  support  the  evaluation  process  (see  Annex  1).  The 
consultant  refined  the  evaluation  design  and  —  with  the  help  and  under  the  close 
monitoring of the Commission — moved to action. First, it collected data, then validated 
and  analysed  them,  as  explained  in  the  following  sections.  At  the  end,  the  consultant 
summarised its findings in an independent evaluation study (referred to as the CEPS final 
study in this report)20. 

Data collection 

The data collection phase involved a mix of quantitative and qualitative data collection 
methods  and  aimed  to  gather  the  evidence  base  to  respond  to  the  evaluation  questions 
(EQs)  specified  in  the  evaluation  framework  (Annex  4).  More  specifically,  data  were 
drawn from two main sources: consultation activities (primary data) and desk research 
(secondary data). 

Primary data 

Primary  data  were  collected  between  29 November  2018  and  1 March  2019  via  the 
following consultation activities21: 

  short  questionnaire  distributed  during  the  ISA2  Mid-Term  Conference  and  the 

kick-off workshop; 

  public consultation; 

 

 

targeted online surveys; 

targeted in-depth interviews; and 

  kick-off workshop. 

These activities resulted in a total of 129 responses from stakeholders, out of which 14 
replies  arrived  during  the  public  consultation.  The  low  consultation  response  rate  is 
mainly explained by two reasons: 

 

ISA2 is a technical programme producing solutions addressed mainly to European 
public  administrations.  Understanding  how  the  programme  works  requires 
specific  knowledge  in  the  field.  Therefore,  direct  contact  between  the  average 

20  CEPS  final  study:  Evaluation  study  supporting  the  interim  evaluation  of  the  programme  on 
interoperability  solutions  for  European  public  administrations,  businesses  and  citizens  (ISA²). 
https://data.europa.eu/doi/10.2799/13397. 

21  For more details, please see  Error! Reference source not found. and Error! Reference source not 

found.. 

15 

 
                                                 
citizen/business  and  the  programme  tends  to  be  very  limited,  and  this  most 
probably led to the reduced response rate. In order to increase the participation of 
citizens/businesses, a short questionnaire with the same questions as those of the 
public consultation was distributed during the ISA2 Mid-Term Conference, which 
resulted  in  15  additional  responses.  It  is  worth  noting,  however,  that  businesses 
and  citizens  benefit  from  ISA2  solutions  and  activities  indirectly,  through  their 
interactions  with  public  administrations.  Thus,  a  low  response  rate  among  these 
groups of stakeholders does not reflect negatively on the programme. 

  Stakeholder  groups  that  are  in  direct  contact  with  the  programme  (e.g.  solution 
users,  action  owners,  etc.)  were  consulted  via  targeted  online  surveys  and 
interviews,  which  covered  inter  alia  the  same  topics  addressed  by  the  public 
consultation. Therefore, stakeholders belonging to these groups were asked not to 
respond to the public consultation in order to avoid duplicate answers. 

Despite  the  relatively  low  response  rate,  the  consultation  activities  allowed  for  a  wide 
coverage  of  the  different  stakeholders  of  the  ISA2  programme,  ranging  from  actors 
involved  in  programme  governance  to  citizens,  i.e.  the  indirect  beneficiaries  of  ISA2. 
Moreover,  on  average,  the  respondents  were  characterised  by  a  high  level  of 
expertise, which contributed to the quality and reliability of primary data collected. The 
in-depth interviews and targeted surveys provided particularly useful input to the interim 
evaluation.  They  were  tailored  to  stakeholders  closely  related  to  the  programme  and 
allowed them to provide extensive details,  thus making it possible for the evaluation to 
get deep insights into the subject areas covered. In general, primary data contributed to 
the  assessment  of  all  evaluation  criteria.  Overall,  the  answers  collected  are  consistent 
across stakeholder groups. 

For  more  details,  please  consult  Annex  2:  Synopsis  report  of  the  consultation 
activities,  which  presents  a  breakdown  of  responses  by  consultation  activity  and 
stakeholder category along with respondent characteristics. 

Chapter  5  of  this  report  presents  the  results  of  the  consultation  activities  using  mainly 
bar charts with averages. The results of the public and targeted consultation activities 
are  combined  and  presented  in  aggregated  form  throughout  the  evaluation.  For  each 
question,  average  scores  were  computed  for  each  stakeholder  group  that  provided 
feedback on the specific question. Averages do not take into account those respondents 
who answered ‘don’t know/no opinion’ to the question under analysis. The data labels of 
the bar charts show the average score first, then the number of respondents in brackets. 

Secondary data 

The desk research reviewed the following resources: 

  publicly available documents and data sources, including the ISA2 annual rolling 
work  programmes,  the  ISA2  dashboard,  and  the  webpages  dedicated  to  actions 
and solutions; and 

16 

 
  additional  operational  documents  including  monitoring  and  evaluation  reports, 
overviews  of  communication  activities,  and  lists  of  participants  in  the  ISA2 
Committee and Coordination Group. 

Annex 6 provides an overview of the supporting evidence collected from desk research. 

Note that, in order to better guide the data collection activities,  a sample of 20 actions 
was selected from the total of 53 actions funded by the ISA2 programme up to 2018. The 
criteria used for sampling and the sampled actions are presented in Annex 5: Sample of 
actions. 

The  desk  research,  review  of  the  programme's  operational  documents  and  targeted  in-
depth interviews focused on the 20 sampled actions. However, stakeholders involved in 
all sampled actions had the opportunity to respond to either the targeted online surveys or 
the public consultation, to participate in the kick-off workshop and to complete the short 
questionnaire distributed during the ISA2 Mid-Term Conference. 

The  data  collection  was  complemented  by  an  expert  assessment  of  the  programme, 
performed  by  four  technical  interoperability  experts  who  were  part  of  the  consultant’s 
evaluation team. 

Data validation 

The collected data were validated via triangulation in order to ensure the robustness of 
evidence.  The  Commission’s  Better  Regulation  Toolbox  defines  triangulation  as  ‘the 
application and combination of several research methodologies in the study of the same 
phenomenon’22.  In  fact,  for  all  evaluation  criteria  and  questions,  data  were  collected 
from  multiple  sources  and  via  at  least  two  different  data  collection  methods  (e.g. 
interviews, targeted questionnaire, public consultation, desk research). 

Quantitative methods of data analysis 

Besides the statistical analysis of data collected via consultation activities, desk research 
and  the  qualitative  analysis  of  open  responses  provided  by  consulted  stakeholders  and 
information contained in documentary evidence, three specific quantitative methods were 
used in order to evaluate the programme's efficiency: 

  The standard cost model (SCM)23 is a method of assessing administrative costs 
imposed by rules or policies inter alia on businesses and public administrations. 
It  is  based  on  the  identification  of  the  basic  components  of  a  rule,  the 
information  obligations,  whose  costs  for  the  addressees  can  be  measured  and 
quantified.  An  information  obligation  is  a  specific  duty  to  gather,  process  or 

22  European Commission (2017), Better Regulation Toolbox, 

https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_2.pdf, p. 20. 

23  European  Commission  (2017),  ‘Tool  #60.  The  standard  cost  model  for  estimating  administrative 

costs’, in the Better Regulation ‘Toolbox’ (see footnote Error! Bookmark not defined.). 

17 

 
                                                 
submit information to the public authority or a third party. The SCM was used to 
measure the costs borne by action owners in preparing and submitting proposals 
for ISA2 actions. 

  Cost-effectiveness  analysis  (CEA)24  is  a  method  of  assessing  the  merits  of  a 
policy in an interim and  ex post evaluation setting. In a nutshell, CEA measures 
the  value-for-money  of  past  policies,  i.e.  the  amount  of  benefit  generated  by 
unitary costs. Costs are measured in monetary terms, whereas effectiveness is 
measured  in  ‘natural  units’,  and  the  unit  of  account  varies  depending  on  the 
nature  of  the  problem  addressed  (e.g.  the  number  of  users  of  key  and  generic 
interoperability enablers). The heterogeneity of performance indicators available 
for  ISA2  actions  makes  it  difficult  to  draw  conclusions  about  the  programme's 
overall cost-effectiveness and thus using CEA was only possible for certain action 
packages and indicators, as described in Chapter 5. 

 

In  line  with  the  PM2  methodology  developed  by  the  Commission25,  the  earned 
value  management  (EVM)  and  earned  schedule  (ES)  methods  are  currently 
used  to  monitor  and  assess  the  efficiency  of  the  ISA2  programme.  EVM  is  a 
project management technique that helps determine work progress against a given 
baseline, so that costs, time, and scope of a certain activity are constantly tracked. 
In  the  context  of  the  ISA2  programme,  efficiency  is  assessed  at  action  and 
programme  levels.  The  implementation  of  EVM  requires  managers  to  calculate 
the earned value, i.e. a quantification of the ‘worth’ of the work done to date, and 
the  actual  costs,  i.e.  the  executed  budget  for  achieving  the  work,  and  to 
compare them  with  the  planned value of such activity. This  allows for  a better 
understanding  of  the  programme's  performance.  The  ES  is  an  extension  of  the 
EVM method that deepens the level of analysis to a ‘units of time’ layer. In the 
framework  of  the  ISA2  programme,  tailored  versions  of  the  EVM  and  ES 
approaches are adopted26. 

4.2.  Limitations and robustness of findings 

The  consultation  strategy27  and  the  data  collection  plan  put  forward  in  the  Evaluation 
Framework  (Annex  4)  were  fully  implemented.  Therefore,  the  evaluation  was  able  to 
draw  robust  conclusions  for  all  EQs  based  on  both  primary  and  secondary  data. 
Nevertheless, some emphasis needs to be placed on caveats that may have an impact on 
the main findings of the interim evaluation as well as on future evaluation exercises: 

24  European  Commission  (2017),  ‘Tool  #57.  Analytical  methods  to  compare  options  or  assess 
performance’, in the Better Regulation Toolbox (see footnote Error! Bookmark not defined.). 

25  European  Commission 

(2018),  PM² 
https://data.europa.eu/doi/10.2799/957700. 

project  management  methodology, 

available 

at 

26  More specifically, the tailored version is based on the Earned Value Management Tutorial, Module 1: 
Introduction  to  Earned  Value  Management,  prepared  by  Booz,  Allen  and  Hamilton,  Department  of 
Energy, the United States of America, and on the guidelines on Earned Schedule in Action, developed 
by Kim Henderson, from the Project Management Institute (PMI) Oklahoma, 13.7.2007. 

27  Please see: https://ec.europa.eu/isa2/sites/isa/files/consultation_strategy_isa2_interim_evaluation.pdf. 

18 

 
                                                 
  The timing of this evaluation is compliant with Article 13 of the ISA2 Decision, 
which requires the Commission to present an interim evaluation to the European 
Parliament  and  the  Council  by  30 September  2019.  Given  this  constraint,  the 
evaluation  confined  the  analysis  to  actions  funded  during  the  programme's  first 
three  years (2016, 2017  and 2018).  In particular, some actions  under evaluation 
are still ongoing; hence, their outputs and outcomes can only be recorded to some 
extent. On a more general note, as discussed in the intervention logic (Annex 3), 
impacts occur over the long term and it is difficult to capture them in an interim 
the  coming  years  ISA2  actions  may  see 
evaluation.  In  principle, 
improvements28 in the way they achieve the programme’s objectives and deliver 
the expected results. 

in 

Only action owners had access to the contact details of their solution users due to 
confidentiality and data protection reasons. Therefore, action owners were requested to: 
i) invite users to complete the relevant targeted online survey; and ii) share with 
DIGIT.D2 the details of those users who gave their consent to participate in interviews. 
This two-step approach may have limited the number of answers received from solution 
users. In addition, some action owners explained that they were unable to contact their 
users for the purpose of evaluation, as the users’ consent to be involved in consultation 
activities was never collected. Against this background, ISA2 may consider requesting its 
users to provide their consent to be contacted by Commission staff and contractors for 
evaluation purposes in the future. 

28  The analysis presented in section 5.2 shows that more mature solutions perform relatively better. For 
further details regarding the performance of indicators for older and newer actions implemented under 
ISA2, please see Error! Reference source not found.. 

19 

 
 
                                                 
5.  ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS 

In  line  with  the  Evaluation  Framework  (Annex  4),  the  interim  evaluation  focused  on 
seven  evaluation  criteria:  relevance,  effectiveness,  efficiency,  coherence,  EU  added 
value,  utility  and  sustainability.  The  below  chapters  present  the  following  for  each  of 
them: a) the pre-defined evaluation questions (EQ) and related judgment criteria; b) the 
results of the analysis fed by the collected data; and finally c) the answer to the EQ. 

5.1.  Relevance 

Evaluation framework for 'relevance' 

Relevance refers to the alignment between the programme's objectives and the evolving 
needs  and  problems29  experienced  by  stakeholders.  As  shown  in  Box  1,  the  interim 
evaluation translated this criterion into the following EQ: 

EQ1:  To what extent are the objectives of the ISA² programme still pertinent in 
relation to the evolving needs and problems at both national and EU levels? 

To  answer  this  question,  the  evaluation  assessed  two  main  judgment  criteria  (JCs),  by 
relying on data collected via consultation activities as well as on expert assessment: 

JC1:  Alignment  between  needs  and  problems  addressed  by  the  programme  and 
current  needs  and  problems  experienced  at  the  EU,  national  and  subnational 
level; and 

JC2:  Alignment  between  the  objectives  of  the  programme  and  current  needs  and 

problems experienced at the EU, national and subnational level. 

Analysis 

JC1: Alignment between original and current needs and problems 

The rapidly evolving nature of ICT and interoperability begs the question as to whether 
the needs and problems originally targeted by the programme are the same as the current 
needs and problems experienced by stakeholders. 

Figure 3 shows the  average scores derived from the  answers received on this question, 
per  stakeholder  group.  Overall,  more  than  70 %  of  respondents  consider  that  the  three 
original  needs,  as  presented  in  Figure  3,  are  still  relevant  to  a  high  extent  or  to  the 
fullest extent (93 out of 128 respondents for the first need, 99 out of 127 respondents for 
the second need, and 94 out of 125 respondents for the third need). The persistence of the 
problem  of  administrative  e-barriers  originally  identified  by  the  programme  is 
confirmed as well. Two-thirds of the respondents (79 out of 118) consider this problem 
still relevant to a high extent or the fullest extent, in comparison to almost 13 % (15 out 
of 118) who believe the problem is relevant only to a limited extent or not at all. 

29  Needs are defined here as prerequisites for the efficient delivery of European public services; problems 

consist of specific bottlenecks that make it impossible to meet these needs. 

20 

 
                                                 
Figure 3: Extent to which needs and problems originally addressed by ISA2 are 
currently experienced by European public administrations, businesses and/or 
citizens (average score of answers, number of respondents) 

Need 1: The need for cooperation among public
administrations with the aim to enable more
efficient and secure public services

Need 2: The need for exchanging information
among public administrations to fulfil legal
requirements or political commitments

Need 3: The need for sharing and reusing
information among public administrations to
increase administrative efficiency and cut red tape
for citizens and businesses

Problem: Administrative e-barriers are leading to
the fragmentation of the Internal Market

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

Action owners

Solution users

0

1

2

3

4

5

Programme governance

Stakeholders responsible for linked EU initiatives

Standardisation organisations

Wider public

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO). 
Source: CEPS final study 

There  are  some  additional  needs  and  problems  that  the  programme  could  aim  to 
address,  as  pointed  out  by  39 %  of  the  consulted  stakeholders  (46  out  of  129).  More 
specifically, several respondents converged on the following needs and problems related 
to  interoperability  of  public  services  that  European  public  administrations,  businesses 
and/or citizens currently experience: 

EU level: 

  The need for a more binding legal framework for interoperability30, including 
the  need  for  legally  binding  standards  or  the  general  need  for  a  more  binding 

30  This  need  was  indicated  by  respondents  from  the  following  stakeholder  groups:  programme 

governance (at the Member State level), solution users and action owners. 

21 

 
 
                                                 
approach to interoperability in the EU. 

  The  need  for  a  more  prescriptive  approach  to  designing  interoperable  public 
services  (‘interoperability  by  design’).  It  is  worth  noting,  however,  that  the 
revised  EIF  adopted  in  2017  explicitly  mentions  ‘interoperability  by  design’  as 
the  conceptual  model  for  interoperable  public  services31.  Therefore,  this  need  is 
taken into account through the wider framework of the EIF. 

  The need to improve the way administrations communicate with one another, 
functioning  of  public 
forms  of  organisation  and 

the  different 

given 
administrations in the Member States. 

  The need to share best practices. 

  The  need  to  account  for  new  developments  such  as  blockchain,  privacy-by-
design  and  self-sovereign  identities32,  which  are  changing  the  interoperability 
landscape. 

National/subnational level: 

  Resource  constraints,  such  as  shortage  of  qualified  IT  staff,  experienced  by 

national and local public administrations. 

  The  different  political  priorities  among  Member  States  hindering  a  consistent 

approach to interoperability in the EU. 

  The  limited  awareness  of  ISA2  and  other  initiatives  related  to  interoperability, 

especially at the regional and local levels33. 

JC2: Alignment between needs and problems and the programme's objectives 

After ensuring that all the original needs and problems are still relevant and identifying 
additional needs  and problems experienced by stakeholders, the  evaluation checked the 
degree to which the programme's objectives address these needs and problems. 

The majority of consulted stakeholders agree that, through its general  objective34, ISA2 
addresses  the  main  needs  and  problems  identified  in  the  field  of  interoperability  of 

31  See Annex 2 to the Communication from the Commission to the European Parliament, the Council, the 
European  Economic  and  Social  Committee  and  the  Committee  of  the  Regions,  European 
Interoperability Framework — Implementation Strategy, Brussels, 23.3.2017, COM(2017) 134 final. 
32  A ‘self-sovereign identity’ is an emerging concept referring to the creation and management of digital 
identities by individuals. For more information, please see: Der, Uwe, Stefan Jähnichen, Jan Sürmeli 
(2017),  Self-sovereign  Identity  −  Opportunities  and  Challenges  for  the  Digital  Revolution,  available 
online at https://arxiv.org/abs/1712.01767. 

33  The way information trickles down from the national level to the regional and local levels is important. 
34  As outlined in the intervention logic, the programme's general objective is as follows: ‘To promote the 
ICT-based  modernisation  of  the  public  sector  in  Europe  and  to  facilitate  addressing  the  needs  of 
businesses  and  citizens  via  improved  interoperability  of  European  public  administrations,  thus 
contributing to the completion of the Digital Single Market and, ultimately, to economic growth and 
the global competitiveness of the European Union’. 

22 

 
                                                 
digital public services at the time the programme was set up, either to a high extent or to 
the  fullest  extent  (Figure  4).  This  result  holds  true  across  all  stakeholder  groups.  Over 
80 %  of  respondents  indicate  that,  by  achieving  its  general  objective,  ISA2  can  address 
the original needs to a high extent or to the fullest extent (103 out of 125 respondents for 
the  first  need,  103  out  of  123  respondents  for  the  second  need,  and  103  out  of  124 
respondents  for  the  third  need).  When  it  comes  to  the  problem  of  administrative  e-
barriers,  approximately  two-thirds  of  respondents  (77  out  of  116)  agree  that  ISA2  can 
address  the  problem  through  its  general  objective,  while  only  11 %  (13  out  of  116) 
believe that the programme can only address the problem to a limited extent or not at all. 
These results are also confirmed by the independent assessment carried out by technical 
experts, who pointed out that ISA2 highlights the topic of interoperability and helps build 
a  community  of  professionals  in  this  area,  thus  addressing  the  needs  and  problems 
identified in the intervention logic. 

Figure 4: Extent to which achieving ISA2 objectives contributes to addressing the 
needs and problems originally addressed by the programme (average score of 
answers, number of respondents) 

Need 1: The need for cooperation among public
administrations with the aim to enable more
efficient and secure public services

Need 2: The need for exchanging information
among public administrations to fulfil legal
requirements or political commitments

Need 3: The need for sharing and reusing
information among public administrations to
increase administrative efficiency and cut red tape
for citizens and businesses

Problem: Administrative e-barriers are leading to
the fragmentation of the Internal Market

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

Action owners

Solution users

0

1

2

3

4

5

Programme governance

Stakeholders responsible for linked EU initiatives

Standardisation organisations

Wider public

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO). 
Source: CEPS final study 

23 

 
 
When it comes to the new needs and problems identified by consulted stakeholders, on 
average,  respondents  believe  that  the  programme,  through  its  general  objective,  could 
address them at least to some extent (Figure 5). However, three out of the six respondents 
responsible  for  linked  EU  initiatives  who  indicated  additional  needs  and  problems 
consider  that  the  general  objective  of  ISA2  cannot  address  the  additional  needs  and 
problems, or can only address them to a limited extent, as some of the issues mentioned 
fall  outside  the  programme's  scope  (for  instance,  Member  States'  different  political 
priorities). 

Figure 5: Extent to which achieving ISA2 objectives contributes to addressing 
additional (current) needs and problems identified by consulted stakeholders 
(number of respondents by stakeholder category) 

9

8

7

6

5

4

3

2

1

0

Action owners

Programme governance

Solution users

Stakeholders responsible for
linked EU initiatives

Standardisation organisations

Not at all

To a
limited
extent

To some
extent

To a high
extent

To the
fullest
extent

Do not
know/No
opinion

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent; (DK/NO) do not know/no opinion. 
Source: CEPS final study 

Answer to the EQ on 'relevance' 

EQ1: To what extent are the objectives of the ISA² programme still pertinent in relation 
to the evolving needs and problems at both national and EU levels? 

The original needs and problems that the programme intended to address are still 
fully  relevant  and  the  objectives  of  ISA2  are  pertinent  to  addressing  them.  In 
addition,  consulted  stakeholders  have  identified  a  number  of  new  needs  and  problems 
related  to  the  interoperability  of  digital  public  services.  Some  of  these  fall  outside  the 
programme's  scope  (such  as  resource  constraints  or  different  political  priorities  at 
national level). Others are already being addressed by existing ISA2 actions (for example, 
the  'Innovative  Public  Services'  action  provides  support  for  identifying  the  innovation 
potential  and  framing  conditions  of  emerging  disruptive  technologies).  Finally,  ISA² 
could  (better)  address  some  of  these  additional  needs  and  problems  by  considering  the 
below measures for future actions and future editions of the programme: 

24 

 
 
  Increasing awareness of interoperability beyond national administrations by 
fostering  more  collaboration  and  exchanges  of  views  with  regional  and  local 
administrations.  The  recently  launched  Interoperability  Academy  action  could 
help  with  this  by  developing  training  materials  tailored  to  the  needs  of  regional 
and local governments. 

  Performing  studies  to  assess  the  costs  and  benefits  of  implementing  ISA2 
solutions.  The  findings  of  such  studies  could  feed  into  promotional  activities, 
targeting the sub-national layer of public administrations as well. 

  Improving  the  sharing  of  best  practices  among  public  administrations, 
academia  and  interested  professionals  thus  raising  awareness  of  interoperability 
and increasing the take-up of ISA2 solutions across Member States. 

  Relying more on potential influencers (like researchers, teachers or committed 
interoperability.  This  could 

public  servants)  when  communicating  about 
eventually create a community of interoperability ambassadors. 

  Building  advisory  capabilities  around  interoperability.  These  would  help 
interested  stakeholders  to  pick  the  appropriate  interoperability  solutions  that 
match  their  needs,  and  would  at  the  same  time  provide  support  services  and 
technical assistance for effective implementation. 

  Assessing  the  rationale  and  impacts  of  a  possible  binding  interoperability 
instrument. Such assessment should rely on the findings of the final evaluation 
of  the  ISA2  programme  and  on  the  evidence  that  will  be  gathered  when  the 
implementation of the EIF is evaluated in 2021. 

5.2.  Effectiveness 

Evaluation framework for 'effectiveness' 

Effectiveness focuses on the extent to which the programme has achieved its objectives 
and generated the expected results. Analysing the effectiveness of ISA2 requires answers 
to two EQs: 

EQ2:  How far are the ISA²  programme’s results in the process  of achieving the 

programme’s objectives? 

EQ3:  Are there aspects that are more or less effective than others are, and if so, 

what lessons can be drawn from this? 

To answer these questions, the evaluation applied the following judgment criteria (JCs): 

JC1:  Alignment  between  the  programme's  actual  results,  objectives  and  expected 

results; 

JC2: 

Impact of external factors on the programme's performance; 

JC3:  Awareness of the programme; 

JC4:  Performance indicators; and 

25 

 
JC5:  Alignment with principles specified in Article 4(b) of the ISA2 Decision. 

The  analysis  relies  on  data  collected  via  consultation  activities,  a  desk  review  and  an 
expert assessment. 

Analysis 

JC1: Alignment between the actual results, objectives and expected results of ISA2 

The focus of this judgment criterion falls on the solutions developed by ISA2, as they 
represent the most tangible results achieved so far. Figure 6 shows the extent to which 
the programme's actual results are aligned with its objectives, according to the consulted 
stakeholders.  45 %  of  respondents  (49  out  of  109)  agree  that  ISA2  solutions  have 
contributed  to  achieving  the  programme's  general  objective  to  a  high  extent  or  to  the 
fullest  extent.  In  contrast,  only  15 %  of  respondents  (16  out  of  109)  believe  that  ISA2 
solutions  have  made  either  no  contribution  or  a  limited  contribution.  The  positive 
feedback is higher when it comes to the programme's specific objectives. For instance: 

  More  than  56 %  of  respondents  (59  out  of  106)  confirm  that  the  solutions 
developed have contributed to developing, maintaining and promoting a holistic 
approach to interoperability in the Union (specific objective 1) to a high extent or 
to the fullest extent; 

  More  than  56 %  of  respondents  (60  out  of  108)  indicate  that  the  solutions 
developed have contributed to  facilitating the reuse of  interoperability solutions 
by European public administrations  (specific objective 5) to  a high extent  or to 
the fullest extent.  

the  national,  regional  and 

However,  respondents  to  the  consultation  activities  express  a  lower  level  of  agreement 
regarding  the  contribution  to  the  development  of  a  more  effective,  simplified  and  user-
friendly  e-administration  at 
levels  of  public 
administration (specific objective 3). Only 34 % of respondents (36 out of 106) consider 
that  ISA2  solutions  have  contributed  to  this  objective  to  a  high  extent  or  to  the  fullest 
extent, while 22 % (23 out of 106) believe that there has been either no contribution or 
only a limited contribution.  This  is  aligned with  the assessment of the EU added value 
criterion (see Figure 17 in section 5.5), which shows that Member States’ initiatives tend 
to perform relatively better when it comes to achieving specific objective 3. 

local 

26 

 
Figure 6: Extent to which ISA2 solutions contributed to achieving the programme’s 
objectives (average score of answers, number of respondents) 

General objective: To promote the ICT-based
modernisation of the public sector in Europe and to
facilitate addressing the needs of businesses and
citizens via improved interoperability of public
administrations, contributing to the DSM and to
growth

Specific objective 1: To develop, maintain and
promote a holistic approach to interoperability in
the Union in order to eliminate fragmentation in the
interoperability landscape in the Union

Specific objective 2: To facilitate efficient and
effective electronic cross-border or cross-sector
interaction between European public
administrations, and between European public
administrations and businesses and citizens

Specific objective 3: To contribute to the
development of a more effective, simplified and
user-friendly e-administration at the national,
regional and local levels of public administration

Specific objective 4: To identify, create and operate
interoperability solutions supporting the
implementation of Union policies and activities

Specific objective 5: To facilitate the reuse of
interoperability solutions by European public
administrations

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

Standardisation organisations Wider public

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

The desk review of secondary data also shows good progress towards the achievement of 
ISA2  objectives.  The  20  sampled  actions  have  already  produced  35  solutions,  and  a 
further  21  solutions  are  under  development.  Most  of  the  operational  solutions  are  used 
across the 28 Member States and across different EU institutions, thus contributing to a 
coherent approach to interoperability in the EU (Annex 6: Table 9, Table 10 and Table 
11). 

The Interinstitutional Register for Delegated Acts35 is a good example that illustrates how 
the various ISA2 solutions help achieve the programme’ objectives. It was set up under 

35  See: https://webgate.ec.europa.eu/regdel/#/home. 

27 

 
 
                                                 
the ‘RegDel’ action36 in order to respond to the joint commitment of the Commission, the 
Council  and  the  European  Parliament  to  facilitate  access  to  delegated  acts  through  a 
dedicated register. By doing so, the register also ‘supports the implementation of Union 
policies  and  activities’  as  envisaged  by  specific  objective  4  of  the  ISA2  programme. 
Launched  in  December  2017,  the  solution  has  amassed  2202  active  subscriptions  and 
over 144 000 views (up to 1 February 2019)37. 

Another  example  is  the  European  Interoperability  Reference  Architecture  (EIRA©)38, 
which helps classify and organise interoperability building blocks used in the delivery of 
digital  public  services.  EIRA  facilitates  the  reuse  of  interoperability  solutions  by 
European public  administrations (specific 5 of  ISA2) and is  appreciated by  its users,  as 
shown in Box 2. 

Box 2: EIRA user testimonial 

Source: ISA2 communication team 
Other  examples  include  the  ‘Joinup’  solution39  and  events  such  as  the  ‘Sharing  and 
Reuse Conference’40 that facilitate access to and the reuse of interoperability solutions, as 
well as the exchange of best practices. 

36  See: https://ec.europa.eu/isa2/actions/improving-transparency-delegated-acts_en. 
37  The statistics are based on input received from the action owner. 
38  See: https://ec.europa.eu/isa2/solutions/eira_en. 
39  See: https://ec.europa.eu/isa2/solutions/joinup_en. 
40  See for instance: https://joinup.ec.europa.eu/event/sharing-reuse-conference-2017. 

28 

 
 
                                                 
Turning  to  the  operational  objectives  (Figure  7),  the  highest  achievements  of  ISA2 
solutions pertain to the following ones: 

  Operational objective 7 – 68 % of respondents (42 out of 62) confirmed that the 
existing solutions have contributed to supporting and promoting the maintenance, 
updating,  promotion  and  monitoring  of  the  implementation  of  the  EIS,  the  EIF 
and the EIRA to a high extent or to the fullest extent, and only 1 % (1 out of 62) 
believe that there was a limited contribution. One good example is the European 
Interoperability Architecture (EIA), which has developed a conceptual reference 
architecture  that  is  already  deployed  in  eight  Member  States.  Moreover,  the 
European  Interoperability  Reference  Architecture  (EIRA)  solution  that  is 
available  on  the  ‘Joinup’  platform  has  been  downloaded  1 364  times  so  far 
(Annex  6:  Table  12).  In  addition,  the  National  Interoperability  Framework 
Observatory  contributes 
interoperability 
developments in Member States and national alignment with the EIF. 

this  objective  by  assessing 

to 

  Operational objective 9 — according to 68 % of respondents (50 out of 73), ISA2 
solutions  have  contributed  to  supporting  and  promoting  the  maintenance  and 
publication  of  a  platform  allowing  access  to  and  collaboration  with  regard  to 
best  practices,  functioning  as  a  means  of  raising  awareness  and  disseminating 
available solutions to a high extent or to the fullest extent, compared to 11 % (8 
out of 73) who believe that the contribution has been limited. The ‘Joinup’ action, 
by  maintaining  and  further  developing  the  collaborative  platform  with  the  same 
name,  is  directly  related  to  this  operational  objective.  The  platform  counts  over 
13 000  members,  namely  professionals  working  in  the  field  of  eGovernment.  It 
allows  members  to  work  together  on  over  109  thematic  working  spaces,  called 
‘collections’ (Annex 6: Table 12). 

Based on the feedback received, there are two areas for potential improvement: 

  Operational objective 4  — only 23 % of respondents (15 out of 64) believe that 
ISA2 solutions have contributed to supporting and promoting the identification of 
legislation gaps, at Union and national level, that hamper cross-border or cross-
sector interoperability between European public administrations to a high extent 
or to the fullest extent. 

  Operational objective 5 — only 25 % of respondents (17 out of 68) indicate that 
ISA2 solutions have contributed to supporting and promoting the development of 
mechanisms  that  measure  and  quantify  the  benefits  of  interoperability  solutions 
including  methodologies  for  assessing  cost-savings  to  a  high  extent  or  to  the 
fullest extent. 

The fact that the ‘Legal Interoperability’ action, responsible for the advancement of these 
two operational objectives, has until now mainly focused on the Commission’s internal 
policy-making process, may explain the lower awareness levels among the programme’s 
stakeholders. 

29 

 
Figure 7: Extent to which ISA2 solutions contributed to achieving the programme’s 
operational objectives (average score of answers, number of respondents) 

1: To support and promote the assessment,
improvement, operation and re-use of existing
cross-border or cross-sector interoperability
solutions and common frameworks

2: To support and promote the development,
establishment, bringing to maturity, operation
and re-use of new cross-border or cross-sector
interoperability solutions and common
frameworks

3: To support and promote the assessment of
the ICT implications of proposed or adopted
Union law

4: To support and promote the identification of
legislation gaps, at Union and national level, that
hamper cross-border or cross-sector
interoperability between European public
administrations

5: To support and promote the development of
mechanisms that measure and quantify the
benefits of interoperability solutions including
methodologies for assessing cost-savings

6: To support and promote the mapping and
analysis of the interoperability landscape in the
EU through the establishment, maintenance and
improvement of the EIRA and the EIC reuse
facilitators

7: To support and promote the maintenance,
updating, promotion and monitoring of the
implementation of the EIS, the EIF and the EIRA

8: To support and promote the assessment,
updating, promotion of existing/new common
specifications through the EU's standardisation
platforms and in cooperation with
standardisation organisations

9: To support and promote the maintenance and
publication of a platform allowing access to and
collaboration on best practices, functioning as a
means of raising awareness of existing solutions

10: To support and promote the bringing of new
interoperability services and tools to maturity,
and maintaining and operating existing
interoperability services and tools on an interim
basis
11: To support and promote the identification of
best practices, to develop guidelines to
coordinate interoperability initiatives and to
animate and support communities working in the
field
12: To develop a communication strategy aiming
to increase awareness with regard to ISA² and
its benefits, targeting businesses and citizens,
and employing user-friendly means on the ISA²
website

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

Action owners

Programme governance

Solution users

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

30 

 
 
As the programme has only just passed its midway point, it is reasonable to assume that 
its  actual  results  might  not  fully  match  the  expected  results,  since  actions  are 
ongoing and solutions will continue to be developed until the programme ends in 2020. 
Experts confirmed this assumption, pointing out that ISA2 solutions have contributed 
to,  for  instance,  ‘a  coherent  interoperability  landscape  in  the  Union’,  but  more  time  is 
needed for the expected results to be fully realised. The feedback received as part of the 
consultation activities provides further evidence of this (Figure 8). 

Figure 8: Extent to which ISA2 solutions achieved the programme's expected results 
(average score of answers, number of respondents) 

A coherent interoperability landscape in the Union
based on a holistic approach to interoperability

Advancement of Union policies and activities by
supporting their implementation

Contribution to the successful achievement of the
Digital Single Market

Efficient and effective electronic cross-border or
cross-sectoral interactions between European Public
Administrations as well as between European public
administrations and businesses/citizens

Improved efficiency and productivity of the European
public administrations, which can be a strong driver
of economic growth through its support for, and
governance of, the private sector

Increased speed, efficiency and quality in the
creation and delivery of cross-border and cross-
sector electronic public services to meet the needs of
businesses and citizens

More effective, simplified and user-friendly e-
administrations in European public administrations

Reduction of the cost and administrative burden of
cross-border interaction, removal of administrative
e-barriers and contribution to the swift
implementation of ICT systems supporting EU
legislation

Take-up and re-use of the ISA² programme’s results 
by European public administrations 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

Standardisation organisations

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

31 

 
 
in 

landscape 

the  EU  based  on  a  holistic  approach 

The  developed  solutions  are  mostly  aligned  with  the  expected  result  of  a  coherent 
to 
interoperability 
interoperability.  In  fact,  around  50 %  of  respondents  (37  out  of  77)  believe  that  ISA2 
solutions  have  achieved  this  expected  result  to  a  high  extent  or  to  the  fullest  extent,  in 
comparison to 8 % (6 out of 77) who argue that there has been limited achievement or no 
achievement in this area. The achievement of expected results is further assessed below, 
when the performance indicators are presented. 

Turning  to  areas  of  potential  improvement,  only  30 %  of  respondents  (23  out  of  76) 
consider  that  ISA2  solutions  led  to  improved  efficiency  and  productivity  of  the 
European  public  administrations. 
interviews  some 
respondents pointed out that it is difficult to reply to this specific question, as no specific 
studies have been carried out on the efficiency and productivity effects of ISA2 solutions, 
although such effects are expected. Future actions may aim to rectify this and ensure that 
impacts  generated  by  ISA2  solutions  for  public  administrations  are  identified  and 
quantified. 

Interestingly,  during 

the 

JC2: Impact of external factors on the programme's performance  

In  line  with  the  Evaluation  Framework,  assessing  the  programme's  effectiveness  also 
implies  considering  the  positive  and  negative  impacts  of  external  factors  on  the 
programme's overall performance. 

To this end, consulted stakeholders were asked to identify and rank the external factors 
that  on  the  one  hand  contribute  to  and  on  the  other  hand  jeopardise  the  programme's 
performance. There is a clear pattern in the answers of all groups consulted on this matter 
(programme  governance  stakeholders,  action  owners  and  solution  users).  The  calls  for 
common  standards  and  frameworks  from  public  administrations  are  considered 
particularly beneficial to the programme's performance (Figure 9), as they can guide the 
development of new solutions to address the issues raised by public administrations, i.e. 
the direct beneficiaries of the solutions. Most respondents, 81 % (63 out of 75), believe 
that  these  calls  contribute  to  the  performance  of  ISA2  to  a  high  extent  or  to  the  fullest 
extent.  This  also  shows that  interoperability  is  among  the  priorities  of  European  public 
administrations. 

32 

 
Figure 9: Extent to which the following external factors contribute to the 
performance of ISA2 (average score of answers, number of respondents) 

Calls for common standards and frameworks from
industry

Calls for common standards and frameworks from
public administrations

National initiatives related to the ICT modernisation
of the public sector

The need for European public administrations to
increase their efficiency due to budget constraints

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

On  the  other  hand,  institutional  complexity  is  the  external  factor  seen  as  having  the 
greatest  negative  impact  on  the  programme's  performance  (Figure  10).  78 %  of 
respondents  (64  out  of  82)  believe  that  this  factor  jeopardises  the  programme's 
performance to a high extent or to the fullest extent. Institutional complexity comes in the 
form  of  multiple  layers  of  governance  and  differences  between  national  public 
administrations in various Member States as well as between local public administrations 
within  Member  States,  all  of  which  can  cause  coordination  issues.  Technical  experts 
confirmed this result by identifying institutional and legal complexity as the factors that 
most jeopardise the programme’s performance. 

33 

 
 
Figure 10: Extent to which the following external factors jeopardise the 
performance of ISA2 (average score of answers, number of respondents) 

Institutional complexity

Legal complexity

Limited resources of European public
administrations

Technical challenges (e.g. updating existing legacy
applications)

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance

Solution users

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

JC3: Awareness of the programme 

When  analysing  the  feedback  collected  via  the  consultation  activities,  each  stakeholder 
group's level of awareness of interoperability issues and, more specifically, of the ISA2 
programme, must be taken into consideration. As illustrated in  Annex 2 (Figure 30 and 
Figure  31),  the  consulted  stakeholders  were  asked  to  assess  their  knowledge  of  digital 
public services and interoperability as well as their knowledge of ISA2. On average, the 
results show that most respondents have expert knowledge of interoperability and of 
ISA2. 71 % of respondents (91 out of 128) are familiar with digital public services and 
interoperability to a high extent or to the fullest extent, in comparison to 11 % (14 out of 
128)  who  either  are  not  familiar  with  the  field  or  are  familiar  with  it  only  to  a  limited 
extent.  In  addition,  63 %  (81  out  of  128)  of  respondents  are  familiar  with  the  ISA2 
programme  to  a  high  extent  or  to  the  fullest  extent,  compared  to  20 %  (25  out  of  128) 
who report only a low degree of familiarity or no familiarity with the programme. 

One factor that explains these positive awareness results is the ISA2 programme’s strong 
commitment  to  ‘enhance  information  and  increase  awareness  with  regard  to  the 
programme and its benefits’, as stated in Article 3 of the ISA2 Decision. In fact, raising 
public  administrations’  awareness  of 
three 
recommendations stemming from the final evaluation of the ISA programme41. The ISA2 
programme addressed this by adopting its communication strategy and engagement plan 

interoperability  was  one  of 

the 

41  Report  from  the  Commission  to  the  European  Parliament  and  the  Council,  Results  of  the  final 

evaluation of the ISA programme, Brussels, 1.9.2016, COM(2016) 550 final. 

34 

 
 
                                                 
in  201742  and  by  increasing  its  presence  at  conferences  and  other  events,  as  shown  in 
Box 3. 

The relatively low participation in the interim evaluation’s consultation activities and the 
need  to  better  liaise  with  regional  and  local  administrations,  identified  under  the 
Relevance  criterion  (see  section  5.1),  call  for  continued  and  even  improved  awareness 
raising efforts. 

JC4: Performance indicators 

In  line  with  the  Evaluation  Framework,  the  desk  research  collected  data  on  several 
performance  indicators  (Annex  6:  Table  12)  in  order  to  assess  the  extent  to  which  the 
programme’s  operational  objectives  had  been  achieved  and  to  complement  the  data 
gathered via consultation activities. 

When  talking  about  performance  indicators,  it  is  important  to  distinguish  between 
different categories of ISA2 actions: 

  actions continued from ISA and actions started during ISA2; and 

  actions  already  completed  at  the  time  of  the  interim  evaluation  and  actions  still 

ongoing. 

The different stages of an action imply different levels of achievement of objectives and 
expected results, as some solutions are operational while others are in the early proof-of-
concept stage. This should be kept in mind when looking at the performance indicators’ 
results (Box 4).  

Data  show  that  the  20  sampled  actions  have  developed  35  solutions  so  far,  out  of 
which 10 emerged under ISA2. As shown in Table 2, the solutions fall into one of the 
following  four  categories:  common  tools/services,  common  frameworks,  common 
specifications/standards, and studies. 

Table 2: Overview of solutions developed under the sampled actions, by type 

Type of solution 

Total number of solutions 

Common tools/services 
Common frameworks 
Common specifications/standards 
Studies 
TOTAL 
Source: CEPS final study 

20 
9 
3 
3 
35 

New solutions 
developed by ISA2 
4 
2 
1 
3 
10 

Aside  from  these  four  solution  types  associated  with  ISA2  actions,  there  is  also  a 
Interoperability  Awareness  — 
particular  case 

represented  by 

‘Raising 

the 

42  European  Commission,  DG  DIGIT  (2017),  ISA2  programme.  Communication  strategy  and 

stakeholders engagement plan: https://ec.europa.eu/isa2/sites/isa/files/comms_strategy.pdf. 

35 

 
                                                 
Communication  Activities’  action,  which  is  part  of  the  ‘Accompanying  measures’ 
package.  There  are  no  solutions  associated  with  this  action,  and  its  outputs  are  of  a 
specific nature, namely events organised or events to which ISA2 representatives actively 
contributed. Box 3 presents an overview of these communication activities. 

Box 3: Communication activities 

From  2016  to  the  end  of  2018,  ten  major  events  were  organised  as  part  of  the  ‘Raising 
Interoperability  Awareness  —  Communication  Activities’  action,  with  an  average  of  211 
participants per event. The events were organised in six different countries: Belgium, Bulgaria, 
Greece, Italy, Malta, and Portugal, thus ensuring that a wide pool of professionals was reached. 

In addition, ISA2 representatives participated in 60 events organised in 18 Member States and one 
EU candidate country (Montenegro) between 2016 and 2018. The number of participants in these 
events  ranged  from  125  to  6 000,  thus  ensuring  that  information  about  the  programme  was 
disseminated to numerous stakeholders. 

Tables 13 and 14 in Annex 6 present an overview of these events. The ISA2 website also lists past 
and upcoming events43. 

An  array  of  metrics  are  in use  to  assess  the  effectiveness  of  solutions,  for  example, 
the  number  of  users,  instances  of  use,  downloads  and  page  views.  When  these 
performance indicators are assessed in a more in-depth way, it is clear that they support 
the feedback received from  consulted stakeholders that  the programme  is  on  track  to 
achieving  the  expected  results.  However,  it  is  also  clear  that  more  time  is  needed  to 
fully  achieve  these  results.  This  pattern  is  also  confirmed  when  the  performance 
indicators  for  actions  continued  from  ISA  are  compared  with  those  for  actions  started 
during ISA2, as detailed in Box 4. 

Box 4: Comparing solutions and performance indicators of actions continued from 
ISA and actions started during ISA2 

The pattern that emerges when the performance indicators associated with actions continued from 
ISA and those started during ISA2 are assessed can be illustrated using the two sampled actions 
that are part of the semantic interoperability package: 

 

 

‘Public  Multilingual  Knowledge  Management  Infrastructure’  (PMKI)  for  the  Digital 
Single Market (initiated during ISA2); and 

‘SEMIC:  Promoting  semantic  interoperability  amongst  the  European  Union  Member 
States’ (a continuation from the action with the same name that was in place during ISA). 

The ‘PMKI’ action has so far provided two solutions: i) a common framework in the form of a 
core data model for multilingual taxonomies/terminologies; and ii) a common specification in the 
form  of  semantic  links  between  different  language  resources.  The  most  suitable  performance 
indicator associated with the core data model is measured by ‘instances of inclusion in academic 
journals  and  international  conference’.  Since  the  model  was  published  in  2017,  two  scientific 
papers  based  on  it  have  been  published  in  international  journals,  as  part  of  the  action.  In 

43  For further details see: https://ec.europa.eu/isa2/events_en. 

36 

 
 
                                                 
addition,  the  work  carried  out  on  the  model  has  been  presented  at  seven  international 
conferences. With regard to the semantic links, the main performance indicator is measured by 
instances  of  collaboration  with  European  public  administrations  with  a  view  to  establishing 
resources  and  EuroVoc44.  Four 
semantic 
collaborations have been set up so far. 

interoperability  between  national 

language 

In comparison, for the ‘SEMIC’ action, which has been in place longer than PMKI, the relevant 
performance  indicators  are  the  number  of  solution  users,  in  particular  the  number  of  public 
administrations using the solutions. For the four solutions that are part of SEMIC, the numbers 
of users are as follows: 

 

 

 

 

the  ‘Asset  Description  Metadata  Schema’  (ADMS)  is  used  in  eight  public 
administrations  in  the  Member  States  and  the  Commission,  and  by  one  company.  In 
addition, the solution has been used to describe approximately 4 700 interoperability 
solutions on Joinup; 

the  ‘Core  Vocabularies’  are  used in  11  public  administrations  in the  Member  States 
and in the Commission; 

the  ‘Data  Catalogue  Vocabulary  Application  Profile  for  Data  Portals  in  Europe’ 
(DCAT-AP)  is  used  by  29  public  administrations  in  the  Member  States  and  the 
Commission, and by associations and universities; and 

‘VocBench3’,  which  also  falls  under  SEMIC45  and  previous  versions  of  which  are 
used  in  14  public  administrations in  the  Member  States and  the  Commission,  and  by 
universities, institutes, and international organisations. 

An  analysis  of  the  solutions  developed  under  the  PMKI  and  SEMIC  actions  suggests  that  the 
performance indicators assessing actions continued from ISA tend to show a higher degree 
of maturity and take-up than those developed under ISA2. Therefore, it is clear that solutions 
need time to show fully-fledged results. This is a natural conclusion, and since there is still over 
one year and a half left until the ISA2 programme concludes, there is still time for solutions to be 
further developed.  

JC5: Alignment with principles specified in Article 4(b) of the ISA2 Decision 

The  evaluation  of  effectiveness  also  requires  an  assessment  of  the  programme’s 
compliance with the principles listed in Article 4(b) of the ISA2 Decision. The rolling 
work  programme  provides  details  on  how  the  ISA2  actions  should  consider  these 
principles.  The  overall  judgment  stemming  from  the  desk  analysis  of  the  rolling  work 
programme  is  positive  (Annex  6:  Table  15).  Supporting  examples  are  provided  by 
examining  the  contribution  of  ISA2  actions  to  the  implementation  of  the  EIF,  as  the 
principles in question also fall under the scope of the EIF (Annex 6: Table 23). 

44  EuroVoc  is  the  EU’s  multilingual  and  multidisciplinary  thesaurus,  containing  terms  in  23  EU 
languages.  For  further  details,  see:  https://publications.europa.eu/en/web/eu-vocabularies/th-dataset/-
/resource/dataset/eurovoc. 

45  See: https://ec.europa.eu/isa2/solutions/vocbench3_en. 

37 

 
 
                                                 
Answer to the EQs on ‘effectiveness’ 

EQ2:  How  far  are  the  ISA²  programme’s  results  in  the  process  of  achieving  the 
programme’s objectives? 

The  results  achieved  by  ISA2  so  far  are  aligned  with  the  programme’s  objectives. 
However, they do not yet fully match the expected results, as most of the actions are still 
ongoing  and  solutions  are  still  being  developed.  More  time  is  needed  in  order  to 
achieve  all  the  expected  results.  This  is  confirmed  by  the  assessment  of  key 
performance  indicators  and  a  comparison  between  actions  continued  from  ISA  and 
actions started under ISA2. 

External  factors  could  improve  but  also  jeopardise  the  way  in  which  the 
programme  achieves  its  objectives  and  delivers  its  results.  The  main  findings  hold 
true  across  the  different  stakeholder  groups  and  are  further  confirmed  by  expert 
assessments. The call for common standards and frameworks from public administrations 
is  an  external  factor  that  contributes  to  the  programme’s  performance.  In  contrast, 
institutional  complexity  could  harm  the  achievement  of  cross-border  and  cross-sectoral 
interoperability. 

EQ3:  Are  there  aspects  that  are  more  or  less  effective  than  others  are,  and  if  so,  what 
lessons can be drawn from this? 

While  in  general  the  programme  is  on  track  to  achieving  its  objectives,  it  performs 
relatively  less  well  when  it  comes  to  developing  a  more  effective,  simplified  and 
user-friendly  e-administration  at  the  national,  regional  and  local  levels  of  public 
administration. This is an area where Member States can in fact have a greater impact. 
National  initiatives  related  to  the  ICT  modernisation  of  the  public  sector  can  also 
improve  the  overall  performance  of  ISA2  by  increasing  the  capacity  for  the  take-up  of 
ISA2 solutions. 

Regarding  the  take-up  rate  of  solutions,  the  evaluation  found  that  the  actions  that 
continued from previous editions of the programme have produced solutions that are now 
more widely used than solutions resulting from newly established ISA2 actions. 

As  regards  the  specific  request  of  Article 13(5)  of  the  ISA2  Decision  to  assess  the 
programme’s  compliance  with  the  principles  set  out  in  Article 4  of  this  decision,  the 
evaluation  found  that  the  rolling  work  programme  process  effectively  ensures  that 
the  ISA2  actions  take  into  consideration  the  core  principles  listed  in  the  ISA2 
Decision. 

When it comes to communication and awareness-raising activities, the evaluation results 
suggest  that  ISA2 duly acted upon the recommendation of the final  ISA evaluation  and 
increased  general  awareness  of  the  programme.  Still,  some  aspects  of  ISA2  could 
benefit  from  more  promotion.  For  example,  as  legal  interoperability  is  specifically 
mentioned  among  the  operational  objectives,  the  solutions  developed  for  assessing  the 
ICT  and  interoperability  impacts  of  legislation  could  be  more  prominently  featured  in 

38 

 
order to improve awareness among stakeholders. Moreover, ISA2 could better stress the 
benefits  of  implementing  certain  interoperability  solutions.  To  this  end,  it  could  run 
studies  that  quantify  the  impact  of  interoperability  solutions  on  the  efficiency  and 
productivity of public administrations. Such studies would improve the conditions for 
assessing  the  programme’s  effectiveness  and  could  have  positive  spillover  effects  on 
dissemination  and  communication  activities:  the  monetary  benefits  deriving  from  ISA2 
solutions  would  be  better  emphasised,  thus  providing  a  better  basis  for  stakeholder 
support  of  future  EU  efforts  promoting  interoperability.  Finally,  as  noted  under  section 
5.1 Relevance, ISA2 communication activities should better target the sub-national layers 
of administrations. 

5.3.  Efficiency 

Evaluation framework for ‘efficiency’ 

Efficiency  relates  to  the  extent  to  which  the  programme’s  objectives  are  achieved  at  a 
minimum cost. This evaluation criterion includes two EQs and one sub-question: 

EQ4:  To what extent has the programme been cost-effective? 

EQ5:  Which  aspects  of  the  programme  are  the  most  efficient  or  inefficient, 

especially in terms of resources mobilised? 

EQ5.1:  How  is  the  programme  performing  relative  to  the  planned  work  and 

budget? 

To answer these questions, the evaluation focused on three judgment criteria, relying on 
data  collected  from  the  ISA2  dashboard,  the  monitoring  and  evaluation  reports  and 
consultation activities: 

JC1:  Efficiency  of  the  selection  process  of  the  actions  to  be  included  in  the  rolling 

work programme; 

JC2:  Cost-effectiveness  analysis  to  assess  the  ratio  between  allocated  funds  and 

actual results of the programme; and 

JC3:  Earned value management analysis. 

Analysis 

JC1: Efficiency of the action selection process 

The  ISA2  programme  is  implemented  in  four  phases:  i)  submission;  ii)  evaluation;  iii) 
adoption; and iv) implementation (further details are provided in Chapter 3). The process 
to  select  ISA2  actions  includes  the  first  three  phases,  i.e.  submission,  evaluation,  and 
adoption.  Note  that  this  section  does  not  assess  the  selection  of  contractors  who  will 
implement the actions, as this process follows the general European public procurement 
rules. 

Several  elements  of  the  ‘submission’  phase  need  to  be  explained  here.  First,  it  is 
important  to  note  that,  in  order  to  prepare  and  submit  a  proposal  to  be  included  in  the 

39 

 
ISA2 rolling work programme, the applicant must: i) find out about the call for proposals; 
ii) study the documents of the call for proposals and understand the rules and procedures; 
iii)  prepare  a  concise  description  of  the  proposed  action  in  compliance  with  the  ‘work 
programme entry template’; iv) collect the required internal authorisations to submit the 
proposal;  and  v)  submit  the  proposal  by  e-mail.  Second,  it  is  important  to  distinguish 
between  two  types  of  proposal,  namely  proposals  for  new  actions  and  proposals  for 
continuing actions that have already started. This distinction is relevant when it comes to 
assessing the overall workload required for submitting a proposal. 

The  efficiency  of  the  action  selection  process  (which  includes  the  submission, 
evaluation and adoption phases) is assessed based on feedback received from programme 
governance representatives and action owners, as they are the stakeholder groups directly 
involved  in  the  process.  Most  respondents  consider  the  process  fit-for-purpose  at 
least to some extent: 16 out of 19 consulted programme governance stakeholders and 21 
out  of  23  consulted  action  owners  (Figure  11).  However,  13  out  of  the  42  consulted 
stakeholders  indicated  that  some  measures  could  still be  taken  to  further  streamline 
the process. The measures mentioned by several stakeholders include: 

  Simplifying the rolling work programme, which can be done by: 

o  Making the template more flexible to account for action specificities. This 
would ensure better input of information, which would make the process 
of drafting the proposals more efficient, and would also make it easier for 
anyone  seeking  specific  information  about  ISA2  actions  to  read  and 
understand the work programme. 

o  Extending the duration of a rolling work  programme to  cover more than 
one year, thus allowing for multiannual actions. In principle, actions to be 
run on a multiannual basis can be proposed, but the budget is allocated on 
an  annual  basis.  Therefore,  action  owners  must  submit  an  updated 
proposal  every  year  as  part  of  the  rolling  work  programme  in  order  to 
access funding. 

o  Setting up an electronic data input system rather than full text editing. 

  Simplifying the overall process by launching thematic calls for actions, thus 

ensuring that the selection is more objective-driven. 

40 

 
Figure 11: Extent to which the action selection process is fit-for-purpose (number of 
respondents by stakeholder category) 

12

10

8

6

4

2

0

Action owners

Programme governance

Not at all

To a
limited
extent

To some
extent

To a high
extent

To the
fullest
extent

Do not
know/No
opinion

Source: CEPS final study 

As  regards  the  submission  phase  of  the  selection  process,  26  out  of  the  42  consulted 
action  owners  and  programme  governance  representatives  indicated  that  they  had 
prepared  and  submitted  proposals.  Out  of  these  26  respondents,  21  also  specified  the 
amount of time they spent on preparing the proposal. The answers provided vary between 
1  to  30  person-days,  with  the  average  being  9  person-days.  The  large  variation  can  be 
primarily explained by the differences in the efforts required to submit a proposal for  a 
new  action  versus  the  amount  of  effort  required  to  renew  a  proposal  for  an  existing 
action. 

As  regards  the  time  needed  for  submission,  Figure  12  shows  a  clear  concentration  of 
answers:  14  of  the  respondents  indicated  that  they  finalised  their  proposal  in  1  to  6 
person-days.  However,  four  respondents  indicated  that  20  or  more  person-days  were 
needed. This distribution of answers and the qualitative inputs provided by some of the 
respondents during the consultation activities makes it possible to differentiate between 
preparing  a  renewed  proposal  and  a  brand-new  proposal. Renewing  a  proposal  for 
an action that has already been included in a previous rolling work programme tends to 
require between 1 and 6 person-days at most, while preparing a new proposal takes 
significantly  longer,  between  10  and  30  person-days.  In  most  cases,  stakeholders 
prepare the proposals themselves, without relying on external contractors. 

Additional  time  is  also  required  to  merge  similar  proposals,  upon  request  by  the 
Commission team in charge of shortlisting the proposals to be funded. 

41 

 
 
Figure 12: Distribution of the number of days needed to prepare and submit a 
proposal for an ISA2 action (number of respondents) 

y
c
n
e
u
q
e
r
F

6

5

4

3

2

1

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

Number of days 

Note: The histogram was constructed using a bin size of 2 days, which grouped the answers of respondents 
in intervals of 2 days. This bin size was deemed most appropriate after reviewing the answers provided. 
Source: CEPS final study 

The time spent to prepare and submit a proposal for an action to be included in the rolling 
work  programme,  whether  it  is  a  new  or  a  renewed  proposal,  can  be  used  to  measure 
regulatory  costs  by  applying  the  standard  cost  model46,  presented  in  the  Better 
Regulation  Toolbox47.  First,  person-days  are  converted  into  person-hours  by  assuming 
that each person-day corresponds to eight hours48. Then, the overall number of hours is 
multiplied by a standard tariff, i.e. the 2017 hourly labour cost for the service sector at 
the Member State level (see Annex 6: Table 22)49. 

Two respondents indicated that they relied on external service providers to prepare the 
proposal.  In  this  case,  regulatory  costs  would  also  include  out-of-pocket  expenses 
incurred for such services. Table 3 presents the average regulatory costs both taking into 

46  SCM Network (2005), ‘The International SCM Manual; Measuring and Reducing Administrative 

Burdens for Businesses’, available at 
http://www.administrativeburdens.com/filesystem/2005/11/international_scm_manual_final_178.doc. 
47  European  Commission,  Better  Regulation  ‘Toolbox’,  7 July  2017,  Tool  #60  available  at 

https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_2.pdf. 

48  Eurofound (2016), ‘Working time developments in the 21st century: Work duration and its regulation 

in the EU’. 

49  The country of activity was recorded for each respondent who provided information on the number of 
days spent preparing the proposal. The hourly labour costs  were identified  for each country. For the 
Commission, the hourly labour costs in either Belgium or Luxembourg  were used, depending on the 
Commission service involved. The regulatory costs were computed  for each case recorded during the 
consultation  activities,  taking  into  consideration  the  country  of  activity.  At  the  end,  the  average 
regulatory cost was calculated. Note that labour costs include wage and non-wage costs less subsidies. 
They do not include vocational training costs or other expenditure such as recruitment costs, spending 
on  working  clothes,  etc.  For 
see:  https://ec.europa.eu/eurostat/statistics-
explained/index.php/Hourly_labour_costs#Source_data_for_tables_and_figures_on_this_page_.28MS. 

further  details 

42 

 
 
                                                 
 
account  the  fees  related  to  external  service  providers  and  without  accounting  for  these 
additional fees. The average respondent faced regulatory costs of approximately €1 077 
for renewing a proposal and €5 651 for preparing a new proposal, without accounting for 
fees incurred when relying on external service providers. Considering the two instances 
when  external  service  providers  were  used  in  the  preparation  of  proposals,  the  average 
regulatory cost  for a renewed proposal  amounted to  €1 138 and for a new proposal  the 
average regulatory cost was €6 590. At the same time, the average funding allocated to 
an ISA2 action between 2016 and 2018 amounted to €1 217,56650. The costs incurred to 
prepare a proposal are thus very small, ranging between 0.09 % and 0.5 % of the 
potential funding that could be allocated to the proposal once accepted. 

Table 3: Average regulatory costs incurred by respondents who prepared and 
submitted proposals for ISA2 actions 

Average regulatory 
cost, including 
external services (€) 

Average regulatory 
cost without external 
services (€) 

Average funding 
allocated per action (€) 

1 138 

Renewed 
proposal 
New proposal 
Note: The calculations are based on a sample of 21 respondents who specified the time spent on preparing 
proposals. 
Source: CEPS final study 

1 217,566 

6 590 

1 077 

5 651 

One respondent mentioned that the reason they relied on external service providers in this 
process was the need to tailor the proposal to the requirements of the rolling work 
programme  template.  As  such,  one  potential  cost-reduction  method  could  be  the 
simplification  of  the  rolling  work  programme  template,  thus  further  corroborating  the 
measures proposed to streamline the action selection process. 

JC2: Cost-effectiveness analysis 

Based  on  the  performance  indicators  discussed  in  section  5.2  on  effectiveness  and  on 
data  related  to  the  actual  costs  of  actions,  a  cost-effectiveness  analysis51  could  be 
performed.  However,  it  is  important  to  note  the  limitations  of  this  method.  First,  the 
heterogeneity of performance indicators makes it difficult to draw conclusions about the 
programme’s  overall  cost-effectiveness  (Box  5  shows  the  variety  of  ISA2  performance 
indicators).  Consequently,  computing  an  aggregate  value  that  is  representative  of  the 
programme  or  action  packages  is  quite  difficult.  Second,  such  data  are  only  partially 
available, as some ISA2 solutions are still under development. In addition, as concluded 
in section 5.2, it is expected that the values of the performance indicators will improve 
over time, as  solutions  tend to  require longer periods to  fully show their results.  Third, 

50  The average was calculated based on the data available on the ISA2 dashboard for all ISA2 actions. 
51  According to Tool #57 of the Better Regulation Toolbox, cost-effectiveness analysis (CEA) entails the 
quantification (but not monetisation) of the benefits that one euro of costs imposed on society would 
generate.  See  page  452  at  https://ec.europa.eu/info/sites/info/files/file_import/better-regulation-
toolbox-57_en_0.pdf. 

43 

 
 
                                                 
data on effectiveness are not yet available for a limited number of solutions developed so 
far. 

Against  this  background,  the  cost-effectiveness  analysis  can  be  exemplified  by  using 
some of the ISA2 packages that rely on similar performance indicators. Two packages fit 
this profile: ‘Key and generic interoperability enablers’ and ‘eProcurement / eInvoicing 
—  supporting  instruments’.  The  sampled  actions  included  in  both  packages  use  the 
number  of  public  administrations  (at  EU,  national  and  regional  levels)  that  use  the 
solutions  generated  by  the  actions  as  a  performance  indicator.  Table  4  presents  the 
estimated  average  cost  per  solution  user  based  on  the  available  data  on  the  number  of 
solution  users  (public  administrations)  and  the  actual  costs  of  the  action  between  2016 
and  Q3 2018.  For  the  ‘Key  and  generic  interoperability  enablers’  package,  the  average 
actual  cost52  per  public  administration  using  the  solutions  that  are  part  of  this  package 
amounts  to  €19 420.  For  the  ‘eProcurement  /  eInvoicing  —  supporting  instruments’ 
package, the average actual cost  amounts to  €25 870. Note that these  average  costs are 
calculated  per  public  administration  using  the  solution  rather  than  per  single  end-user. 
Each administration serves millions of end-users (citizens/businesses); hence,  costs per 
end-user are estimated to be low. 

Table 4: Costs per user of ISA2 solutions 

Package 

Actual costs of 
sampled actions in 
the package (€) 

Number of public 
administrations 
using the solutions of 
the package 

Average actual cost 
(€) 

212 

4 117,069.81 

Key and 
generic 
interoperability 
enablers 
eProcurement 
/ eInvoicing 
— supporting 
instruments 
Note: The  calculations are based on the sampled actions of the indicated ISA2 packages over the period 
2016 and Q3 2018. A full overview of the number of users and other performance indicators is presented in 
Annex 6: Table 12. The overview of costs per package is shown in Figure 13. 
Source: CEPS final study 

3 544,135.98 

25 870 

19 420 

137 

The appraisal of the programme’s cost-effectiveness  could  be made more consistent by 
ensuring  that  monitoring  and  evaluation  reports  quantify  some  performance  indicators 
common  to  all  actions.  Examples  of  such  indicators  include  the  number  of  EU  public 
administrations  using  a  solution,  the  number  of  public  administrations  participating  in 
pilot  projects,  a  consistent  approach  to  reporting  the  number  of  downloads  from  the 
‘Joinup’ platform for solutions that can be downloaded, and/or the number of single users 
per solution. 

52  Actual 

refer 
https://ec.europa.eu/isa2/dashboard/faq-page. 

costs 

the 

to 

executed 

budget. 

For  more 

information, 

see: 

44 

 
                                                 
Box 5: Summary of the main costs and benefits identified 

Costs. The main costs associated with the intervention fall into two categories: regulatory costs 
of  the  selection  process  for  ISA2  funded  actions  (see  Table  3)  and  the  costs  per  user  of  ISA2 
solutions (see Table 4). 

Benefits.  In  turn,  the  benefits  generated  are  expressed  in  the  form  of  performance  indicators 
attached  to  each  solution  developed  as  part  of  ISA2  actions,  which  can  be  measured  using  a 
variety of metrics. The benefits generated by the programme can thus be summarised as follows 
(the list is not exhaustive): 

  280 Commission proposals have been screened for ICT impacts since 2015; 

  13 440 professionals working in the field of eGovernment are registered on ‘Joinup’ in 

order to access interoperability solutions and collaborate with one another; 

  over  8.8  million  documents  have  been  exchanged  so  far  between  the  Commission,  the 

Council, the Member States, and companies using the ‘e-TrustEx’ platform; 

 

 

the ‘European Single Procurement Document’ (ESPD) service53 has been visited around 
4 million times (until April 2019); and 

the ‘Core Public Service Vocabulary-Application Profile’ (CPSV-AP) solution has been 
downloaded from ‘Joinup’ over 2 500 times. 

Additional  performance  indicators,  collected  as  part  of  the  desk  review  of  secondary  data,  are 
included in Annex 6: Table 12. 

JC3: Earned value management (EVM) analysis 

EQ5  and  EQ5.1  focus  on  progress  made  in  implementing  the  programme  based  on 
the  EVM  analysis.  This  analysis  helps  determine  progress  of  work  against  a  given 
baseline,  so  that  costs,  time,  and  scope  of  a  certain  activity  are  constantly  tracked.  To 
support  this,  the  ISA2  dashboard  provides  quarterly  data  on  the  earned  value,  planned 
value, and actual costs of each monitored action54. As regards the sample of 20 actions 
grouped in the respective packages, Figure 13 presents the data at the end of Q3 2018 for 
the earned and planned value, as well as for the actual costs recorded. The earned value is 
close  to  equalling  the  planned  value,  which  shows  that  the  work  is  progressing  as 
expected. Even where  a  bigger  gap exists (e.g. in the ‘Key  and  generic interoperability 
enablers’ package), the indicators are expected to converge at the end of the year as the 
ISA2 budget year closes. 

53  See: https://ec.europa.eu/tools/espd. 
54  The earned value represents a quantification of the work accomplished to date. The planned value is a 
quantification of how far along the work on the programme is supposed to be in terms of the schedule 
and cost estimate at a given point in the programme. The actual costs represent the executed budget. 

45 

 
 
                                                 
Figure 13: Earned value, actual costs and planned value sampled actions, by 
package (Q3 2018)55 

1. Key and generic interoperability enablers

2. Semantic Interoperability

3. Access to data/data sharing/open data

4. Geospatial Solutions

5. e-Procurement/e-invoicing-Supporting
instruments

6. Decision making and legislation-Supporting
instruments

7.  EU  Policies-Supporting instruments

8.  Supporting instruments for public
administrations

9. Accompanying measures

Actual Cost (Euro)

Planned Value (Euro)

Earned Value (Euro)

Source: CEPS final study 

The data available also make it possible to perform a historical analysis of work progress 
for  each  package  of  actions.  Progress  can  also  be  shown  in  the  form  of  a  schedule 
performance index (SPI), which is the ratio of earned value to planned value (Figure 14). 
An SPI value above one indicates that the item analysed is ahead of schedule, while an 
SPI  value  below  one  suggests  that  the  item  analysed  is  behind  schedule.  Since  the 
beginning  of  the  programme  and  up  until  Q3  of  2018,  the  SPIs  of  the  sampled  actions 
grouped  by  packages  have  shown  fluctuations,  but  now  tend  to  converge  on  one, 
although values below one are still reported for almost all packages. These data show that 
the  ‘Semantic  interoperability’  and  ‘Supporting  instruments  for  public  administrations’ 
packages are on track, while the others are close to achieving the planned level of work. 
Given  that  some  actions  started  later  on  in  the  programme,  as  is  the  case  of  the 
‘Geospatial solutions’ package, data for the EVM analysis are not always available from 
Q2 of 2016, but only from a later point in time. 

SPI values above two were reported for the ‘Key  and generic interoperability enablers’ 
package in Q2 and Q3 of 2016. This indicates that the work on the package was ahead of 
schedule  in  that  period.  While  this  result  might  seem  surprising  at  first,  it  can  be 
explained by the fact that some of the actions involved started under ISA and continued 

55  The  PV  and  AC  are  equal  in  all  cases.  As  explained  on  the  ISA2  Dashboard,  most  of  the  specific 
contracts under the ISA2 actions are on fixed price, therefore it is not relevant to keep track of the AC 
before the end of a Work Programme year. Taking this into account, the AC is then considered equal 
to  the  PV.  For  more  details  please  see:  ISA2  Dashboard  —  Frequently  Asked  Questions, 
https://ec.europa.eu/isa2/dashboard/faq-page. 

46 

 
 
                                                 
under ISA2. For these actions, processes had already been established during the previous 
programme,  so  work  under  ISA2  could  start  much  faster  than  planned,  resulting  in  a 
higher  SPI  value  at  the  beginning  of  the  process.  In  fact,  both  of  the  actions  sampled 
from  the  ‘Key  and  generic  interoperability  enablers’  package  (‘Trusted  Exchange 
Platform  e-TrustEx’  and  ‘Catalogue  of  Services’)  had  been  launched  under  the  ISA 
programme. 

Figure 14: Schedule performance indices of the sampled actions grouped by 
packages 

2,4

2,2

2

1,8

1,6

1,4

1,2

1

0,8

0,6

0,4

1,8

1,6

1,4

1,2

1

0,8

0,6

1. Key and generic interoperability enablers

2. Semantic Interoperability

3. Access to data/data sharing/open data

4. Geospatial Solutions

5. e-Procurement/e-invoicing-Supporting instruments

6. Decision making and legislation-Supporting instruments

7.  EU  Policies-Supporting instruments

8.  Supporting instruments for public administrations

9. Accompanying measures

Source: CEPS final study 

47 

 
 
 
Answer to the ‘efficiency’ EQs 

EQ.4: To what extent has the programme been cost-effective? 

The heterogeneity of performance indicators makes it difficult to draw conclusions about 
the programme’s overall cost-effectiveness. For those packages where it was possible to 
apply  cost-effectiveness  techniques,  costs  per  end-user  (e.g.  business,  citizens,  etc.) 
have been estimated as low. In this respect, monitoring and evaluation reports could 
converge  towards  some  common  metrics  allowing  for  a  more  consistent  appraisal  of 
cost-effectiveness.  For  example,  the  agreed  performance  indicators  could  comprise  the 
number  of  EU  public  administrations  using  a  given  solution,  the  number of  downloads 
from the ‘Joinup’ platform for downloadable solutions, and/or the number of single users 
per solution. 

EQ5: Which aspects of the programme are the most efficient or inefficient, especially in 
terms of resources mobilised? 
  EQ5.1: How is the programme performing relative to the planned work and budget? 

The process of selecting actions to be funded by ISA2 is considered relatively efficient 
and  fit  for  purpose.  On  average,  six  person-days  at  most  are  required  to  prepare  and 
submit  a  proposal  for  actions  that  had  already  been  included  in  previous  rolling  work 
programmes. On the other hand, between 10 and 30 person-days are required to prepare a 
proposal for a new action. The  costs incurred in preparing a proposal are relatively 
small, ranging between 0.09 % and 0.5 % of the potential funds that could be allocated to 
the  proposal  once  accepted.  Nonetheless,  this  process  could  be  further  streamlined  by 
simplifying  the  rolling  work  programme  and  launching  thematic  calls  for  actions, 
thus ensuring that the selection process is driven by objectives. 

When looking  at  progress made in the programme’s implementation,  it  is clear that  all 
packages  are  either  on  track  or  close  to  achieving  the  planned  level  of  work.  As 
some actions started during ISA and continued under ISA2, some processes had already 
been established during the previous programme. Therefore, work on these actions under 
ISA2 could start faster than planned. 

5.4.  Coherence 

Evaluation framework for ‘coherence’ 

Article 13(4)  of  the  ISA2  Decision  requires  that  the  coherence  of  ISA2  actions  is 
assessed;  this  is  referred  to  as  the  programme’s  ‘internal  coherence’.  In  addition, 
Article 13(6) pays specific attention to the programme’s ‘external coherence’ in order to 
‘identify potential overlaps, examine coherence  with  areas for improvement,  and verify 
synergies  with  other  Union  initiatives,  in  particular  with  the  CEF’.  The  coherence 
criterion translates into two EQs: 

EQ6:  To what extent do the ISA2 actions form part of a ‘holistic’ approach within 

the framework of the programme? (Internal coherence) 

48 

 
EQ7:  To  what  extent  is  the  ISA2  programme  coherent  with  other  EU 
interventions,  which  have  similar  objectives  and  with  global  initiatives  in 
the same field? (External coherence) 

To assess both the internal and the external coherence of the programme, findings from 
desk research on links between ISA2 actions and the relationship between ISA2 and other 
relevant  EU  programmes/policies/initiatives  were  used  in  addition  to  the  information 
provided by stakeholders. In this context, the following judgment criteria are considered: 

JC1: 

Internal coherence: 

  Degree of coherence among actions funded by the ISA² programme; 

  Level  of  reuse  of  results  of  a  funded  action  by  another  action  within  the  ISA² 

programme; 

JC2:  External coherence: 

  Degree  of  coherence  between  the  programme  and  other  EU  supported 

programmes; 

  Level of reuse of results delivered by ISA2 actions by other EU programmes; 

  Degree of coherence between the programme and other EU policies; and 

  Degree of coherence between the programme and global initiatives in the field. 

Analysis 

JC1: Internal coherence 

Given that the ISA2 programme currently manages 53 actions56, it is important to ensure 
that they work together to create benefits and that work is not duplicated. The stakeholder 
groups consulted on this matter (programme governance, action owners, solution users) 
tend to agree that ISA2 actions generate synergies (Figure 15). At the same time, most 
stakeholders indicate only limited overlaps between ISA2 actions. Stakeholder feedback 
on synergies and overlaps, as two sides of the same coin, paints a positive picture of the 
internal coherence of the ISA2 programme. 

Only 4 % of respondents (2 out of 53) identify a high degree of overlaps between ISA2 
actions,  while  the  vast  majority,  77 %  (41  out  of  53),  consider  overlaps  as  either  not 
existing  or  being  limited.  In  addition,  respondents  point  to  the  existence  of  synergies: 
41 % of respondents (27 out of 66) identify synergies as existing to a high extent or to the 
fullest extent and 17 % (11 out of 66) identify either no synergies or limited synergies. It 
is  relevant  to  note  that  a  significant  number  of  solution  users  indicated  that  they  were 
unaware  of  either  synergies  or  overlaps  between  ISA2  actions.  This  feedback  can  be 
explained by stakeholders’ various levels of knowledge about ISA2 packages. As not all 
stakeholders are familiar with all ISA2 packages, they might not be in a suitable position 
to identify potential synergies or overlaps. 

56  Situation as of 2018. 

49 

 
                                                 
Figure 15: Extent to which there are synergies / overlaps between ISA2 actions 
(average score of answers, number of respondents) 

Overlaps

Synergies

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance

Solution users

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

The review of documentary evidence confirms the results of the consultation activities, 
particularly regarding the existence of synergies between ISA2 actions. The indicator of 
the ‘network’ of relationships between the programme’s actions57, as shown on the ISA2 
dashboard, is particularly relevant here. The quantitative assessment of the data provided 
online for the sample of 20 actions under assessment shows (see Annex 6: Table 17) that 
there  are  multiple  instances  of  internal  reuse  of  ISA2  solutions,  with  actions  like 
‘SEMIC’ and ‘Joinup’ playing a particularly important role in the system. 

At the same time, some actions tend to be less central in the network. This is the case 
particularly  for  the  ‘EUSurvey’58  and  ‘European  Citizens’  Initiative  and  European 
Parliament Election’59 actions. The specificity of these actions makes it more likely that 
their solutions  are reused by other programmes  or initiatives, rather than by other  ISA2 
actions. 

Throughout the consultation activities, additional inputs were collected from respondents 
concerning synergies between actions and examples of instances when results delivered 
by an ISA2 action were used by another action (see Box 6). 

57  These  data  are  available  for  each  monitored  action  under  the  ‘coherence’  tab  of  the  ISA2  dashboard 

58 

page dedicated to the specific action. 
‘EUSurvey’ maintains a tool — with the same name — designed to facilitate the gathering of feedback 
and opinions quickly and reliably via online surveys. 

59  This action developed two solutions: the online collection software supporting the European Citizens’ 
Initiative and the European Parliament crypto tool. The first tool enables citizens to bring initiatives to 
the attention of the Commission; the second puts in place mechanisms to prevent double voting. 

50 

 
 
 
 
                                                 
Box 6: Examples of internal coherence between ISA2 actions 

  The ‘DCAT-AP’ specification, developed under the ‘SEMIC’ action, has been reused in the 
development of the ‘CPSV-AP’ under the ‘Catalogue of Services’ action. ‘Access to base 
registries’ reuses ‘Core Vocabularies’, ‘DCAT-AP’ (both part of the ‘SEMIC’ action) and 
‘EIRA’ (part of the ‘European Interoperability Architecture’ action). In  addition, ‘SEMIC’ 
solutions have also been used by the ‘ELISE’ action. 

  The ‘Re3gistry’ solution, part of the ‘ELISE’ action, is used to share codes developed for 

‘Core Vocabularies’ (part of the ‘SEMIC’ action). 

  The platform for secure file exchange ‘e-TrustEx’ (part of the action with the same name) is 
used  in  the  area  of  eProcurement  by  the  ‘European  Public  Procurement  Interoperability 
Initiative’ action, in particular by the ‘e-Prior’ solution developed as part of the action. ‘E-
Prior’ is a module built based on ‘e-TrustEx’. 

  As regards synergies with the ‘Joinup’ platform, there is a particularly strong relationship 
between  the  ‘Sharing  and  Reuse’  action  and  ‘Joinup’.  The  former  aims  to  support  public 
administrations  in  sharing  collaborative  development,  reusing  IT  solutions  and  promoting 
best practices; the latter is a collaborative platform providing the means to achieve this. 

  The ‘National  Interoperability  Framework  Observatory’  (NIFO)  action  is reusing  some  of 
the  information  from  the  ‘Access  to  base  registries’  action  in  order  to  compile  the 
eGovernment factsheets. 

JC2: External coherence 

In order to assess the external coherence of ISA2, stakeholders were asked to assess the 
level  of  synergies  and  overlaps  existing  between  ISA2  and  four  EU  programmes 
identified  as  relevant  for  the  field  of  digitalisation  of  public  administrations  and 
interoperability: i) CEF60, ii) ESF61, iii) Horizon 202062 and iv) SRSP63. The results are 
presented in Figure 16. 

60  See  footnote  Error!  Bookmark  not  defined.  and  https://ec.europa.eu/inea/en/connecting-europe-

facility. 

61  See footnote Error! Bookmark not defined.. 
62  See footnote Error! Bookmark not defined.. 
63  See footnote Error! Bookmark not defined.. 

51 

 
 
 
 
                                                 
Figure 16: Extent to which there are synergies / overlaps between ISA2 and other 
relevant EU programmes (average score of answers, number of respondents) 

e
p
o
r
u
E

g
n
i
t
c
e
n
n
o
C

t
r
o
p
p
u
S
m
r
o
f
e
R

l

a
r
u
t
c
u
r
t
S

y
t
i
l
i
c
a
F

d
n
u
F

l

a
i
c
o
S
n
a
e
p
o
r
u
E

0
2
0
2

n
o
z
i
r
o
H

e
m
m
a
r
g
o
r
P

r
e
h
t
O

Overlaps

Synergies

Overlaps

Synergies

Overlaps

Synergies

Overlaps

Synergies

Overlaps

Synergies

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners
Programme governance
Solution users
Stakeholders responsible for linked EU policies initiatives
Wider public

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

52 

 
 
 
 
 
 
 
 
Out  of  the  four  programmes,  respondents  to  the  consultation  activities  identify  the 
CEF  as  having  the  highest  level  of  synergies  with  ISA2.  According  to  78 %  of 
respondents  (51  out  of  65),  ISA2  and  CEF  have  a  high  level  or  a  very  high  level  of 
synergies.  As  an  example,  both  programmes  contribute  to  the  Commission’s  open  data 
policy:  the DCAT-AP  solution  of  ISA2 provides a metadata specification used by open 
data  portals  across  the  EU  and  CEF  ensures  funding  for  the  public  open  data 
infrastructure  that  aims  to  increase  the  reuse  of  open  public  data  in  the  EU.  However, 
18 % of respondents (10 out of 57) also note the existence of overlaps to a high extent or 
to  the  fullest  extent,  which  could  be  addressed  by  better  delineating  the  scope  of  the 
concerned solutions. 

When  it  comes  to  the  relationship  between  Horizon  2020  and  ISA2,  half  of  the 
respondents  (31  out  of  63)  identify  synergies  to  a  high  extent  or  to  the  fullest  extent, 
while only 12 % (6 out of 51) indicate the existence of overlaps to a high extent or to the 
fullest extent. The situation is similar for SRSP, with 52 % of respondents (22 out of 42) 
pointing out  that there are synergies between  ISA2 and SRSP either to  a high extent or 
the  fullest  extent,  and  only  6 %  (2  out  of  32)  indicating  a  high  or  very  high  degree  of 
overlaps. 

At the opposite end of the spectrum, out  of the four programmes listed above, the  ESF 
seems to be the programme with the lowest level of coherence with ISA2. Only 26 % of 
respondents (6 out of 23) indicate the existence of synergies between the ESF and ISA2 
to a high extent or to the fullest extent.  

Respondents  also  pointed  to  synergies  and  overlaps  with  other  EU  initiatives.  More 
specifically,  three  additional  EU  programmes  and  initiatives  were  mentioned:  the 
European  Regional  Development  Fund  (ERDF)64,  corporate  IT  governance65  and  the 
Single Digital Gateway66. This feedback is discussed in the points below. 

  While  the  synergies  between  the  ESF  and  ISA2  appear  to  be  limited,  two 
respondents pointed to the existence of a high synergy level between the  ERDF 
and ISA2. At the same time, one respondent indicated that there could be overlaps 
between  ISA2  and  ERDF,  possibly  due  to  a  lack  of  coordination.  More 
specifically,  national  IT  systems  developed  using  ERDF  funds  could  be 
incompatible with ISA2 solutions, thus preventing the adoption of these solutions. 

  A high degree of complementarity was also identified between the Commission’s 
corporate  IT  governance  and  ISA2,  as  explained  by  one  respondent.  The 
corporate  IT  governance  team  maintains  that  the  Commission’s  proposed  IT 
investments  take  into  consideration  the  reuse  of  IT  solutions  that  have  already 
been developed. In this sense, the work of this team supports synergies between 
ISA2 and other initiatives, ensuring that ISA2 solutions and any other IT solutions 

64  See footnote Error! Bookmark not defined.. 
65  For further details, please see: https://ec.europa.eu/info/departments/informatics/it-governance_en. 
66  See footnote Error! Bookmark not defined.. 

53 

 
                                                 
developed so far are reused in the future. In addition, the team helps mitigate the 
risk  of  developing  very  similar  IT  solutions  under  different  spending 
programmes, which would create overlaps. 

  One respondent also mentioned the existence of synergies with the Single Digital 
Gateway  (SDG).  In  fact,  this  is  clear  given  that  ISA2  implemented  an  action 
dedicated to the interoperability requirements of the SDG in 201867. ISA2 helped 
detail  the  IT  architecture  of  the  SDG  and  identify  functional,  technical  and 
semantic interoperability challenges in its implementation. Further ISA2 solutions 
like ‘eForms’68 and ‘e-Documents’69 also support the SDG’s implementation. 

Table  18  in  Annex  6  shows  that  ISA2  actions  have  multiple  links  outside  of  the 
programme, as they rely on other EU programmes / policies / initiatives to deliver their 
results and their solutions are used by other programmes / policies / initiatives. 

Beyond examining alignment with the initiatives mentioned above, it is worth analysing 
the degree of alignment of ISA2 with other initiatives or broader policies, like the Digital 
Single Market Strategy, the eGovernment Action Plan, ICT standardisation or the Tallinn 
Declaration on eGovernment. 

When it comes to the Digital Single Market (DSM) Strategy70, the desk review of the 
rolling work plan reveals that  ISA2 contributes to all three of the strategy’s pillars  (see 
Annex 6: Table 19): (i) better access for consumers and businesses to digital goods and 
services  across  Europe;  (ii)  creating  the  right  conditions  and  a  level  playing  field  for 
digital  networks  and  innovative  services  to  flourish;  and  (iii)  maximising  the  growth 
potential  of  the  digital  economy.  Based  on  the  20  sampled  actions,  ISA2  in  particular 
contributes to the strategy’s third pillar through: 

 

implementation  of 
the  revision  and 
Framework, which is explicitly mentioned in the DSM strategy; 

the  new  European  Interoperability 

  work on supporting the implementation of the once-only principle, carried out as 
part  of  several  actions  such  as  ‘SEMIC’  and  ‘European  Interoperability 
Architecture’; and 

  support for the implementation of the Single Digital Gateway. 

Together with the launch of the DSM strategy for Europe, a new  eGovernment Action 
Plan  was  announced  for  2016-2020,  aiming  to  ‘remove  existing  digital  barriers  to  the 

67  The  ISA2 action  mentioned is ‘2017.05 Interoperability requirements for the Single Digital Gateway 
implementation’.  For  more  information  see:  https://ec.europa.eu/isa2/actions/common-architecture-
single-digital-gateway_en. 

68  Please see the description of the ‘ABCDE — Administrations, business and citizens’ data exchanges in 
the  2018  ISA2  rolling  work  programme: 

the  domain  of  case  management’  solution 
https://ec.europa.eu/isa2/library/isa%C2 %B2-work-programme_en. 

in 

69  Please see: https://ec.europa.eu/isa2/solutions/e-documents-reference-architecture_en. 
70  See footnote Error! Bookmark not defined.. 

54 

 
                                                 
Digital Single Market  and to prevent further fragmentation arising in the context of the 
modernisation  of  public  administrations’71.  The  ways  in  which  ISA2  contributed  to 
implementing the DSM strategy also directly relate to the eGovernment Action Plan. In 
particular,  the  revised  version  of  the  EIF  and  the  ISA2  actions  monitoring  its 
implementation in the Member States (‘EIF implementation and governance models’ and 
‘NIFO’)  address  Action  4  of  the  eGovernment  Action  Plan.  Furthermore,  ISA2  is  also 
contributing  to  Actions  1  and  6  through  its  work  in  the  field  of  eProcurement.  For 
instance,  the  ‘European  public  procurement  interoperability  initiative’  action  facilitates 
the implementation of eProcurement in European public administrations and supports the 
EU’s TOOP72 project by adjusting the necessary tools for the once-only principle. 

In  the  field  of  ICT  standardisation,  the  Rolling  Plan  for  ICT  Standardisation73 
provides  a  good  overview  of  the  EU’s  main  initiatives  and  priorities  in  this  area.  The 
Commission  drafts  this  plan  in  collaboration  with  the  European  Multi-Stakeholder 
Platform  on  ICT Standardisation, which includes  representatives of the Member States, 
EFTA  countries,  European  and  international  standardisation  organisations,  industry, 
SMEs  and  consumers.  The  2018  Rolling  Plan  on  ICT  Standardisation  points  out  two 
main areas with highly relevant synergies between ISA2 and ICT standardisation: 

1.  eGovernment.  The  Rolling  Plan  emphasises  that  in  order  to  leverage  the 
applicability  of  technical  specifications  developed  under  ISA2,  one  solution 
would  be  to  suggest  using  them  as  standards  by  the  relevant  standardisation 
organisations for the specifications in question. In this respect, it is worth noting 
that  W3C,  an  international  standardisation  forum,  has  discussed  some  ISA2 
solutions (like the ‘ADMS’ specification74) and has published a W3C note75. 

2.  Public  Sector  Information,  Open  Data  and  Big  Data.  The  Rolling  Plan  also 
mentions the usefulness of introducing a common standard for referencing open 
data in  European open data portals.  The ‘DCAT  Application Profile’76, an ISA2 
solution  that  is  already  used  by  several  open  data  portals  across  the  EU,  stands 
out as a candidate for this purpose. The Rolling Plan invites the CEN ‘to support 
process’.  
and 
There are several potential issues with standardisation though. More specifically, 
the evaluation found that the coherence between intellectual property rights for 

standardisation 

DCAT-AP 

assist 

the 

71  See footnote Error! Bookmark not defined., p. 1. 
72  The ‘once-only’ principle project, http://www.toop.eu/. 
73  Please see: https://ec.europa.eu/digital-single-market/en/rolling-plan-ict-standardisation. 
74  Please see: https://ec.europa.eu/isa2/solutions/asset-description-metadata-schema-adms_en. 
75  See details about the W3C note process: https://www.w3.org/Consortium/Process/NOTE.html. 
76  Please see: https://ec.europa.eu/isa2/solutions/dcat-application-profile-data-portals-europe_en. 

55 

 
                                                 
ISA2  solutions  and  CEN/CENELEC  standards  needs  to  be  clarified  and 
resolved before the standardisation system could be fully implemented77. 

Finally, in order to gain a comprehensive understanding of EU-level coherence, it is also 
useful to analyse the alignment of ISA2 with the priorities set at the highest political level 
in the Tallinn Declaration on eGovernment78. The assessment in Box 10 under Annex 
6  concludes  that  there  is  no  contradiction  between  the  Tallinn  Declaration  and  the 
ISA2 programme: even though they have different scope and aims, they proceed in the 
same  direction  and  share  common  goals.  However,  strong  coordination  is  required 
between them to build synergies and avoid overlaps and duplication. 

Answer to the ‘coherence’ EQs 

EQ6: To what extent do the ISA² actions form part of a ‘holistic’ approach within the 
framework of the programme? (Internal coherence) 

ISA2  actions  are  characterised  by  substantial  synergies  with  each  other  and  limited 
overlaps. Stakeholder feedback on synergies and overlaps shows a positive picture of the 
internal  coherence  of  the  ISA2  programme.  There  are  multiple  instances  of  internal 
reuse  of  ISA2  solutions,  with  actions  like  ‘SEMIC’  and  ‘Joinup’  playing  a  particularly 
important  role  in  the  system.  At  the  same  time,  some  actions  tend  to  be  less  central 
within  the  network.  This  is  for  instance  the  case  of  the  ‘EUSurvey’  and  ‘European 
Citizens’  Initiative  and  European  Parliament  Election’  actions  and  can  be  explained  by 
the particular nature of these actions. 

This  strong  internal  coherence  is  in  line  with  the  second  recommendation  of  the 
final ISA evaluation, which called for taking a holistic approach on interoperability 
within ISA2. 

EQ7: To what extent is the ISA² programme coherent with other EU interventions, which 
have similar objectives and with global initiatives in the same field? (External coherence) 

When  it  comes  to  external  coherence,  ISA2  followed  the  third  recommendation  of  the 
final  ISA  evaluation  and  developed  close  cooperation  with  other  EU  policies  and 
initiatives.  Respondents  to  the  consultation  activities  identified  the  highest  level  of 
synergies between ISA2 and CEF. They also pointed out the synergies with ‘other’ EU 
initiatives  such  as  the  ERDF,  corporate  IT  governance  and  the  Single  Digital 
Gateway.  Synergies  also  exist  between  ISA2  and  some  broader  EU  initiatives  and 
policies,  such  as  the  Digital  Single  Market  strategy,  the  eGovernment  Action  Plan, 
the  Rolling  Plan  for  ICT 
the  Tallinn  Declaration  on  eGovernment,  and 
standardisation. 

77  On  a  more  general  note,  it  is  not  only  the  various  European  standards  that  may  need  to  be  better 
aligned; the links between European, international and global standards could also be better clarified. 
The  final  ISA2  evaluation  may  investigate  what  the  role  of  a  European  interoperability  programme 
should be in this regard. 

78  See footnote Error! Bookmark not defined.. 

56 

 
                                                 
In  conclusion,  ISA2  actions  have  multiple  links  outside  of  the  programme,  as  they 
rely  on  other  EU  programmes  /  policies  /  initiatives  to  deliver  their  results  and  their 
solutions are used by other programmes / policies / initiatives. This further corroborates 
the positive feedback on the programme’s external coherence. 

5.5.  EU added value 

Evaluation framework for EU added value 

The EU added value captures the programme's impacts additional to those that would be 
achieved if the issues addressed by ISA2 were left solely in the hands of national and sub-
national authorities. This evaluation criterion seeks to answer the following EQ: 

EQ8:  What is the additional value resulting from the ISA² programme, compared 
to what could reasonably have been expected from Member States acting at 
national, regional and/or local levels? 

To answer this question, the evaluation analysed the following judgment criteria based on 
both primary and secondary data: 

JC1:  Achievement of objectives that could not be otherwise attained with national or 

sub-national interventions; 

JC2:  Achievement  of  objectives  at  a  cost  lower  than  what  could  be  attained  via 

national or sub-national interventions; 

JC3:  Achievements in terms of cross-border interoperability; and 

JC4:  The contribution to the advancement of common EU policies. 

Analysis 

JC1: Achievement of objectives that could not have been achieved without EU-level 
intervention 

Most  respondents  believe  that  national  or  sub-national  interventions  would  not  be 
able to achieve the general and specific objectives in the field of interoperability that 
ISA2  is  pursuing,  or  that  they  would  achieve  only  be  able  to  achieve  them  to  a 
limited  extent  (Figure  17).  This  observation  is  particularly  evident  when  it  comes  to 
developing,  maintaining  and  promoting  a  holistic  approach  to  interoperability  in  the 
Union (specific objective 1), as 83 % of respondents (95 out of 115) believe that national 
or  sub-national  interventions  would  not  be  able  to  achieve  this  objective  or  would 
achieve  it  only  to  a  limited  extent.  Additionally,  72 %  of  respondents  (83  out  of  116) 
point to limited achievements or no achievement from national or sub-national initiatives 
with  regard  to  facilitating  the  efficient  and  effective  electronic  cross-border  or  cross-
sector  interaction  between  European  public  administrations,  and  between  European 
public administrations and businesses and citizens (specific objective 2). These examples 
show that the coordination provided by a unique EU-level programme plays an important 
role  in  improving  overall  interoperability  among  European  public  administrations.  The 
technical experts supporting the evaluation have unanimously confirmed this conclusion. 

57 

 
Figure 17: Extent to which national or sub-national interventions would be able to 
achieve the ISA2 objectives without the programme (average score of answers, 
number of respondents) 

General objective: To promote the ICT-based
modernisation of the public sector in Europe and to
facilitate addressing the needs of businesses and citizens
via improved interoperability of public administrations,
contributing to the DSM and to growth

Specific objective 1: To develop, maintain and promote a
holistic approach to interoperability in the Union in order
to eliminate fragmentation in the interoperability
landscape in the Union

Specific objective 2: To facilitate efficient and effective
electronic cross-border or cross-sector interaction
between European public administrations, and between
European public administrations and businesses and
citizens

Specific objective 3: To contribute to the development of
a more effective, simplified and user-friendly e-
administration at the national, regional and local levels of
public administration

Specific objective 4: To identify, create and operate
interoperability solutions supporting the implementation
of Union policies and activities

Specific objective 5: To facilitate the reuse of
interoperability solutions by European public
administrations

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance

Solution users

Stakeholders responsible for linked EU policies initiatives

Wider public

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

There is, however, one specific area in which national or sub-national interventions 
would be able to contribute in the absence of an EU-level programme. A quarter of 
respondents (29 out of 115) believe that national or sub-national interventions would be 
able to contribute to the development of a more effective, simplified and user-friendly e-
administration at the national, regional and local levels (specific objective 3) to a high 
extent  or  to  the  fullest  extent.  This  result  shows  how  well  EU  and  national  initiatives 

58 

 
 
complement each other, since specific objective 3 is the one where the ISA2 programme 
shows the worst performance in terms of effectiveness (see Figure 6 in section 5.2). 

It is worth emphasising that while national initiatives related to the ICT modernisation of 
the  public  sector  cannot  achieve  the  objectives  pursued  by  ISA2,  they  are  still  very 
important to improving the programme’s performance as they increase the capacity 
for  the  take-up  of  ISA2  solutions.  Initiatives  pursued  by  Member  States  in  this  field 
show that interoperability is a priority, which encourages working together with ISA2. In 
addition,  from  the  point  of  view  of  ICT  development,  national  initiatives  put  Member 
States in a better position to use the solutions provided by ISA2. 

JC2: Achievement of objectives at a lower cost 

In  addition,  83 %  of  respondents  (91  out  of  109)  emphasised  that  ISA2  is  probably  or 
definitely  able  to  achieve  its  objectives  at  a  lower  cost  than  comparable  national  or 
sub-national  initiatives  (Figure  18).  The  main  stakeholders  of  ISA2  appreciate  the 
economies of scale that this EU-level programme generates. 

Figure 18: Ability of ISA2 to achieve the relevant objectives at a lower cost than 
comparable national or sub-national interventions (average score of answers, 
number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

[CELLRANGE] 

Solution users

[CELLRANGE] 

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

1

2

3

4

[CELLRANGE] 

Standardisation organisations

Wider public

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

JC3: Achievements in terms of cross-border interoperability 

ISA2 focuses on providing both cross-border and cross-sectoral interoperability solutions. 
From  the  point  of  view  of  EU  added  value,  the  contribution  to  cross-border 
interoperability  is  particularly  relevant,  as  it  shows  the  extent  to  which  the  programme 
works towards achieving a holistic approach to interoperability in the EU. 

Over half of the respondents (50 out of 93) believe that ISA2 has contributed to a high 
extent or to the fullest extent to improving cross-border interoperability in the EU. 
This result holds true across the consulted stakeholder groups, in comparison to less than 
6 % (6 out of 93) who only see a limited contribution in this area (Figure 19). 

59 

 
 
Figure 19: Extent to which ISA2 contributed to improving cross-border 
interoperability in the EU (average score of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

[CELLRANGE] 

Solution users

[CELLRANGE] 

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

Standardisation organisations

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Several respondents also emphasised the following aspects: 

  Awareness: ISA2 raises awareness of interoperability in the EU and helps put the 
topic on national agendas; without this, little progress would be made in the field. 

  Organisational  contribution:  ISA2  brings  people  together,  creating  networks 
and helping national organisations meet their counterparts in different countries, 
thus facilitating exchanges between Member States. 

Desk research also supports the two examples mentioned above. First, ISA2 is committed 
to  raising  awareness  about  interoperability  through  its  communication  activities 
presented  in  Box  3.  Second,  it  contributes  to  the  creation  of  a  network  of  officials 
working on interoperability by holding regular  ISA2 Committee and  ISA2  Coordination 
Group meetings, which bring together representatives of Member States responsible for 
this  topic.  There  were  five  ISA2  Committee  meetings  and  three  Coordination  Group 
meetings  between  2016  and  2018.  In  most  of  them,  all  28  Member  States  were 
represented by at least one participant (with a maximum of three participants per meeting 
per  Member  State).  In  addition,  Norway  was  represented  in  all  Coordination  Group 
meetings and in four of  the five Committee meetings,  Iceland was represented in  three 
Committee  meetings  and  one  Coordination  Group  meeting,  and  finally  Montenegrin 
representatives were present  in  three Committee meetings and one Coordination  Group 
meeting (see Annex 6: Table 20 and Table 21). 

The fact that countries beyond the EU have either become members of ISA2 or cooperate 
with  the  programme  shows  that  the  programme’s  benefits  do  not  stop  at  the  EU 
borders.  An  additional  example  is  the  administrative  cooperation  between  DG  DIGIT 
and the Agency for the Development of the Government of Electronic Management and 
the Information and Knowledge Society of Uruguay, officially started on 26 March 2018 

60 

 
 
and consisting of delegation visits, exchanges of experts, workshops, and sharing of best 
practices. 

Further evidence comes from analysing the cross-border interoperability angle of the 20 
sampled ISA2 actions: 29 out of the 35 operational solutions developed by these actions 
have a cross-border feature (Annex 6: Table 11). One example is the IMAPS solution79, 
which  facilitates  the  assessment  of  public  services  at  the  local,  national,  and  EU  level 
from  the  point  of  view  of  their  interoperability  maturity.  In  practice,  it  consists  of  a 
compact and user-friendly self-assessment carried out through the IMAPS online survey. 
This 30-minute questionnaire looks at the context of the public service, how the service is 
delivered, how it consumes existing services and how it is  managed. On completion of 
the IMAPS survey, the results give tailored recommendations and confidential guidance 
for improvement. 68 public administrations completed the assessment in 2017 and 51 in 
2018 — at local, national and EU level. The solution has thus had a cross-border impact, 
since  the  assessments  are  directed  at  public  administrations  all  across  the  EU  as 
confirmed by the user testimonial in Box 7. 

Box 7: IMAPS user testimonial 

Source: ISA2 Communication Team 

79  See: https://ec.europa.eu/isa2/solutions/imaps_en. 

61 

 
 
 
                                                 
Further key achievements include (Annex 6: Table 22): 

  The  ‘e-TrustEx’  platform  is  a  key  enabler  of  secure  information  exchange  for 
seven  pan-European  projects  in  which  over  200  public  administrations  from  all 
Member States participate. 

  The  ‘Big  Data  for  Public  Administrations’  action  created  a  working  group  to 
assess  the  needs  in  the  area  of  big  data  analytics  for  policy  making  with  the 
participation of the Czech Republic, Estonia, Hungary, Malta, Norway, Slovenia, 
Spain, and Portugal. 

  Several actions foster multilingualism: 

o 

through  their  work  on  providing  common  frameworks,  the  actions  on 
semantic interoperability contribute to addressing multilingual challenges 
to  the  cross-border  interaction  of  European  public  administrations;  one 
example is  the translation of the six ‘Core Vocabularies’ into 23 official 
EU languages80; 

o 

tools  have  been  developed  by  ISA2  with  the  specific  purpose  of  being 
ready  for  use  in  all  official  EU  languages;  one  example  is  ‘EUSurvey’; 
and 

o  CPSV-AP  is  used,  for  instance,  in  Finland  and  Estonia  to  create  cross-

border federated catalogues of public services. 

One potential improvement would be to  further increase the  programme’s  focus 
on  multilingualism  by  providing  more  information  about  ISA2  in  different  EU 
official languages, as the current website is only available in English. 

JC4: Contribution to the advancement of common EU policies 

Half  of  the  respondents  (44  out  of  84)  indicate  that  ISA2  has  contributed  to  a  high 
extent  or  to  the  fullest  extent  to  the  advancement  of  common  EU  policies  or 
initiatives (Figure 20). For instance, stakeholders noted that ISA2 plays a central role in 
the  implementation  of  the  EIF  by  abiding  by  its  principles  and  tracking  its 
implementation.  The  review  of  secondary  data  provides  additional  evidence  supporting 
this.  The  Communication  on  the  revised  EIF81  is  accompanied  by  an  ‘Interoperability 
Action Plan, which sets out five focus areas and 22 actions to be undertaken in order to 
tackle existing interoperability issues during the 2017-2020 period. The Communication 
is  also  accompanied  by  47  specific  recommendations  for  improving  interoperability  in 
the  EU.  ISA2  actions  can  be  clearly  linked  to  the  focus  areas,  actions,  and 
recommendations outlined in the revised EIF (see Annex 6: Table 23). 

80  See: https://ec.europa.eu/isa2/news/interested-semantic-interoperability-read-about-semic-highlights-

second-half-2018_en. 

81  See footnote Error! Bookmark not defined.. 

62 

 
                                                 
Figure 20: Extent to which ISA2 contributed to the advancement of common EU 
policies (average score of answers, number of respondents) 

[CELLRANGE] 

Action owners

[CELLRANGE] 

Programme governance

[CELLRANGE] 

Solution users

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

By  contributing to  the implementation  of the revised EIF,  ISA2 is  also contributing  to 
the establishment of the Digital Single Market, as explained in section 5.4 on external 
coherence. The technical experts involved in the evaluation note that the programme has 
increased attention paid to the importance of interoperability and cross-border e-services 
for sustaining the Digital Single Market. In this way, the programme fully meets the aims 
of  advancing  common  shared  policies  within  the  EU,  as  stated  in  the  Tallinn 
Declaration. 

Answer to the ‘EU added value’ EQ 

EQ8: What is the additional value resulting from the ISA² programme, compared to what 
could  reasonably  have  been  expected  from  Member  States  acting  at  national,  regional 
and/or local levels? 

ISA2  ensures  EU-level  coordination,  which  is  necessary  for  improving  overall 
interoperability  among  European  public  administrations.  Without  the  programme, 
national  and/or  sub-national 
the 
interoperability-specific  objectives  that  ISA2  is  aiming  at.  In  addition,  ISA2  is  able  to 
achieve  these  objectives  at  a  lower  cost  than  comparable  national  or  sub-national 
initiatives.  These  conclusions  hold  true  across  the  different  stakeholder  groups  and  are 
further corroborated by expert assessment. 

interventions  would  not  be  able 

to  achieve 

Nonetheless,  national  or  sub-national  interventions  would  be  able  to  contribute  to  the 
development  of  a  more  effective,  simplified  and  user-friendly  e-administration  at  the 
national, regional and local levels even if there was no EU-level programme. This is the 
specific objective where the ISA2 programme appears to be less effective, thus showing 
how well EU and national initiatives complement each other. National initiatives on 
the ICT modernisation of the public sector can even improve the performance of ISA2 
by increasing capacity for the take-up of ISA2 solutions. 

63 

 
 
ISA2  also  creates  additional  value  by  improving  cross-border  interoperability  in  the 
EU,  as  well  as  advancing  common  EU  policies.  As  regards  the  former,  ISA2  raises 
awareness  of  interoperability  across  EU  Member  States  and  helps  put  the  topic  on 
national agendas. It also brings people together, thus creating networks, helping national 
organisations  meet  their  counterparts  in  different  countries  and  facilitating  exchanges 
between Member States in the field of interoperability. As regards the latter, ISA2 plays a 
central role in implementing the EIF and supports the establishment of the Digital Single 
Market. 

5.6.  Utility 

Evaluation framework for ‘utility’ 

Article 13(4)  of  the  ISA2  Decision  requires  an  assessment  of  the  programme’s  utility. 
Utility refers to (i) the extent to which the results of ISA2 meet stakeholders’ needs; and 
(ii) the level of stakeholder satisfaction with ISA2 solutions. This criterion translates into 
the following EQ: 

EQ9:  How  do  the  ISA2  programme’s  actions  and  results,  achieved  and 
anticipated, compare with the needs they are supposed to address? 

The assessment of this EQ is based on primary data collected via consultation activities 
as well as on evidence gathered via desk research, and relies on the following judgment 
criteria: 

JC1:  Alignment  between  stakeholders’  perception  of  needs  and  problems  at  EU, 

national and sub-national levels and the results of the programme; and 

JC2:  User satisfaction, with a breakdown by stakeholder group. 

Analysis 

JC1: Alignment between EU- and Member State-level needs and problems and the 
programme’s results  

Respondents  from  both  EU  and  Member  State  public  administrations  confirm  that  the 
solutions  developed  or  maintained  by  ISA2  have  contributed  to  addressing  the 
original needs and problems identified in the field of interoperability (Figure 21 and 
Figure  22)82.  Overall,  42 %  of  respondents  at  the  EU  level  (38  out  of  91)  and  46 %  of 
respondents  at  the  Member  State  level  (13  out  of  28)  consider  that  the  solutions  have 
contributed to addressing the needs and problems originally covered by the programme 
to a high extent or to the fullest extent. In comparison, only 5 % of EU-level respondents 
(5 out of 91) and 11 % of Member State-level respondents (3 out of 28) see a limited or 
no contribution. 

82  See Error! Reference source not found. for the original needs and problems. 

64 

 
                                                 
Figure 21: Extent to which ISA2 solutions contributed to addressing the needs and 
problems originally covered by the programme: EU-level respondents (average 
score of answers, number of respondents)

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance (from
DIGIT)

Solution users (from EC)

Stakeholders responsible for
linked EU policies initiatives (with
the exception of CIOs)

Standardisation organisations

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Figure 22: Extent to which ISA2 solutions contributed to addressing the needs and 
problem originally covered by the programme: respondents from the Member 
States (average score of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Programme governance
(Committee members and
Coordination group members)

Solution users (except from EC)

Stakeholders responsible for
linked EU policies initiatives
(CIOs)

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

As the programme is still ongoing, the ways in which solutions meet user needs may still 
improve.  It  is  expected  that  the  take-up  rate  of  solutions  will  increase  over  time, 
especially  for  the  programme’s  newer  solutions,  a  conclusion  that  was  explained  in 
section  5.2  on  effectiveness.  With  a  higher  adoption  rate  of  ISA2  solutions  across 
European  public  administrations,  the  needs  and  problems  would  be  increasingly 
better addressed, thus leading to higher user satisfaction. 

The  feedback  from  consultation  activities  is  further  supported  by  secondary  data. 
When it comes to the need for cooperation among public administrations with the aim to 
enable more efficient and secure public services, one example of the contribution of ISA2 
is  the  “Access  to  Base  Registries”  action.  The  action  has  developed  and  is  constantly 

65 

 
 
 
updating a “Catalogue of Solutions”83 for base registries, facilitating interconnection and 
cooperation  between  public  administrations.  In  addition,  a  set  of  guidelines  for 
addressing  potential  challenges  has  been  created.84  Both  outputs  help  implement  the 
once-only  principle  thus  increasing  the  efficiency  and  user-friendliness  of  public 
services.  Another  example  of  an  ISA2  action  fostering  cooperation  in  order  to  enhance 
efficiency in the public sector is the “Communication and information resource centre for 
administrations,  businesses  and  citizens  (CIRCABC)”  action  and  tool85.  CIRCABC 
enables 
institutions,  European  public 
administrations  and  other  stakeholders,  by  offering  a  free-of-charge,  trustworthy  tool. 
Feedback is constantly collected from users in order to improve the way the tool meets 
their needs. 

the  efficient  collaboration  between  EU 

As  regards  the  need  for  exchanging  information  among  public  administrations  to  fulfil 
legal  requirements  or  political  commitments,  the  “Open  e-TrustEx”  solution,  for 
instance,  provides  a  platform  for  the  secure  exchange  of  information  between  public 
administrations  in  the  EU  and  in  the  Member  States.  It  is  used  in  seven  pan-European 
projects, including the e-Prior Open Source e-Procurement platform86 and the European 
eJustice  Portal87,  contributing  to  cross-border  interoperability.  Another  example  is  the 
“Inter-institutional register of delegated acts (RegDel)”88, which directly responds to the 
commitment  of  the  Commission,  Council,  and  European  Parliament  to  increase 
transparency and traceability of the delegated acts process.89  

Turning to the need for sharing and reusing information among public administrations to 
increase  administrative  efficiency  and  cut  red  tape  for  citizens  and  businesses,  the 
“Joinup”  platform  can  be  mentioned.  It  provides  a  collaborative  space  where  solutions 
can  be  shared  between  stakeholders.  Additionally,  the  platform  also  hosts  a  number  of 
national repositories where five Member States have made national solutions available to 
other interested parties. An additional example in this regard is the “Development of an 
Open  Data  Service,  Support  and  Training  Package  in  the  Area  of  Linked  Open  Data, 
Data Visualisation and Persistent Identification” action, which contributes to the sharing 
of  best  practices,  by  providing  a  set  of  reusable  tools  for  data  visualisation  and  data 
manipulation, available on the EU Open Data Portal.90 

As noted in section 5.1 on relevance, several respondents identified additional needs and 
problems  to  those  explicitly  mentioned  in  the  programme’s  intervention  logic.  In  this 

83   See: 

https://joinup.ec.europa.eu/collection/european-interoperability-reference-architecture-

eira/catalogue-solutions 

84   See:  https://joinup.ec.europa.eu/collection/european-interoperability-reference-architecture-eira/good-

practices 

85   See: https://ec.europa.eu/isa2/solutions/circabc_en 
86  See: http://ec.europa.eu/isa2/library/eprior/ 
87  See: https://e-justice.europa.eu/home.do?action=home 
88  See: https://webgate.ec.europa.eu/regdel/#/home 
89   See: http://www.europarl.europa.eu/doceo/document/TA-8-2016-0081_EN.html?redirect 
90   See: https://data.europa.eu/euodp/en/visualisation-home 

66 

 
                                                 
respect,  27 %  of  EU-level  respondents  (11  out  of  41)  and  50 %  of  Member  State-level 
respondents  (6  out  of  12)  indicated  that  solutions  have  contributed  to  addressing  these 
additional  needs  and  problems  to  a  high  extent  or  to  the  fullest  extent.  In  comparison, 
22 % of EU-level respondents (9 out of 41) and 25 % of Member State-level respondents 
(3 out  of 12) believe that  this contribution  has been either limited or absent  (Figure 23 
and Figure 24). 

Figure 23: Extent to which ISA2 solutions contributed to addressing additional 
needs and problems identified by consulted stakeholders: EU-level respondents 
(average score of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance (from
DIGIT)

Solution users (from EC)

Stakeholders responsible for
linked EU policies initiatives (with
the exception of CIOs)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Figure 24: Extent to which ISA2 solutions contributed to addressing additional 
needs and problems identified by consulted stakeholders: respondents from the 
Member States (average score of answers, number of respondents)

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Programme governance
(Committee members and
Coordination group members)

Solution users (except from EC)

Stakeholders responsible for
linked EU policies initiatives
(CIOs)

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

JC2: Stakeholder satisfaction 

The  feedback  received  from  respondents  regarding  user  satisfaction  tends  to  be 
positive (Figure 25). More specifically, half of the respondents (57 out of 110) indicate 

67 

 
 
 
that they are satisfied with the way solutions meet user needs to a high extent or to the 
fullest extent, while only 6 % (7 out of 110) indicate limited satisfaction. 

Figure 25: Extent to which ISA2 solutions are meeting user needs in the fields of 
cross-border and cross-sectoral interoperability of digital public services (average 
score of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

Stakeholders responsible for
linked EU policies initiatives
Standardisation organisations

0

1

2

3

4

5

Wider public

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Respondents also provided feedback on measures that could be taken to improve the 
programme’s  utility.  This  feedback  centres  on  the  following  aspects  and  shows 
significant  complementarities  with  the  measures  proposed  to  improve  the  relevance  of 
ISA2 (see section 5.1): 

  Best  practices  and  user  support.  An  increased  focus  on  sharing  best  practices 
and providing support to users would have a beneficial effect on user satisfaction. 

  More focus on already existing solutions. According to the feedback from the 
consultation activities, another way to better address user needs would be to focus 
more on the already existing solutions and on improving their quality. This would 
improve the way in which user needs are considered in the further development of 
a smaller group of actions. 

  Promotion.  Respondents  indicated  that  they  would  like  to  see  more  activities 
promoting ISA2 at the Member State level in order to increase the level of take-up 
of  solutions.  An  increased  level  of  take-up  would  contribute  to  addressing  the 
needs identified in the field of interoperability, thus improving overall satisfaction 
levels. Furthermore, feedback received from standardisation organisations shows 
that  more  awareness  is  needed  among  practitioners  as  well,  particularly  in  the 
field  of  standards.  Organising  workshops  targeted  to  specific  groups  of 
professionals could be useful. Formal presentations made at the plenary sessions 
of standards development organisations could also provide more exposure. 

  Co-creation.  Another  measure  could  be  to  involve  users  not  only  in  the  testing 
phase  of  solutions,  but  also  in  the  design  phase,  and  to  establish  a  co-creation 
process.  This  includes  more  extensive  on-boarding  services  (ensuring  that  users 
are  provided  with  extensive  information  about  the  developed  solutions),  an 
increased  focus  on  the  sustainability  of  (re)user  communities,  as  well  as  the 

68 

 
 
integration  of  different  solutions  and  services  into  one  service,  thus  making  it 
easier for users to navigate the landscape of available solutions. 

  Commitment. The ISA2 Committee could reinforce the results of the programme 
by  using  it  to  set  clear  commitments  to  interoperability  in  the  future.  Member 
States could show their commitment to using the solutions stemming from  ISA2 
actions beyond simply participating in the selection of actions for the rolling work 
programme. 

The ISA2 programme’s communication team and action owners already make an effort to 
capture users’ feedback and information on additional needs related to existing solutions. 
Their outreach activities made it possible to put together a number of user testimonials91, 
which support the promotion of solutions. For example, the user story presented in Box 8 
encourages  the  use  of  Joinup.  In  the  underlying  detailed  questionnaire,  the  interviewee 
shared  her  ideas  about  possible  improvements,  including  more  active  marketing  of  the 
solution and the need to keep information on the Joinup platform up-to-date. 

Box 8: ‘Joinup’ user testimonial 

Source: ISA2 Communication Team 

91  See for example: Error! Reference source not found. and Error! Reference source not found.. 

69 

 
 
 
                                                 
Answer to the ‘utility’ EQ 

EQ9:  How  do  the  ISA²  programme’s  actions  and  results,  achieved  and  anticipated, 
compare with the needs they are supposed to address? 

According  to  consulted  stakeholders,  solutions  developed  or  maintained  by  ISA2  have 
contributed  to  addressing  the  original  needs  and  problems  identified  in  the  field  of 
interoperability. As the programme is still ongoing, the way solutions meet user needs 
may  still  improve.  In  particular,  these  needs  and  problems  are  expected  to  be  better 
addressed  due  to  more  extensive  adoption  of  ISA2  solutions  across  European  public 
administrations. 

In line with the conclusion drawn on the relevance criterion (section 5.1), whereas some 
of  the  new  needs  and  problems  experienced  by  consulted  stakeholders  are 
addressed  by  the  programme,  some  others  fall  beyond  its  scope  as  it  is  currently 
defined.  Therefore,  the  solutions  developed  so  far  cannot  respond  to  these  particular 
needs and problems. 

The  feedback  received  from  respondents  in  terms  of  user  satisfaction  tends  to  be 
positive. Nevertheless, both  primary and secondary  data show that  additional measures 
could increase the programme’s utility. For example, ISA2 could place more emphasis on 
providing  support  to  users  of  its  existing  solutions.  Moreover,  it  could  already  involve 
users in the design phase of future solutions, thus increasing quality and user satisfaction. 

5.7.  Sustainability 

Evaluation framework for ‘sustainability’ 

Sustainability is the final evaluation criterion that has to be assessed as part of the interim 
evaluation  according  to  Article 13  of  the  ISA2  Decision.  Sustainability  measures  the 
likelihood that the results of the ISA² programme last beyond its completion. The below 
EQ addresses this criterion: 

EQ10: To  what  extent  is  the  financial,  technical  and operational  sustainability  of 
the  developed  solutions  —  maintained  and  operated  through  the  ISA² 
programme — ensured? 

The  evidence  base  for  assessing  the  sustainability  of  ISA2  consists  of  primary  data 
collected  via  the  consultation  activities,  complemented  by  expert  assessment.  The 
evaluation relies on the following judgment criteria: 

JC1:  Extent to which the results achieved by the ISA2 programme are expected to last 
if funding for the actions covered by the programme would not be available in 
the future; and 

JC2:  Extent to which ‘cost recovery’ solutions could be introduced. 

70 

 
 
Analysis 

JC1: Results would last if funding for actions would not be available in the future 

This  part  of  the  assessment  relies  on  the  hypothetical  scenario  in  which,  after  the 
completion of ISA2, no other similar programme is implemented at the EU level. Overall, 
the  consulted  stakeholders  tend  to  have  a  positive  outlook  on  the  sustainability  of 
results achieved so far (Figure 26), with 79 % of respondents (66 out of 84) believing 
that  the  programme  results  probably  or  definitely  would  last  in  the  absence  of  future 
funding for the actions currently supported by ISA2. 

Figure 26: Likelihood that results achieved so far would last if funding for actions 
covered by the programme would not be available in the future (average score of 
answers, number of respondents) 

Action owners

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Programme governance

Solution users

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

0

1

2

3

4

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

the  specificities  of  different  solutions  and 

Although the global results of the programme are expected to last (as the progress made 
in  the  field  of  interoperability  will  not  be  undone),  certain  issues  may  arise  when 
towards 
considering 
interoperability. In fact, 65 % of respondents (55 out of 85) consider that operations and 
maintenance costs92 required to run the existing solutions would have a highly or fully 
negative  effect  on  the  ability  of  solutions  to  deliver  their  results  once  the  programme 
ends.  In  addition,  56 %  (48  out  of  85)  hold  the  same  opinion  about  the  technical  and 
operational support93 needed for the solutions (Figure 27). 

future  steps 

Beyond these two factors, respondents indicated other aspects that would have a negative 
effect on the solutions once ISA2 came to an end and no other programme replaced it: 

  Lack  of  further  development  for  existing  solutions.  Given  the  rapid  pace  of 
changes  in  the  field  of  ICT,  solutions  that  do  not  benefit  from  continued 
development support and updating face the risk of becoming obsolete over time. 

92  Operations and maintenance costs consist of the  monetary costs incurred to run, monitor, and ensure 

the proper functioning of a solution. 

93  Technical  and  operational  support  refers  to  the  human  resources  mobilised  in  activities  such  as 

helpdesks or providing IT support to users. 

71 

 
 
                                                 
This  factor  emerged  from  the  consultation  activities  and  was  mentioned  by  the 
technical experts as well94. 

  Lack  of  coordination.  In  the  absence  of  a  single  programme  overseeing  the 
development  of  solutions,  future  changes  to  solutions  might  lead  to  diverging 
results  between  public  administrations  or  bilateral  solutions  between  countries 
and a large variety of outcomes. This would be contrary to the goal of achieving a 
holistic landscape of interoperability in the EU. 

  Dissemination  and  communication.  ISA2  also  plays  an  important  role  in 
promoting  interoperability  and  creating  a  space  for  discussions  on  the  topic. 
Without such a programme, the interest in a common approach to interoperability 
would be jeopardised. 

Figure 27: Extent to which financial, technical and operational aspects would 
prevent ISA2 solutions from delivering their results if the programme was 
terminated and not replaced (average score of answers, number of respondents) 

d
n
a

n
o
i
t
a
r
e
p
O

e
c
n
a
n
e
t
n
a
m

i

e
b

o
t

s
t
s
o
c

d
e
r
e
v
o
c

d
n
a

l

a
c
i
n
h
c
e
T

l

a
n
o
i
t
a
r
e
p
o

d
e
d
e
e
n

t
r
o
p
p
u
s

r
e
h
t
O

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance

Solution users

Stakeholders responsible for linked EU policies initiatives

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

The elements  presented  above explain why 87 % of respondents  (75 out  of 86) believe 
that the achievement of the ISA2 general objective would — either probably or definitely 

94  The technical experts involved in the evaluation noted that for most solutions the continued updating 

and maintenance might be more expensive than the initial development. 

72 

 
 
                                                 
 
 
 
 
 
—  be  jeopardised  should  ISA2  be  terminated  without  being  replaced  (Figure  28).  This 
confirms  that  the  ISA2  programme  plays  a  central  role  in  improving  the 
interoperability landscape in the EU by promoting the ICT-based modernisation of the 
public  sector  in  Europe  and  making  it  possible  to  address  the  needs  of  businesses  and 
citizens via improved interoperability of European public administrations. 

Figure 28: Likelihood that the ISA2 general objective would be jeopardised if the 
programme was terminated and not replaced (average score of answers, number of 
respondents) 

[CELLRANGE] 

Action owners

[CELLRANGE] 

Programme governance

[CELLRANGE] 

Solution users

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

1

2

3

4

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

JC2: Cost recovery 

Given  the  adverse  effects  of  operational  and  maintenance  costs  as  well  technical  and 
operational  support,  stakeholders  were  asked  to  assess  the  feasibility  of  introducing  a 
fee for users in order to gain access to ISA2 solutions, in case funding would no longer 
be provided for the actions currently  carried out by  ISA2. Over half of the respondents 
(47 out of 79) would probably or definitely not pay such a fee, in comparison to 40 % of 
respondents (32 out of 79) who indicated that they would probably or definitely pay a fee 
in order to use the solutions (Figure 29). The main reason underlying these responses is 
that users are accustomed to having access to ISA2 solutions free of charge. Changing 
this to a pay-for-access system may lead users to search for other solutions that are open 
source and free of charge, depending on the level of the fee requested. Nevertheless, for 
some  of  the  more  mature  solutions,  a  fee  could  be  acceptable  if  the  level  of  take-up  is 
already quite high, making the solution indispensable to some extent. 

73 

 
 
Figure 29: Likelihood that users would agree to pay a fee to continue accessing ISA2 
solutions (average score of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

Stakeholders responsible for
linked EU policies initiatives

1

2

3

4

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Answer to the ‘coherence’ EQs 

EQ10:  To  what  extent  is  the  financial,  technical  and  operational  sustainability  of  the 
developed solutions - maintained and operated through the ISA² programme - ensured? 

Overall,  consulted  stakeholders  tend  to  have  a  positive  view  of  the  sustainability  of 
results achieved so far. However, while the programme’s global results are expected to 
last (as the progress made in the field of interoperability will not be lost), certain issues 
may  arise  when  considering  the  specificities  of  different  solutions  and  future  steps 
towards improved interoperability. 

In fact, there are solutions that require both operational and maintenance costs as well 
as  technical  and  operational  support,  which  would  render  the  solutions  unable  to 
deliver  further  results  if  the  programme  was  terminated.  Additional  obstacles  to  the 
programme’s  sustainability  include  the  lack  of  further  development  for  existing 
solutions;  the  lack  of  coordination  between  national  administrations;  and  the  limited 
dissemination  of  information  and  communication  about  the  interoperability  of  digital 
public  services.  Consequently,  ISA2  should  investigate  possibilities  to  increase  the 
sustainability  of  the  results  of  its  existing  solutions.  It  could  assess  the  feasibility  and 
costs  of  various  sustainability  measures  and  invest  in  them.  The  programme  could  for 
example  consider  transferring  some  of  its  solutions  to  open  source  communities  or 
encouraging companies to build services around free ISA2 solutions under the European 
Union Public Licence (EUPL)95. 

To conclude, the findings suggest, that the ICT-based modernisation of the public sector 
in  Europe  and  the  possibility  to  address  the  needs  of  businesses  and  citizens  via 
improved interoperability of European public administrations — the general objective of 
ISA2 — would be jeopardised in  the absence of  ISA2 or any similar EU initiative. The 
consulted stakeholders’ reluctance to pay for ISA2 solutions makes this conclusion even 
more probable. 

95  See: https://eupl.eu/. 

74 

 
 
                                                 
6.  CONCLUSIONS 

6.1.  Context 

ISA2 is a European funding programme offering interoperability solutions and common 
frameworks for European public administrations, businesses and citizens. It is the fifth in 
a series of European instruments fostering interoperability in the European public sector 
since  1995.  It  follows  the  ISA  programme,  which  terminated  in  2015,  and  covers  the 
2016-2020 period. 

ISA2 was established by Decision (EU) 2015/2240 of the European Parliament and of the 
Council (‘ISA2 Decision’), which legally requires the European Commission to carry out 
an interim evaluation of the programme by 30 September 2019, and to communicate the 
results. 

6.2.  Method 

Work  on  the  evaluation  started  in  Q2 2018  with  the  involvement  of  interested 
Commission services, who discussed and outlined the evaluation design. External experts 
supported the evaluation  process  as well by  finalising the evaluation  design, collecting, 
validating  and  analysing  data,  and  summarising  their  findings  in  an  independent 
evaluation study96. 

The  agreed  evaluation  framework  (Annex  4)  covered  all  seven  evaluation  criteria 
identified by the ISA2 Decision, namely: relevance, effectiveness, efficiency, coherence, 
EU added value, utility and sustainability. It also described the evaluation questions and 
related judgment criteria for each evaluation criterion, together with the data sources and 
data collection methods. For data collection, the evaluation relied on both primary data 
gathered  through  stakeholder  consultation  activities  and  secondary  data  collected  via 
desk research between November 2018 and March 2019 (Chapter 4). 

Although  the  evaluation  was  able  to  draw  robust  conclusions  for  all  evaluation  criteria 
based on the collected data, there were two limitations:  

1) Timing. As this is an interim evaluation, the analysis covered only the first three years 
of  the  programme  (2016,  2017  and  2018).  The  ongoing  ISA2  actions  are  expected  to 
produce  further  results  in  the  coming  years,  which  will  influence  the  programme’s 
overall achievement of objectives.  

2)  Reach.  ISA2  is  a  very  technical  programme,  and  mainly  involves  interoperability 
experts.  This  explains  the  rather  low  participation  in  the  stakeholder  consultation 
activities.  Another  reason  stems  from  the  stringent  confidentiality  and  data  protection 
rules:  without  their  specific  consent,  the  evaluation  team  could  approach  ISA2  solution 
users only through the action owners acting as intermediaries. 

96  See footnote Error! Bookmark not defined.Error! Bookmark not defined.. 

75 

 
                                                 
6.3.  Findings 

Based  on  data  collected  from  129  consulted  stakeholders,  extensive  desk  research,  and 
expert assessments, the interim evaluation confirms that  ISA2 performs  well in all the 
evaluation  criteria.  Nonetheless,  some  additional  measures  could  be  taken  to  improve 
the  performance  of  ISA2  and  any  future  EU  programme  dedicated  to  interoperability; 
these are presented below. 

Relevance. The original needs and problems that the programme intended to address are 
still fully relevant and the objectives of ISA2 address them. Some stakeholders, however, 
experience a number of additional needs (like a more prescriptive approach to designing 
interoperable public services) and problems (including resource constraints), which ISA2 
can  address  only  partially.  One  additional  need  that  deserves  special  attention  is  the 
request 
local 
to  ensure  more  collaboration  and  exchanges  with  regional  and 
administrations in order to increase awareness of interoperability and the take-up of ISA2 
solutions at the sub-national level. 

Effectiveness. The results  that ISA2 has  achieved so  far are aligned with  its  objectives. 
The 20 sampled actions that were evaluated have so far generated 35 solutions, some of 
which are used by all 28 EU Member States as well as by EU institutions. Nevertheless, 
the programme has not realised all the expected results yet, because most of its actions 
are still ongoing and solutions are still being developed. The programme’s duration also 
influences the take-up rate of solutions. Actions continued from previous editions of the 
programme  have  produced  solutions  that  are  now  more  widely  used  than  solutions 
resulting from actions started under ISA2. 

There is one specific ISA2 objective where the evaluation found that the programme on 
its  own  is  less  effective:  the  development  of  a  more  effective,  simplified  and  user-
friendly  e-administration  at  the  national,  regional  and  local  level.  Here  Member  States 
can  in  fact  have  a  great  impact,  and  complement  the  EU-level  initiatives  on 
interoperability and digitalisation. 

As  regards  external  factors,  they  have  a  mixed  impact  on  ISA2.  They  can  improve  but 
also jeopardise the way in which the programme achieves its objectives and delivers its 
results. 

When it comes to communication and awareness-raising activities, the evaluation results 
suggest  that  ISA2 duly acted upon the recommendation of the final  ISA evaluation  and 
increased the general awareness of the programme through its dedicated communication 
efforts. Still, some aspects of ISA2 could benefit from more promotion. 

Efficiency. The programme’s implementation is progressing as planned, all actions being 
either  on  track  or  close  to  achieving  the  planned  level  of  work.  Nevertheless,  the 
heterogeneity of performance indicators makes it difficult to draw conclusions about its 
overall  cost-effectiveness.  For  those  ISA2  packages  where  it  was  possible  to  aggregate 
performance indicators for different actions, costs per end-user (e.g. businesses, citizens) 
have been estimated as very low. 

76 

 
The  selection  process  for  actions  funded  by  ISA2  is  considered  relatively  efficient. 
Nonetheless,  it  could  be  further  improved  by  simplifying  the  rolling  work  programme 
(e.g.  making  the  template  more  flexible)  and  launching  thematic  calls  for  actions,  thus 
ensuring that the selection process is driven by objectives. 

Coherence.  Substantial  synergies  and  limited  overlaps  among  ISA2  actions  paint  a 
positive picture of the programme’s internal coherence. This strong internal coherence is 
in  line  with  the  second  recommendation  of  the  final  ISA  evaluation,  which  called  for 
taking a holistic approach to interoperability within ISA2. 

ISA2  also  followed  the  final  third  recommendation  of  the  final  ISA  evaluation,  on 
external  coherence,  and  worked  closely  with  other  EU  policies  and  initiatives.  It 
established  synergies  with  the  CEF,  Horizon  2020  and  the  Structural  Reform  Support 
Programme.  ISA2  also  interacts  with  initiatives  such  as  the  Single  Digital  Gateway, 
Digital  Single  Market  Strategy,  eGovernment  Action  Plan,  Tallinn  Declaration  on 
eGovernment, and the Rolling Plan for ICT standardisation. Nonetheless, some overlaps 
were  identified  between  specific  solutions  under  ISA2  and  the  CEF;  these  could  be 
addressed by better delineating the scope of each solution. 

EU added value. ISA2 ensures EU-level coordination, which is necessary for improving 
overall interoperability among European public administrations. Without the programme, 
the 
national  and/or  sub-national 
interoperability-specific objectives that ISA2 aims to achieve. In addition, ISA2 is able to 
achieve  these  objectives  at  a  lower  cost  than  comparable  national  or  sub-national 
initiatives. 

interventions  would  not  be  able 

to  achieve 

ISA2 also creates additional value by improving cross-border interoperability in the EU, 
as well as advancing common EU policies: it plays a central role in the implementation 
of the EIF and supports the establishment of the Digital Single Market. 

Utility. User satisfaction with ISA2 solutions tends to be positive, with only 7 out of 110 
stakeholders reporting ‘limited satisfaction’. As the programme is ongoing, it is expected 
that the take-up of  ISA2 solutions will increase, thus improving the way solutions meet 
user  needs  and,  in  turn,  overall  user  satisfaction.  However,  ISA2  could  place  more 
emphasis on providing support to the users of its existing solutions. Moreover, it could 
involve users already in the design phase of future solutions, which would increase both 
quality and user satisfaction. 

Sustainability.  The  consulted  stakeholders  have  mixed  views  on  the  sustainability  of 
ISA2 solutions. While most of them believe that the programme results would last in the 
absence  of  future  funding,  they  also  recognise  that  ISA2  solutions  may  not  be  able  to 
deliver their results if the related maintenance costs and support needs were not covered. 

Stakeholders  agree,  however,  that  ISA2  plays  a  central  role  in  improving  the 
interoperability landscape in the EU and that its absence would jeopardise the efforts of 
European  public  administrations  to  improve  interoperability  and  foster  the  ICT-based 
modernisation of the public sector in Europe. 

77 

 
6.4.  Next steps 

The  interim  evaluation  identified  several  potential  measures  that  may  improve  the 
performance  of  ISA2  and  its  successor  instruments  (see  Box  9).  It  is  up  to  the 
programme’s  management  to  assess  the  feasibility  of  these  actions  and  put  forward 
specific  recommendations,  which  the  Commission  could  then  put  into  practice  in  the 
future. 

Box 9: Overview of identified measures that would improve the programme’s 
performance 

 

increase  awareness  of  interoperability  beyond  national  administrations  by  fostering  more 
collaboration and exchanges with regional and local administrations; 

  perform studies to assess the costs and benefits of implementing ISA2 solutions and measure 

their impact on the efficiency and productivity of public administrations; 

 

 

improve the sharing of best practices among public administrations; 

rely more on potential influencers when communicating about interoperability; 

  build advisory capabilities around interoperability; 

  assess the rationale and impacts of a possible binding interoperability instrument; 

  continue  and  even  increase  communication  on  and  promotion  of  interoperability  by  

targeting the sub-national layers of administrations too; 

simplify the ISA2 rolling work programme; 

launch thematic calls for actions to ensure that the selection process is driven by objectives; 

 

 

  define some common metrics to allow for a more consistent appraisal of cost-effectiveness; 

  clarify the scope of each solution to avoid any overlaps; 

  build synergies and avoid overlaps with the Tallinn Declaration on eGovernment; 

  assess any potential obstacles to ISA2 technical specifications that may arise with regard to 

the intellectual property rights policy for CEN/CENELEC standards; 

 

increase  the  programme’s  focus  on  multilingualism  by  providing  more  information  about 
ISA2 in different official EU languages; 

  provide more support to the users of existing solutions; 

 

involve users already in the design phase of future solutions; 

  assess the feasibility and costs of various sustainability measures and invest in them; and 

 

transfer some ISA2 solutions to open source communities or encourage companies to build 
services around free ISA2 solutions under the European Union Public Licence. 

78 

 
Annex 1: Procedural information 

1.  LEAD DG, DECIDE PLANNING/CWP REFERENCES 

DG Informatics (DIGIT), PLAN/2018/2654. 

2.  ORGANISATION AND TIMING 

The  Commission  announced  this  interim  evaluation  in  an  evaluation  and  fitness  check 
roadmap97 on 29 May 2018. 

Following the invitation by the Director-General of DIGIT in May 2018, the first Inter-
service Steering Group (ISSG) meeting took place on 8 June 2018, with participants from 
DG CNECT, DIGIT, EMPL, FISMA, GROW, OP, SG, TAXUD and the JRC. The ISSG 
met five times between June 2018 and June 2019, as follows: 

Period 

Activity 

June 2018 – October 2018 

Involvement of external consultants 

November 2018 – May 2019 : 

Running the evaluation process: 

o  November 2018 – March 2019 
o  February 2019 – May 2019 

o  Data collection 
o  Data analysis and synthesis 

June 2019 

Preparation of the Commission report for inter-
service consultation and adoption 

3.  EXCEPTIONS TO THE BETTER REGULATION GUIDELINES 

Not applicable 

4.  CONSULTATION OF THE RSB (IF APPLICABLE) 

Not applicable 

5.  EVIDENCE, SOURCES AND QUALITY 

Evidence & sources 

Data have been drawn from two main sources: 

97  Please see: https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-2768206_en. 

79 

 
 
 
 
                                                 
1) Primary data were collected through the below consultation activities: 

Consultation activity 

ISA2 Mid-Term Conference short questionnaire: A short 
questionnaire based on the public consultation questionnaire 
was distributed during the ISA2 Mid-Term Conference and 
during the Kick-off workshop. 

Kick-off workshop: The workshop was to present the interim 
evaluation and the planned consultation activities to the 
different stakeholder groups of the ISA2 programme. 

Public consultation: This activity gave all relevant parties, 
primarily general public, businesses, stakeholders’ associations 
and public administrations, the opportunity to express their 
opinions and views on the achievements of the ISA2 
programme. The public consultation was available in 23 EU 
languages. 

Targeted online surveys: Five online surveys were made 
available on EUSurvey, targeting the following stakeholder 
groups: programme governance, action owners, solution users, 
stakeholders responsible for linked EU policies/initiatives, and 
standardisation organisations. 

Timeframe 

29 - 30 November 2018  

30 November 2018 

7 December 2018 – 1 March 
2019 (12 weeks) 

14 December 2018 – 23 
January 2019 (the surveys 
remained open after the initial 
deadline of 18 January 2018 to 
accommodate some last-minute 
requests from stakeholders) 

Targeted in-depth interviews: Interviews were conducted 
with stakeholders from the five groups targeted by the online 
surveys: programme governance, action owners, solution 
users, stakeholders responsible for linked EU 
policies/initiatives, and a standardisation expert. The 
interviews, based on questionnaires very similar to those used 
for the online surveys, facilitated the collection of additional 
qualitative data to complement data collected via such surveys. 

12 December 2018 – 23 
January 2019 (several 
interviews were conducted 
after the initial deadline set on 
18 January 2018 as some 
stakeholders were not available 
to be interviewed at an earlier 
time) 

2)  Secondary data were gathered by reviewing: 

The annual rolling work programmes of ISA2 
The ISA2 Dashboard 
The Europa webpages on ISA2 actions and solutions 

Monitoring and evaluation reports: 2016 (annual report), 2017 (quarterly reports and annual 
report), and 2018 (the reports for the first two quarters of 2018) 
Overview of communication activities of events organised by ISA2 between 2016 and 2018 
(including information on the location and the number of participants) and events in which ISA2 
was represented (including information on the location of the events) 
Lists of participants in ISA2 Committee meetings and in ISA2 Coordination Group meetings 
between 2016 and 2018 

Performance indicators directly shared by action owners of the sampled actions 

80 

 
 
 
 
Quality 

To support the  ISA2 interim evaluation,  in  September 2018 the Commission  tasked the 
Centre  for  European  Policy  Studies  (CEPS)  with  delivering  an  independent  evaluation 
study  by  May  2019.  The  ISSG  members  closely  monitored  the  work  of  CEPS.  They 
discussed the consultant’s various deliverables at the ISSG meetings and also commented 
on the documents in writing. The ISSG accepted CEPS final evaluation study by written 
procedure  on  24  May  2019.  This  final  ‘sign  off’  by  the  ISSG  also  built  on  the  quality 
assessment results, which concluded that the external contractor's study and work met the 
required quality standards. 

81 

 
 
Annex 2: Synopsis report of the consultation activities 

Introduction 

In line with the consultation strategy98, consultation activities were carried out to collect 
stakeholders’  views  and  opinions,  thus  contributing  to  the  evidence  base  needed  to 
achieve the two main objectives of the interim evaluation of the ISA2 programme: 

  Evaluating the implementation of the programme by considering seven evaluation 

criteria: 

o  Relevance  ‒  the  alignment  between  the  programme’s  objectives  and  the 

current needs and problems experienced by stakeholders; 

o  Effectiveness  ‒  the  extent  to  which  the  programme  has  achieved  its 

objectives; 

o  Efficiency ‒ the extent to which the programme’s objectives are achieved 

at a minimum cost; 

o  Coherence ‒ the alignment between the programme and comparable EU 

initiatives as well as the overall EU policy framework; 

o  EU added value - the additional impacts generated by the programme, as 
opposed to leaving the subject matter in the hands of Member States; 

o  Utility  ‒  the  extent  to  which  the  programme  meets  stakeholders’  needs; 

and 

o  Sustainability  ‒  the  likelihood  that  the  programme’s  results  will  last 

beyond its completion. 

  Identifying  areas  of potential  improvement  and  devising  recommendations  to 
its 

the  programme’s  performance  and  bring  more  value  to 

improve 
stakeholders. 

Feedback received on the evaluation roadmap 

Before  the  interim  evaluation  was  launched,  one  stakeholder  provided  feedback  on  the 
ISA2 evaluation roadmap93, noting that the ‘initiative is good, well planned, and funded’, 
but also that ‘it could have benefitted from more advertising’99. This input was received 
in June 2018. 

98  Please see: https://ec.europa.eu/isa2/sites/isa/files/consultation_strategy_isa2_interim_evaluation.pdf. 
99  Please see: https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-

2768206/feedback_en?p_id=239250. 

82 

 
 
 
                                                 
Types of consultation activities conducted 

A mix of consultation activities was organised during the evaluation process to reach a 
variety of the programme's stakeholders. The following seven stakeholder groups were 
targeted: 

  Programme  governance:  Commission 

from  DIGIT’s 
Interoperability  unit  and  members  of  the  ISA2  Committee  and  Coordination 
Group; 

representatives 

  Action owners: Commission representatives in charge of specific actions defined 

under ISA2; 

  Solution  users:  Commission  and  Member  State  representatives  who  are 

using/reusing ISA2 solutions; 

  Stakeholders  responsible  for  linked  EU  policies/initiatives:  Commission 
representatives who are  in  charge of EU initiatives linked to  ISA2, and who are 
thus  in  a  position  to  provide  feedback,  on  the  external  coherence  of  the 
programme among other things; 

  Experts: experts in IT and eGovernment able to perform an expert assessment of 
the programme and of the issues and developments in the field of interoperability; 

  Standardisation organisations: Representatives of standardisation organisations 
who  can  provide  feedback  on  ISA2  activities  related  to  standards  and  technical 
specifications; and 

  Indirect beneficiaries and the wider public: Other programme beneficiaries or 
parties  who  are  involved  in  or  affected  by  the  ISA2  programme  in  any  way, 
including the private sector and the general public. 

Between 29 November 2018 and 1 March 2019, five consultation activities were carried 
out in accordance with the Commission’s better regulation requirements: 

Targeted online surveys (14.12.2018 – 23.1.2019) 

Five  online  surveys  were  made  available  on  EUSurvey,  targeting  the  following 
stakeholder groups: programme governance, action owners, solution users, stakeholders 
responsible for linked EU policies/initiatives, and standardisation organisations.  

The online surveys were initially set to close on 18 January 2019, but the deadline was 
extended  to  23  January  in  order  to  accommodate  some  last-minute  participants. 
Altogether 80 responses were collected through these surveys. 

Targeted in-depth interviews (12.12.2018 – 23.1.2019) 

Interviews  were  held  with  the  same  five  stakeholder  groups  as  the  online  surveys: 
programme  governance,  action  owners,  solution  users,  stakeholders  responsible  for 
linked EU policies/initiatives, and a standardisation expert. The interviews were based on 

83 

 
questionnaires that were very similar to  those used for the online surveys. They helped 
collect additional qualitative data from 20 stakeholders complementing the data collected 
from the surveys.  

The in-depth interviews were initially supposed to run until 18 January 2019. However, 
as some stakeholders were not available in the original time-frame interviews continued 
until 23 January 2019. 

Public consultation (7.12.2018 – 1.3.2019) 

This  activity  gave  all  relevant  parties  —  mainly  the  public,  businesses,  stakeholders' 
associations  and  public  administrations,  the  opportunity  to  express  their  views  on  the 
ISA2 programme's achievements. Experts could participate in the public consultation and 
state their level of expertise in the programme and in the interoperability of digital public 
services. The public consultation100 was available on EUSurvey in 23 EU languages and 
yielded  14  replies.  This  low  response  rate  is  due  to  two  main  reasons.  First,  ISA2  is  a 
technical  programme  producing  solutions  addressed  mainly  to  European  public 
administrations; therefore, the direct contact between the average citizen/business and the 
programme  is  limited.  Second,  stakeholder  groups  that  are  in  direct  contact  with  the 
programme  (e.g.  solution  users,  action  owners,  etc.)  were  consulted  via  targeted  online 
surveys and interviews. 

ISA2 Mid-Term Conference short questionnaire (29 – 30.11.2018) 

A  short  questionnaire  based  on  the  public  consultation  questionnaire  was  distributed 
during the  ISA2 Mid-Term Conference (29 November 2018)101 and during the Kick-off 
workshop  (30  November  2018)  of  the  interim  evaluation.  15  stakeholders  shared  their 
views on the programme through this channel. 

Kick-off workshop  

The purpose of the kick-off workshop of 30 November 2018 was to present the interim 
evaluation  and the planned consultation  activities to  the  ISA2 programme’s stakeholder 
groups102. There were 40 participants in the workshop. 

Methodology and tools used to process data 

The consultation questionnaires were mainly based on ‘Likert scale’ questions, meaning 
that respondents were asked to give their opinion on a scale from (1) to (5) or from (1) to 
(4), depending on the type of question: 

  1 - not at all; 2 - to a limited extent; 3 - to some extent; 4 - to a high extent; 5 - to 

the fullest extent; or 

100  Please see: https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-2768206/public-

consultation_en. 

101  Please see: https://ec.europa.eu/isa2/isa2conf18_en. 
102  Please see: https://ec.europa.eu/isa2/events/isa%C2%B2-interim-evaluation-kick-workshop_en. 

84 

 
                                                 
  1  –  definitely  would  not;  2  –  probably  would  not;  3  –  probably  would;  4  – 

definitely would. 

This approach helped the comparison of answers between respondents. In addition to the 
Likert scale questions, a number of open-ended questions were included to gather more 
in-depth  information  from  consulted  stakeholders.  In  addition,  the  interviews  allowed 
more detailed feedback to be collected.  

Characteristics of respondents 

With a total of 129 respondents, the various consultations reached all of the following 
groups: programme governance, action owners, solution  users, stakeholders responsible 
for  linked  EU  policies  /  initiatives,  standardisation  organisations  and  the  wider  public 
(Table  5).  In  addition,  a  team  of  technical  experts  assessed  the  programme  and  their 
conclusions are presented throughout the evaluation report. 

Note  that  the  results  of  the  public  and  targeted  consultation  activities  have  been 
combined and analysed jointly in this report. 

Table 5: Number of consulted stakeholders by consultation activity and stakeholder 
category 

Stakeholder 
category 

In-depth 
interview 

Online 
survey 

Public 
consultation 

Programme 
Governance 
Action owners 
Solution users  
Stakeholders 
related to linked 
EU policies/ 
programmes/ 
initiatives 
Standardisation 
organisations 
Wider public 
TOTAL 

4 

3 
6 
6 

1 

- 
20 

15 

20 
37 
4 

4 

- 
80 

- 

- 
- 
- 

- 

14 
14 

Short 
question-
naire 
- 

- 
- 
- 

- 

15 
15 

TOTAL 

19 

23 
43 
10 

5 

1 
129 

Note:  Regarding  the  ‘Expert’  stakeholder  group,  four  technical  experts  independently  assessed  the 
programme.  In  addition,  most  of  the  respondents  from  the  consulted  stakeholder  groups  are  experts  in 
interoperability and ISA2, as shown in Figure 30 and Figure 31. 
Source: CEPS final study 

85 

 
 
 
The  consulted  stakeholders  claimed  to  have  high  levels  of  expertise  both  on  the 
interoperability  of  public  services  (Figure  30)  and  on  the  ISA2  programme  as  a  whole 
(Figure 31). 

Figure 30 Knowledge of digital public services and interoperability (average score 
of answers, number of respondents) 

Action owners

Programme governance

Solution users

Stakeholders responsible for linked EU
policies initiatives

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Standardisation organisations

[CELLRANGE] 

Wider public

[CELLRANGE] 

0,00

1,00

2,00

3,00

4,00

5,00

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).  
Source: CEPS final study 

Figure 31 Knowledge of ISA2 (average score of answers, number of respondents) 

Action owners

Programme governance

[CELLRANGE] 

[CELLRANGE] 

Solution users

[CELLRANGE] 

Stakeholders responsible for linked EU policies
initiatives

[CELLRANGE] 

Standardisation organisations

[CELLRANGE] 

Wider public

[CELLRANGE] 

0

1

2

3

4

5

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).  
Source: CEPS final study 

One slight exception is solution users who reported a relatively more limited knowledge 
of  the  entire  programme  compared  to  the  other  stakeholder  groups  consulted.  Even  so, 
most respondents are familiar with at least 5 packages of the 9 ISA2 packages of actions 
(Figure 32). 

86 

 
 
 
Figure 32 Knowledge of ISA2 action packages (number of respondents familiar with 
a given number of packages)103  

32 

21 

9 packages

8 packages

7 packages

6 packages

5 packages

4 packages

6 

3 packages

2 packages

1 package

8 

11 

12 

12 

8 

10 

0

5

10

15

20

25

30

35

Source: CEPS final study 

Regarding  geographical  distribution,  solution  users  from  various  Member  States 
provided input in the in-depth interviews and the online surveys: 

  At  EU  level,  solution  users  who  participated  in  the  consultations  work  in  EU 
institutions  located  in  a  number  of  EU  countries:  Belgium,  Czechia,  France, 
Hungary, Italy, Luxembourg, Poland, Portugal, and Sweden; 

  At  national  and  regional  levels,  solution  users  (including  respondents  who 
answered  as  individuals)  who  provided  feedback  come  from  the  following  13 
countries:  Belgium,  Bulgaria,  Croatia,  Czechia,  France,  Germany,  Greece, 
Hungary, Italy, Luxembourg, the Netherlands, Portugal, and Sweden. 

Feedback on the public consultation was provided by individuals in six Member States 
(Belgium,  France,  Germany,  Greece,  Spain,  and  the  United  Kingdom)  and  by  public 
administrations from three Member States (Czechia, Greece, and Spain). 

103   Stakeholders were considered to be familiar with a package if they indicated that they had knowledge 

of the package to some extent, to a high extent or to the fullest extent. 

87 

 
 
 
 
                                                 
Results 

The  consultation  activities  revealed  a  general  consensus  among  the  different 
stakeholder groups consulted and confirmed that ISA2 performs well in all the seven 
evaluation criteria under investigation. The feedback from the consultation activities is 
summarised as follows:  

Relevance 

The  respondents  to  the  consultation  activities  emphasised  that  ISA2  is  fully  relevant 
with respect to the evolving needs and problems in the field of interoperability. 

The  majority  of  respondents  indicate  that:  i)  the  needs  and  problems  originally 
addressed by the programme are still experienced by European public administrations, 
businesses and/or the wider public;  and ii) by achieving its general  objective,  ISA2 can 
address the needs and problems identified when the programme was established. 

Respondents  pointed  out  that  there  are  other  needs  and  problems  currently 
experienced by stakeholders in the field, namely: 

 

 

 

 

 

 

the  need  for  a  more  binding  legal  framework  for  interoperability  and  a  more 
prescriptive approach to designing interoperable public services; 

the need to improve the way administrations communicate with one another; 

the need to share best practices; 

the resource constraints experienced by national and local public administrations; 

the  different  political  priorities  of  Member  States  that  prevent  a  consistent 
approach to interoperability in the EU; and 

the  limited  awareness  of  ISA2  and  other  interoperability-related  initiatives, 
especially at regional and local level. 

In general, respondents to the consultation activities believe that the programme, through 
its general objective, the programme could  address these new needs and problems to 
some extent, as some of the issues mentioned fall outside the scope of the programme. 

Effectiveness 

The  results  achieved  so  far  by  ISA2  are  aligned  with  the  programme’s  objectives 
according to the consulted stakeholders, in particular when it comes to: 

  developing, maintaining and promoting a holistic approach to interoperability in 

the EU; and 

 

facilitating 
administrations. 

the  reuse  of 

interoperability  solutions  by  European  public 

88 

 
However, the results to date still fall short of the expected results, as most of the actions 
are  ongoing  and  solutions  are  still  being  developed.  For  instance,  respondents 
emphasised that relatively more could be done to achieve the expected result of improved 
efficiency and productivity of the European public administrations. 

External  factors  could  improve  but  also  jeopardise  the  way  in  which  the  programme 
achieves  its  objectives  and  delivers  its  results.  The  vast  majority  of  respondents 
confirmed  that  the  calls  for  common  standards  and  frameworks  from  public 
administrations  are  important  external  factors  contributing  to  the  performance  of  the 
programme.  In  contrast,  institutional  complexity  in  the  form  of  multiple  layers  of 
governance  and  differences  between  national  public  administrations  from  various 
Member  States  and  local  public  administrations  within  Member  States  could  lead  to 
coordination issues that hamper cross-border and cross-sectoral interoperability. 

Efficiency 

Respondents  from  the  programme  governance  and  action  owners  gave  feedback  on  the 
efficiency of the process for selecting actions funded by ISA2. The process is considered 
relatively  efficient  and  fit-for-purpose  by  the  respondents  from  these  two  stakeholder 
groups.  However,  some  measures  could  be  taken  to  further  streamline  the  selection 
process, such as: 

  simplifying the rolling work programme; and 

  simplifying the overall process by launching thematic calls for actions. 

Respondents  from  the same two groups  also  provided information  about  the amount of 
time they spent on preparing and submitting a proposal for an action to be considered 
by  the  ISA2  programme.  On  average,  six  person-days  are  required  to  renew  a  proposal 
for actions already included in previous rolling work programme. In contrast, between 10 
and 30 person-days are required to prepare and submit a proposal for a new action.  

Coherence 

The  consultation  activities  focused  on  both  the  internal  coherence  of  the  programme, 
i.e.  the  synergies  or  overlaps  existing  between  the  ISA2  actions,  and  its  external 
coherence,  namely  the  synergies  or  overlaps  between  ISA2  and  other  EU  initiatives, 
policies, or programmes. 

Internal coherence 

ISA2  actions  are  characterised  by  substantial  synergies  with  each  other  and  limited 
overlaps. During the consultation activities, respondents provided examples of synergies 
to support their assessment. Examples mentioned include: 

  solutions  developed under the ‘SEMIC’  actions  are reused by the ‘Catalogue of 

Services’ and the ‘Access to Base Registries’ actions; 

89 

 
 

the ‘Re3gistry’ solution, part of the ‘ELISE’ action, is used to share the codes for 
the ‘Core Vocabularies’, part of the ‘SEMIC’ action; and 

  The  solutions  developed  as  part  of  the  ‘e-TrustEx’  actions  are  reused  by  the 

action ‘European Public Procurement Interoperability Initiative’. 

External coherence 

On external coherence, respondents to the consultation activities identified most synergy 
between  ISA2  and  CEF.  In  contrast,  of  the  EU  programmes  interacting  with  ISA2,  the 
ESF  tends  to  have  the  least  synergies.  Respondents  indicated  that  Horizon  2020  (in 
particular the work related to the once-only principle) and the SRSP (through the support 
given to Member States to align and integrate with cross-border services) also have some 
synergies with ISA2. 

Respondents  also  pointed  at  synergies  and  overlaps  with  other  EU  initiatives  such  as 
ERDF, Corporate IT Governance and the Single Digital Gateway. 

EU added value 

Most respondents believe that national or sub-national interventions could not achieve or 
could only slightly achieve the general and specific interoperability objectives that ISA2 
is pursuing. The consulted stakeholders consider that  ISA2 plays an important role in 
ensuring  a  level  of  coordination  that  improves  interoperability  among  European 
public administrations. 

Respondents also emphasised that ISA2 can achieve its objectives at a lower cost than 
comparable  national  or  sub-national  initiatives.  Most  also  believe  that  ISA2  has  helped 
improve  cross-border  interoperability  in  the  EU.  Examples  frequently  mentioned  by 
stakeholders are the work done in the field of semantic interoperability and the fact that 
ISA2  raises  awareness  about  interoperability  in  the  EU,  without  which  little  progress 
would be made in the field.  

When  it  comes  to  advancing  common  EU  policies,  consulted  stakeholders  noted  that 
ISA2  plays  a  central  role  in  implementing  the  EIF  (including  its  2017  revision)  by 
abiding by its principles and tracking its implementation.  Stakeholders emphasised that 
ISA2  has  also  contributed  to  specific  areas  of  broader  EU  policy,  namely  the  Digital 
Single Market. 

Utility 

Respondents  in  both  EU-level  and  national  public  administrations  confirm  that  the 
solutions developed or maintained by ISA2 have contributed to addressing the original 
needs  and  problems  identified  in  the  field  of  interoperability.  Respondents 
emphasised that while some of the new needs and problems experienced in the field are 
addressed by the solutions provided by ISA2, others, such as changing political priorities 
in the Member States, fall beyond the scope of the programme as it is currently defined.  

90 

 
The  feedback  received  from  respondents  in  terms  of  user  satisfaction  tends  to  be 
positive. Nevertheless, a number of measures were identified to increase the utility of the 
programme, namely: 

  placing more emphasis on sharing best practices and providing support to users; 

  establishing a co-creation process with users;  

 

improving the quality of existing solutions by better considering user needs;  

  ensuring the Member States’ commitment to using ISA2 solutions; and 

  strengthening  the promotion of  ISA2 solutions  at  Member State level  as  well as 
standards  development 

specific  groups  of  professionals 

(e.g. 

among 
organisations). 

Sustainability 

Overall,  consulted  stakeholders  have  a  positive  view  of  the  sustainability  of  results 
achieved so far. Nevertheless, while the programme’s global results are expected to last 
(as the progress made in the field of interoperability will not be lost), certain issues may 
arise when considering the specific aspects of different solutions and future steps towards 
increased  interoperability.  In  fact,  there  are  solutions  requiring  both  operations  and 
maintenance  costs  as  well  as  technical  and  operational  support,  which  would  be 
unable to deliver further results if the programme terminated. Additional obstacles to the 
sustainability of the programme include: 

 

 

 

lack of development for existing solutions; 

lack of coordination between European public administrations; and 

limited dissemination and communication related to interoperability. 

Furthermore,  introducing  a  fee  for  users  to  gain  access  to  ISA2  solutions  is  not 
considered  a  feasible  solution,  especially  in  the  event  that  funding  would  no  longer  be 
provided  for  the  actions  currently  carried  out  by  ISA2.  Finally,  the  vast  majority  of 
respondents, from all stakeholder groups, stressed that the ICT-based modernisation of 
the  public  sector  in  Europe  and  the  possibility  to  address  the  needs  of  businesses 
and  the  public  via  improved  interoperability  of  European  public  administrations 
would be jeopardised, if ISA2 were terminated without any similar EU programme being 
established in its place. 

91 

 
Figures presenting the results of the consultation activities per evaluation criterion 

Relevance 

Extent to which needs and problems originally addressed by ISA2 are currently 
experienced by European public administrations, businesses and/or citizens 
(average score of answers, number of respondents) 

Need 1: The need for cooperation among public
administrations with the aim to enable more
efficient and secure public services

Need 2: The need for exchanging information
among public administrations to fulfil legal
requirements or political commitments

Need 3: The need for sharing and reusing
information among public administrations to
increase administrative efficiency and cut red tape
for citizens and businesses

Problem: Administrative e-barriers are leading to
the fragmentation of the Internal Market

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

Action owners

Solution users

0

1

2

3

4

5

Programme governance

Stakeholders responsible for linked EU initiatives

Standardisation organisations

Wider public

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO). 
Source: CEPS final study 

92 

 
 
 
 
Extent to which achieving ISA2 objectives contributes to addressing the needs and 
problems originally addressed by the programme (average score of answers, 
number of respondents) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO). 
Source: CEPS final study 

Extent to which achieving ISA2 objectives contributes to addressing additional 
(current) needs and problems identified by consulted stakeholders (number of 
respondents by stakeholder category) 

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest 
extent; (DK/NO) do not know/no opinion. 
Source: CEPS final study 

93 

 
 
 
 
 
Effectiveness 

Extent to which ISA2 solutions contributed to achieving the programme’s objectives 
(average score of answers, number of respondents) 

General objective: To promote the ICT-based
modernisation of the public sector in Europe and
to facilitate addressing the needs of businesses
and citizens via improved interoperability of
public administrations, contributing to the DSM…

Specific objective 1: To develop, maintain and
promote a holistic approach to interoperability in
the Union in order to eliminate fragmentation in
the interoperability landscape in the Union

Specific objective 2: To facilitate efficient and
effective electronic cross-border or cross-sector
interaction between European public
administrations, and between European public
administrations and businesses and citizens

Specific objective 3: To contribute to the
development of a more effective, simplified and
user-friendly e-administration at the national,
regional and local levels of public administration

Specific objective 4: To identify, create and
operate interoperability solutions supporting the
implementation of Union policies and activities

Specific objective 5: To facilitate the reuse of
interoperability solutions by European public
administrations

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

Standardisation organisations Wider public

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 

94 

 
 
Source: CEPS final study Extent to which ISA2 solutions contributed to achieving the 
programme’s operational objectives (average score of answers, number of 
respondents) 

1: To support and promote the assessment,
improvement, operation and re-use of
existing cross-border or cross-sector
interoperability solutions and common…

2: To support and promote the
development, establishment, bringing to
maturity, operation and re-use of new cross-
border or cross-sector interoperability…

3: To support and promote the assessment
of the ICT implications of proposed or
adopted Union law

4: To support and promote the identification
of legislation gaps, at Union and national
level, that hamper cross-border or cross-
sector interoperability between European…

6: To support and promote the mapping and
analysis of the interoperability landscape in
the EU through the establishment,
maintenance and improvement of the EIRA…

7: To support and promote the maintenance,
updating, promotion and monitoring of the
implementation of the EIS, the EIF and the
EIRA

8: To support and promote the assessment,
updating, promotion of existing/new
common specifications through the EU's
standardisation platforms and in…

9: To support and promote the maintenance
and publication of a platform allowing access
to and collaboration on best practices,
functioning as a means of raising…

10: To support and promote the bringing of
new interoperability services and tools to
maturity, and maintaining and operating
existing interoperability services and tools…

11: To support and promote the
identification of best practices, to develop
guidelines to coordinate interoperability
initiatives and to animate and support…

12: To develop a communication strategy
aiming to increase awareness with regard to
ISA² and its benefits, targeting businesses
and citizens, and employing user-friendly…

Action owners

0

2
Programme governance

1

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

3

4
Solution users

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

95 

 
 
 
Extent to which ISA2 solutions achieved the programme's expected results (average 
score of answers, number of respondents) 

A coherent interoperability landscape in the Union
based on a holistic approach to interoperability

Advancement of Union policies and activities by
supporting their implementation

Contribution to the successful achievement of the
Digital Single Market

Efficient and effective electronic cross-border or
cross-sectoral interactions between European
Public Administrations as well as between
European public administrations and…

Improved efficiency and productivity of the
European public administrations, which can be a
strong driver of economic growth through its
support for, and governance of, the private sector

Increased speed, efficiency and quality in the
creation and delivery of cross-border and cross-
sector electronic public services to meet the needs
of businesses and citizens

More effective, simplified and user-friendly e-
administrations in European public administrations

Reduction of the cost and administrative burden
of cross-border interaction, removal of
administrative e-barriers and contribution to the
swift implementation of ICT systems supporting…

Take-up and re-use of the ISA² programme’s 
results by European public administrations 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

0
Solution users

1

2

3

4

5

Standardisation organisations

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent.  
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

96 

 
 
 
Extent to which the following external factors contribute to the performance of ISA2 
(average score of answers, number of respondents) 

Calls for common standards and frameworks from
industry

Calls for common standards and frameworks from
public administrations

National initiatives related to the ICT modernisation
of the public sector

The need for European public administrations to
increase their efficiency due to budget constraints

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Extent to which the following external factors jeopardise the performance of ISA2 
(average score of answers, number of respondents) 

Institutional complexity

Legal complexity

Limited resources of European public
administrations

Technical challenges (e.g. updating existing legacy
applications)

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance

Solution users

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

97 

 
 
 
 
 
Efficiency 

Extent to which the action selection process is fit-for-purpose (number of 
respondents by stakeholder category) 

12

10

8

6

4

2

0

Action owners

Programme governance

Not at all

To a
limited
extent

To some
extent

To a high
extent

To the
fullest
extent

Do not
know/No
opinion

Source: CEPS final study 

Distribution of the number of days needed to prepare and submit a proposal for an 
ISA2 action (number of respondents) 

y
c
n
e
u
q
e
r
F

6

5

4

3

2

1

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

Number of days 

Note: The histogram was constructed using a bin size of 2 days, which grouped the answers of respondents 
in intervals of 2 days. This bin size was deemed most appropriate after reviewing the answers provided. 
Source: CEPS final study 

98 

 
 
 
 
 
 
 
Earned value, actual costs and planned value sampled actions, by package 
(Q3 2018)104 

Source: CEPS final study 

Schedule performance indices of the sampled actions grouped by packages 

2,4

2,2

2

1,8

1,6

1,4

1,2

1

0,8

0,6

0,4

1. Key and generic interoperability enablers

2. Semantic Interoperability

3. Access to data/data sharing/open data

4. Geospatial Solutions

104  The  PV  and  AC  are  equal  in  all  cases.  As  explained  on  the  ISA2  Dashboard,  most  of  the  specific 
contracts under the ISA2 actions are on fixed price, therefore it is not relevant to keep track of the AC 
before the end of a Work Programme year. Taking this into account, the AC is then considered equal 
to  the  PV.  For  more  details  please  see:  ISA2  Dashboard  —  Frequently  Asked  Questions, 
https://ec.europa.eu/isa2/dashboard/faq-page. 

99 

 
 
 
 
                                                 
1,8

1,6

1,4

1,2

1

0,8

0,6

5. e-Procurement/e-invoicing-Supporting instruments

6. Decision making and legislation-Supporting instruments

7.  EU  Policies-Supporting instruments

8.  Supporting instruments for public administrations

9. Accompanying measures

Source: CEPS final study 

100 

 
 
 
 
Coherence 

Extent to which there are synergies / overlaps between ISA2 actions (average score 
of answers, number of respondents) 

Overlaps

Synergies

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance

Solution users

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

101 

 
 
 
Extent to which there are synergies / overlaps between ISA2 and other relevant EU 
programmes (average score of answers, number of respondents) 

e
p
o
r
u
E

g
n
i
t
c
e
n
n
o
C

t
r
o
p
p
u
S
m
r
o
f
e
R

l

a
r
u
t
c
u
r
t
S

y
t
i
l
i
c
a
F

d
n
u
F

l

a
i
c
o
S
n
a
e
p
o
r
u
E

0
2
0
2

n
o
z
i
r
o
H

e
m
m
a
r
g
o
r
P

r
e
h
t
O

Overlaps

Synergies

Overlaps

Synergies

Overlaps

Synergies

Overlaps

Synergies

Overlaps

Synergies

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners
Programme governance
Solution users
Stakeholders responsible for linked EU policies initiatives
Wider public

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

102 

 
 
 
 
 
 
 
 
 
EU added value 

Extent to which national or sub-national interventions would be able to achieve the 
ISA2 objectives without the programme (average score of answers, number of 
respondents) 

General objective: To promote the ICT-based
modernisation of the public sector in Europe and to
facilitate addressing the needs of businesses and citizens
via improved interoperability of public administrations,
contributing to the DSM and to growth

Specific objective 1: To develop, maintain and promote a
holistic approach to interoperability in the Union in order
to eliminate fragmentation in the interoperability
landscape in the Union

Specific objective 2: To facilitate efficient and effective
electronic cross-border or cross-sector interaction
between European public administrations, and between
European public administrations and businesses and
citizens

Specific objective 3: To contribute to the development of
a more effective, simplified and user-friendly e-
administration at the national, regional and local levels of
public administration

Specific objective 4: To identify, create and operate
interoperability solutions supporting the implementation
of Union policies and activities

Specific objective 5: To facilitate the reuse of
interoperability solutions by European public
administrations

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 
[CELLRANGE] 
[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance

Solution users

Stakeholders responsible for linked EU policies initiatives

Wider public

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

103 

 
 
 
Ability of ISA2 to achieve the relevant objectives at a lower cost than comparable 
national or sub-national interventions (average score of answers, number of 
respondents) 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

[CELLRANGE] 

Solution users

[CELLRANGE] 

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

1

2

3

4

[CELLRANGE] 

Standardisation organisations

Wider public

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Extent to which ISA2 contributed to improving cross-border interoperability in the 
EU (average score of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

[CELLRANGE] 

Solution users

[CELLRANGE] 

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

Standardisation organisations

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

104 

 
 
 
 
 
Extent to which ISA2 contributed to the advancement of common EU policies 
(average score of answers, number of respondents) 

[CELLRANGE] 

Action owners

[CELLRANGE] 

Programme governance

[CELLRANGE] 

Solution users

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

105 

 
 
 
 
 
Utility 

Extent to which ISA2 solutions contributed to addressing the needs and problems 
originally covered by the programme: EU-level respondents (average score of 
answers, number of respondents)

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance (from
DIGIT)

Solution users (from EC)

Stakeholders responsible for
linked EU policies initiatives (with
the exception of CIOs)

Standardisation organisations

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Extent to which ISA2 solutions contributed to addressing the needs and problem 
originally covered by the programme: respondents from the Member States 
(average score of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Programme governance
(Committee members and
Coordination group members)

Solution users (except from EC)

Stakeholders responsible for
linked EU policies initiatives
(CIOs)

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

106 

 
 
 
 
 
Extent to which ISA2 solutions contributed to addressing additional needs and 
problems identified by consulted stakeholders: EU-level respondents (average score 
of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance (from
DIGIT)

Solution users (from EC)

Stakeholders responsible for
linked EU policies initiatives (with
the exception of CIOs)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Extent to which ISA2 solutions contributed to addressing additional needs and 
problems identified by consulted stakeholders: respondents from the Member 
States (average score of answers, number of respondents)

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Programme governance
(Committee members and
Coordination group members)

Solution users (except from EC)

Stakeholders responsible for
linked EU policies initiatives
(CIOs)

0

1

2

3

4

5

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

107 

 
 
 
 
Extent to which ISA2 solutions are meeting user needs in the fields of cross-border 
and cross-sectoral interoperability of digital public services (average score of 
answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

Stakeholders responsible for
linked EU policies initiatives
Standardisation organisations

0

1

2

3

4

5

Wider public

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

108 

 
 
 
 
 
Sustainability 

Likelihood that results achieved so far would last if funding for actions covered by 
the programme would not be available in the future (average score of answers, 
number of respondents) 

Action owners

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Programme governance

Solution users

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

0

1

2

3

4

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Extent to which financial, technical and operational aspects would prevent ISA2 
solutions from delivering their results if the programme was terminated and not 
replaced (average score of answers, number of respondents) 

d
n
a

n
o
i
t
a
r
e
p
O

e
c
n
a
n
e
t
n
a
m

i

e
b

o
t

s
t
s
o
c

d
e
r
e
v
o
c

d
n
a

l

a
c
i
n
h
c
e
T

l

a
n
o
i
t
a
r
e
p
o

d
e
d
e
e
n

t
r
o
p
p
u
s

r
e
h
t
O

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

0

1

2

3

4

5

Action owners

Programme governance

Solution users

Stakeholders responsible for linked EU policies initiatives

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest 
extent. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

109 

 
 
 
 
 
 
 
 
 
Likelihood that the ISA2 general objective would be jeopardised if the programme 
was terminated and not replaced (average score of answers, number of respondents) 

[CELLRANGE] 

Action owners

[CELLRANGE] 

Programme governance

[CELLRANGE] 

Solution users

[CELLRANGE] 

Stakeholders responsible for
linked EU policies initiatives

1

2

3

4

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

Likelihood that users would agree to pay a fee to continue accessing ISA2 solutions 
(average score of answers, number of respondents) 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

[CELLRANGE] 

Action owners

Programme governance

Solution users

Stakeholders responsible for
linked EU policies initiatives

1

2

3

4

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would. 
Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO). 
Source: CEPS final study 

110 

 
 
 
 
 
Annex 3: Intervention Logic 

This  Annex  presents  the  ISA2  programme’s  intervention  logic,  which  comprises  the 
rationale, the inputs, and the expected results of the intervention. This step aims to clarify 
the logic followed by EU decision-makers when establishing the ISA2 programme. The 
intervention  logic  includes  a  detailed  description  of  the  needs,  problems  and  drivers 
that the programme intended to address, the programme’s objectives (with a three-level 
hierarchy: global, specific and operational objectives), its inputs/activities, its expected 
outputs, outcomes and impacts and the logical links between these components.  

1.  THE RATIONALE FOR THE INTERVENTION: ‘WHY DID THE EU ESTABLISH THE ISA² 

PROGRAMME?’ 

The intervention logic starts by understanding the rationale of the ISA² programme. At 
this stage, the reasons for its establishment were identified by (i) assessing the needs and 
problems that it aims to address; and (ii) outlining EU decision-makers’ main objectives. 

Needs and problems105 

As highlighted by the Digital Agenda for Europe106, interoperability plays a  key role in 
maximising  the  social  and  economic  potential  of  ICT.  Against  this  background,  three 
specific needs can be identified in the field of interoperability107:  

 

 

 

the  need  for  cooperation  among  public  administrations  to  enable  more  efficient 
and secure public services; 

the need for exchanging information among public administrations to fulfil legal 
requirements or political commitments; and 

the  need  for  sharing  and  reusing  information  among  public  administrations  to 
increase administrative efficiency and cut red tape for businesses and the public.  

The  Digital  Single  Market  (DMS)  Strategy108  confirmed  these  needs  later  on  by 
recognising  interoperability  as  a  prerequisite  for  ‘efficient  connections  across  borders, 
between communities and between public services and authorities’.  

105  Needs refer to prerequisites for the efficient delivery of European public services, more specifically, in 
what concerns the interoperability dimension. Problems consist of specific bottlenecks that hinder the 
realisation of the needs. The drivers are the underlying causes that lead to the identified problem. 
106  European Commission (2010), Communication from the Commission to the European Parliament, the 
Council, the European Economic and Social Committee and the Committee of the Regions, ‘A Digital 
Agenda for Europe’, COM(2010)245 final. 

107  European Commission (2010), Communication from the Commission to the European Parliament, the 
Council, the European Economic and Social Committee and the Committee of the Regions, ‘Towards 
interoperability  for  European  public  services’  and  ‘Annex  II  –  EIF  (European  Interoperability 
Framework)’, COM(2010) 744 final. 

111 

 
                                                 
When  it  comes  to  problems,  the  ISA2  proposal109  outlines  one  main  issue,  namely  the 
existence of administrative e-barriers leading to a fragmented market. 

This  particular  problem  is  affected  by  the  following  eight  drivers  (as  pointed  out  in  a 
series  of  documents  including  the  ISA2  proposal,  the  Final  Evaluation  of  the  ISA 
programme110,  the  Commission  Staff  Working  Document  accompanying  the  DSM 
Strategy111): 

 

 

 

 

 

 

 

limited  governance  and  coordination  among  Commission  DGs  and  between 
Member  States  and  the  Commission  on  programmes  and  initiatives  related  to 
interoperability; 

limited cooperation among dispersed institutional entities; 

resource constraints in relation to interoperability; 

internal organisational complexity; 

limited ICT architectures and tools enabling interoperability; 

limited availability of common frameworks, guidelines, and specifications; 

limited awareness of the benefits of interoperability; and 

  cultural fragmentation.112 

Objectives 

The  ISA2  objectives  are  presented  in  a  hierarchical  order,  where  the  achievement  of 
lower level objectives is normally a pre-condition for attaining the higher-level ones.  

In this respect, the following three levels of objectives have been identified: 

108  European Commission (2015), Communication from the Commission to the European Parliament, the 
Council, the European Economic and Social Committee and the Committee of the Regions ‘A Digital 
Single Market Strategy for Europe’, COM(2015) 192 final. 

109  See  Section  1.2  of  the  ISA2  Proposal;  Proposal  for  a  DECISION  OF  THE  EUROPEAN 
PARLIAMENT AND OF THE COUNCIL establishing a programme on interoperability solutions for 
European  public  administrations,  businesses  and  citizens  (ISA2)  Interoperability  as  a  means  for 
modernising the public sector, COM(2014) 367 final. 

110  Kurt Salmon, KPMG (2015), Final evaluation of the ISA programme, Final report v2.06, 1 December 

2015. 

111  European  Commission  (2015),  Commission  Staff  Working  Document  ‘A  Digital  Single  Market 
Strategy for Europe – Analysis and Evidence’ Accompanying the document Communication from the 
Commission to the European Parliament, the Council, the European Economic and Social Committee 
and  the  Committee  of  the  Regions  ‘A  Digital  Single  Market  Strategy  for  Europe’,  SWD(2015)  100 
final. 

112  National  administrative  systems  are  often  characterised  by  different  administrative  cultures,  which 
include the beliefs and values on the role of the State and its civil servants. As pointed out by a recent 
Study  published  by  the  Commission,  although  European  public  administrations  share  the  values 
associated with democracy and the rule of law, national administrative cultures show clear differences 
(Thijs  N.,  Hammerschmid  G.,  Palaric  E.  (2017),  A  comparative  overview  of  public  administration 
characteristics and performance in EU28, European Commission). 

112 

 
                                                                                                                                                 
  general  objectives,  concerning  the  overall  rationale  of  an  intervention  and  its 
longer-term and more diffuse effects, i.e. pertaining to the questions: ‘why has the 
ISA²  programme  been  set  up?’  and  ‘what  ultimate  goal  was  it  expected  to 
contribute to?’;  

  specific objectives, providing a basis for assessing an intervention in relation to 
the  short-term  or  medium-term  results  that  occur  at  the  level  of  direct 
beneficiaries/recipients of the intervention; and 

  operational objectives, providing a basis for assessing an intervention in relation 
to  its  direct  outputs,  i.e.  ‘what  is  directly  produced/supplied  during  the  ISA² 
programme’s implementation?’. 

In concrete terms, the ISA2 programme’s objectives are the following: 

General objective:  

  To  promote  the  ICT-based  modernisation  of  the  public  sector  in  Europe  and  to 
facilitate  addressing  the  needs  of  businesses  and  citizens  via  improved 
interoperability  of  European  public  administrations,  thus  contributing  to  the 
completion of the Digital Single Market and, ultimately, to economic growth and 
the global competitiveness of the European Union.113  

Specific objectives: 

  Specific objective #1: To develop, maintain and promote a holistic approach to 
interoperability  in  the  Union  in  order  to  eliminate  fragmentation  in  the 
interoperability landscape in the Union (Article 1(a) of the ISA2 Decision). 

  Specific objective #2: To facilitate efficient and effective electronic cross-border 
or  cross-sector  interaction  between  European  public  administrations  on  the  one 
hand,  and  between  European  public  administrations  and  businesses  and  citizens 
on the other (Article 1(b) of the ISA2 Decision). 

  Specific  objective  #3:  To  contribute  to  the  development  of  a  more  effective, 
simplified  and  user-friendly  e-administration  at  the  national,  regional  and  local 
levels of public administration (Article 1(b) of the ISA2 Decision). 

  Specific  objective  #4:  To  identify,  create  and  operate  interoperability  solutions 
supporting the implementation of Union policies and activities (Article 1(c) of the 
ISA2 Decision). 

  Specific  objective  #5:  To  facilitate  the  reuse  of  interoperability  solutions  by 

European public administrations (Article 1(d) of the ISA2 Decision). 

113  See recitals 34, 36 and 45 of the ISA2 Decision and Section 1.1 of the ISA2 Proposal.  

113 

 
 
                                                 
Operational objectives: 

The operational objectives of the ISA2 programme are listed in the first column of Table 
6. The additional columns are relevant for the programme’s expected results (see below 
for further details). 

Table 6: Overview of operational objectives and expected outputs of the ISA² 
programme  

Operational objectives 

Expected outputs 

Performance indicators 

  Number of common 

services and generic tools 
used by European public 
administrations 
  Adoption of common 

frameworks by European 
public administrations 

  Users uptake 
  User satisfaction 
  Number of new sectors 
covered by existing 
solutions (cross-sector / 
cross-border dimension)  

  Number of new common 
services and generic tools 
by European public 
administrations 

  Adoption of new common 
frameworks by European 
public administrations 

Operational objective (1): 
To support and promote the 
assessment, improvement, 
operation and re-use of 
existing cross-border or 
cross-sector 
interoperability solutions 
and common frameworks 
(Article 3 (a) of the ISA2 
Decision) 

Operational objective (2): 
To support and promote the 
development, 
establishment, bringing to 
maturity, operation and re-
use of new cross-border or 
cross-sector 
interoperability solutions 
and common frameworks 
(Article 3 (b) of the ISA2 
Decision) 

  Reuse of interoperable 
solutions and common 
frameworks 

  Reuse of best practices 
  Continuity in the delivery of 
interoperable solutions  
  Professional delivery of a 
coherent portfolio of 
interoperable solutions and 
common frameworks for 
European public 
administrations 
Improved services and 
infrastructures 
Improved tools 

 
  Continuity in the tools 

 

 

 

provision  
Increased reuse of existing 
components in the 
development of new 
interoperable solutions and 
common frameworks 
Increased availability of 
services and common 
frameworks that meet the 
needs of sectors and 
Member States  

  Delivery of new services 

 

and common frameworks in 
time 
Increased availability of 
tools that meet the needs of 
sectors and Member States  

  Delivery of new tools in 

time  

114 

 
Operational objectives 

Expected outputs 

Performance indicators 

Operational objective (3): 
To support and promote the 
assessment of the ICT 
implications of proposed or 
adopted Union law (Article 
3 (c) of the ISA2 Decision) 

Operational objective (4): 
To support and promote the 
identification of legislation 
gaps, at Union and national 
level, that hamper cross-
border or cross-sector 
interoperability between 
European public 
administrations (Article 3 
(d) of the ISA2 Decision) 

Operational objective (5): 
To support and promote the 
development of 
mechanisms that measure 
and quantify the benefits of 
interoperability solutions 
including methodologies 
for assessing cost-savings 
(Article 3 (e) of the ISA2 
Decision) 

Operational objective (6): 
To support and promote the 
mapping and analysis of 
the overall interoperability 
landscape in the Union 
through the establishment, 
maintenance and 
improvement of the EIRA 
and the EIC as instruments 
to facilitate the re-use of 
existing interoperability 
solutions and to identify 

  Smooth implementation of 

 

EU legislation 
Identification of the needs 
for services and tools in 
time so that requirements 
are fulfilled when the 
legislation comes into force  

  Understanding of ICT 

dimensions of EU policies 
  A defined role and process 
for the ISA² programme to 
support the DGs and 
services undertaking 
assessments of ICT 
implications of new EU 
legislation as part of the 
Commission’s Impact 
Assessment process (Better 
Regulation Guidelines)114  
Identification and resolution 
of legislation gaps that 
hamper cross-border or 
cross-sector interoperability 
between European public 
administrations 

 

  Functioning mechanisms to 
promote the use and monitor 
the adoption of 
interoperability solutions as 
well as their numbers of 
users 

  Cost/Benefit 

model/methodology 
available for assessing cost-
savings based on the study 
related to the cost-benefit of 
interoperability. 

  Existing interoperability 
solutions documented 
within the European 
Interoperability Cartography 
(EIC/EICart) applying the 
principles and guidelines 
from the European 
Interoperability Reference 
Architecture (EIRA) 
  EIRA used by European 
public administrations 

  Number of pieces of 
legislation reviewed 
  Number of assessments of 
ICT implications of new 
EU legislation conducted 

  Number of instances where 
ISA² is involved (e.g. 
participation of ISA² in 
evaluations, impact 
assessments, and REFIT of 
EU legislation) 

  Number of pieces of gaps 

identified 

  Usage of the Cost/Benefit 

model to produce figures to 
show the value of 
interoperability. 

  Usage of EIRA by 
European public 
administrations 

  Usage of EIC/EICart by 

European public 
administrations 

  Number of references in 
implementations of 
interoperable solutions 
  Number of interoperable 

solutions documented in the 

114  European Commission (2017), Better Regulation Guidelines, SWD(2017) 350, Brussels, 7 July 2017. 

115 

 
 
                                                 
Operational objectives 

Expected outputs 

Performance indicators 

  EIC/EICart facilitating the 

EIC/EICart 

deployment of interoperable 
solutions within and 
between European public 
administrations 
Interoperable solutions 
developed in the frame of 
ISA² that take into account 
EIF and EIRA 
Interoperable solutions 
developed by other 
Commission DGs that take 
into account EIF and EIRA 
ISA² actions implementing 
the priorities/actions set by 
the EIS 

 

 

 

  Adoption of ‘standardised’ 
specific outputs from 
previous programme (e.g. 
Core Vocabularies, ADMS) 
by European public 
administrations. 
  Standards, open 

specifications and 
assessment procedures 
aligned with Member States 
International/European 
recognition of standards and 
open specifications 

 

  Established new standards 
and open specifications 
  Alignment of standards with 

EU standardisation 
platforms 

  All existing ready-for-use 
interoperable and best 
practice solutions are 
documented in and 
accessible via Joinup115 to 
raise awareness and avoid 
duplication of efforts among 
stakeholders 

  Usage of EIRA by 
European public 
administrations 

  Number of EIS priorities 

implemented by European 
public administrations 
  Level of implementation of 
the EIF at national and 
Commission level 

  Number of new or revised 

standards used by European 
public administrations 
  Number of new or revised 
common specifications and 
open specifications used by 
European public 
administrations 

  Number of assessments 
(e.g. through multi-
stakeholder platform or 
CAMMS) 

  Number of ready-for-use 
interoperable and best 
practice solutions housed in 
Joinup 

the areas where such 
solutions are still lacking 
(Article 3 (f) of the ISA2 
Decision) 

Operational objective (7): 
To support and promote the 
maintenance, updating, 
promotion and monitoring 
of the implementation of 
the EIS, the EIF and the 
EIRA (Article 3 (g) of the 
ISA2 Decision) 

Operational objective (8): 
To support and promote the 
assessment, updating and 
promotion of existing 
common specifications and 
standards and the 
development, 
establishment and 
promotion of new common 
specifications and open 
specifications and 
standards through the 
Union's standardisation 
platforms and in 
cooperation with European 
or international 
standardisation 
organisations as 
appropriate (Article 3 (h) 
of the ISA2 Decision) 
Operational objective (9): 
To support and promote the 
maintenance and 
publication of a platform 
allowing access to, and 
collaboration with regard 
to, best practices, 
functioning as a means of 
raising awareness and 
disseminating available 
solutions, including 
security and safety 
frameworks, and helping to 

115  For further details see: https://joinup.ec.europa.eu/  

116 

 
 
                                                 
Operational objectives 

Expected outputs 

Performance indicators 

avoid duplication of efforts 
while encouraging the re-
usability of solutions and 
standards (Article 3 (i) of 
the ISA2 Decision) 
Operational objective (10): 
To support and promote the 
bringing of new 
interoperability services 
and tools to maturity, and 
maintaining and operating 
existing interoperability 
services and tools on an 
interim basis (Article 3 (j) 
of the ISA2 Decision) 
Operational objective (11): 
To support and promote the 
identification and 
promotion of best 
practices, to develop 
guidelines to coordinate 
interoperability initiatives 
and to animate and support 
communities working on 
issues relevant to the area 
of electronic cross-border 
or cross-sector interaction 
between end-users (Article 
3 (k) of the ISA2 Decision) 

Operational objective (12): 
To develop a 
communication strategy 
[…], aiming to enhance 
information and increase 
awareness with regard to 
the ISA² programme and 
its benefits, targeting 
businesses, including 
SMEs, and citizens, and 
employing user-friendly 
means on the ISA² 
programme's website 
(Article 3 of the ISA2 
Decision) 
Source: CEPS final study 

  Continued development and 
interim operation of ISA 
solutions, e.g. sTESTA, 
with a view to achieving 
longer-term sustainable 
financing and operational 
governance structures 

  Number of new 

interoperability services 
and tools developed, 
maintained and operated by 
ISA² 

  Number of new 

interoperability services 
brought to maturity 

  Continued use of Joinup to 

house ready-for-use 
interoperable and best 
practice solutions to 
promote interoperability 
  Enhanced coordination, 

notably, through the Inter-
Service Group on Public 
Administration Quality and 
Innovation and also in 
publications, such as the 
NIFO and eGovernment 
factsheets 

  An updated Communication 

Strategy with clear 
stakeholder, monitoring and 
measurement targets. 

  Number of ready-for-use 
interoperable and best 
practice solutions housed in 
Joinup 

  Number of joint-initiatives 

(studies etc.) in 
interoperability by EU-wide 
groups, such as the 
Commission’s Inter-Service 
Group on Public 
Administration Quality and 
Innovation. 
  Number of joint-
publications 
  Monitoring and 

measurement of the 
Communication Strategy 
(events, satisfaction 
surveys, publications). 
  Numbers of stakeholders 

engaged in different events, 
actions, initiatives etc. 

2.  THE INTERVENTION: ‘WHAT ACTIONS ARE SUPPORTED BY THE ISA² PROGRAMME?’ 

After identifying the needs, problems and objectives underlying the ISA² programme, the 
intervention 
features 
(activities/inputs)  of  the  programme  itself.  Within  the  intervention  logic,  activities  and 

the  assessment  of 

logic  approach 

the  main 

requires 

117 

 
inputs represent the means to address the needs and problems, and to meet the identified 
objectives.  In  this  context,  these  means  largely  correspond  to  the  actions  of  the  ISA² 
programme, as defined in the ISA2 Rolling Work Programme116. The ISA2 actions are 
grouped into the following nine packages. 

  Key  and  generic 

interoperability  enablers  are  actions 

that  develop 
interoperability solutions to support public administrations in providing services, 
ranging  from  the  secure  exchange  of  files  to  improving  cross-border  access  to 
government  data  and  information.  Examples  include  TESTA117  and  Open  e-
TrustEx118. 

  The  semantic  interoperability  package  consists  of  initiatives  to  establish  and 
through 
increase  semantic 
information standards and specifications. Examples include Core Vocabularies119 
and DCAT Application Profile for data portals in Europe120. 

interoperability  among  public  administrations 

  Under  the  access  to  data  /  data  sharing  /  open  data  package,  the  ISA² 
programme  funds  actions  that  help  facilitate  the  reuse  of  national  data 
repositories, of data across borders and sectors and widen access to data created 
by the public sector. Example include ‘Providing big data opportunities for public 
administrations’121 and ‘FISMA: Financial Data Standardisation’122. 

  The  geospatial  solutions  package  consists  of  legal/policy,  organisational, 
semantic  and  technical  interoperability  solutions  that  aims  to  facilitate  efficient 
and  effective  electronic  cross-border  and  cross-sector  interaction  between 
European public administrations, and between them and citizens and companies, 
in  the  field  of  location  information  and  services.  This  package  led  to  the 
Re3gistry123 solution. 

  The  eProcurement/eInvoicing  package  helps  update  and  improve  existing  EU 
tools  for  e-procurement.  It  aims  to  develop  a  common  public  procurement 
knowledge  base  to  facilitate  the  creation,  exchange,  dissemination  and  reuse  of 
procurement data. Examples include Open e-Prior124, ESPD125 and eCertis126. 

116  European Commission (2016), ISA2 Work Programme 2016 – Summary. 
117  https://ec.europa.eu/isa2/solutions/testa_en 
118  https://ec.europa.eu/isa2/solutions/open-e-trustex_en 
119  https://ec.europa.eu/isa2/solutions/core-vocabularies_en 
120  https://ec.europa.eu/isa2/solutions/dcat-application-profile-data-portals-europe_en 
121  https://ec.europa.eu/isa2/actions/providing-big-data-opportunities-public-administrations_en 
122  https://ec.europa.eu/isa2/actions/towards-better-financial-data-reporting_en 
123  https://ec.europa.eu/isa2/solutions/re3gistry_en 
124  https://ec.europa.eu/isa2/solutions/open-e-prior_en 
125  https://ec.europa.eu/isa2/solutions/european-single-procurement-document-espd_en 
126  https://ec.europa.eu/isa2/solutions/e-certis_en 

118 

 
                                                 
  The  decision-making  and  legislation  package  covers  actions  that  support  the 
decision-making process, in particular by improving interoperability between EU 
institutions  and  between  them  and  the  Member  States.  These  actions  address 
several processes – ranging from collecting feedback from various stakeholders to 
drafting new legislation and monitoring that this legislation is implemented. The 
package led to the LEOS127 solution. 

  The  EU  policies  —  supporting  instruments  package  covers  actions  that 
support the implementation of EU policies for which interoperability aspects need 
to  be  considered.  The  package  has  led  to  the  following  solution:  Online 
Collection Software to support European Citizens' Initiatives.128 

  The  supporting  instruments  for  public  administrations  package  maps  the 
EU’s  interoperability  landscape  and  provides  solutions  to  support  greater 
interoperability,  ranging  from  sharing  best  practices  to  supporting  the  reuse  of 
solutions.    Examples  include  EUSurvey129,  Joinup  –  European  Collaborative 
Platform  and  Catalogue130  and 
the  National  Interoperability  Framework 
Observatory (NIFO)131. 

interoperability  and  monitoring  and  evaluating 

  The accompanying  measures  package  consists of  activities to  raise  awareness 
of 
the  programme’s 
implementation.  Examples  include  the  annual  SEMIC  Conference132,  the  ISA2 
Mid-Term Conference133 and the ISA2 Dashboard134. 

3.  THE  EXPECTED  RESULTS  OF  THE  INTERVENTION:  ‘WHAT  ARE  THE  EXPECTED 

EFFECTS OF THE ACTIONS SUPPORTED BY THE ISA² PROGRAMME?’ 

When it began, the  ISA² programme was expected to  have certain  effects (i.e. to  cause 
changes) on certain categories of stakeholders. These effects fall under three categories 
(outputs,  outcomes  and  impacts)  depending  on  when  they  occur  and  the  groups  they 
address. Note that external factors and other EU policies may influence the performance 
of the ISA² programme. 

Outputs (expected) 

The  outputs  of  the  ISA²  programme  are  its  most  immediate  effects,  i.e.  the 
deliverables/products  of  the funded  actions.  As the intervention logic looks at  a certain 

127  https://ec.europa.eu/isa2/solutions/leos_en 
128  https://ec.europa.eu/isa2/solutions/ocs-ecis-0_en 
129  https://ec.europa.eu/isa2/solutions/eusurvey_en 
130  https://ec.europa.eu/isa2/solutions/joinup_en 
131  https://ec.europa.eu/isa2/solutions/nifo_en 
132  https://ec.europa.eu/isa2/news/save-date-semic-2018_en 
133  https://ec.europa.eu/isa2/events/isa%C2%B2-mid-term-conference-linking-public-administrations-

businesses-and-citizens_en 

134  https://ec.europa.eu/isa2/dashboard/  

119 

 
                                                 
intervention by simulating an ex ante perspective, one must identify the outputs that were 
expected to stem from the programme when it was initially adopted. Note that expected 
outputs  usually  reflect  the  operational  objectives  identified  in  previous  analytical  steps. 
In  this  respect,  the  expected  outputs  and  related  indicators  are  listed  in  the  ex-ante 
evaluation of ISA². Table 6 above summarises the expected outcomes and a series of key 
performance indicators for each operational objective of the ISA² programme. 

Outcomes (expected) 

The  outcomes  are  the  short-term  and  medium-term  changes  for  the  groups  the 
programme  directly  addresses.  As  outcomes  are  usually  connected  to  the  specific 
objectives, they can be summarised as follows: 

  a coherent interoperability landscape in the Union based on a holistic approach to 

interoperability; 

  efficient  and  effective  electronic  cross-border  or  cross-sectoral  interactions 
between  European  Public  Administrations  as  well  as  between  European  public 
administrations and businesses/citizens; 

  more effective, simplified and user-friendly e-administrations in European public 

administrations; 

  advancement of Union policies and activities by supporting their implementation; 

and 

 

take-up  and  re-use  of  the  ISA2  programme’s  results  by  European  public 
administrations. 

Impacts (expected) 

As  mentioned  above,  the  intervention  logic  also  includes  the  impacts,  i.e.  the  changes 
caused by an EU intervention over a longer period and also affecting the society as a 
whole rather than only the direct addressees of the intervention itself. These changes are 
related to the general objectives of a certain intervention. By analysing the ISA2 Decision 
and the ISA2 proposal, the following impacts are expected: 

 

 

 

increasing the speed, efficiency and quality in the creation and delivery of cross-
border and cross-sector electronic public services to meet the needs of businesses 
and citizens;  

improving the efficiency and productivity of the European public administrations, 
which  can  be  a  strong  driver  of  economic  growth  through  its  support  for,  and 
governance of, the private sector; 

reducing the cost and administrative burden of cross-border interaction, removing 
administrative  e-barriers  and  contributing  to  the  swift  implementation  of  ICT 
systems supporting EU legislation; and 

  contributing to the successful achievement of the DSM. 

120 

 
Annex 4: Evaluation Framework 

This annex presents the evaluation framework135 on which the entire evaluation is based. 

Evaluation questions 

Success/judgment criteria 

Indicators 

Data sources 

Data collection / analysis methods 

Evaluation criterion #1: Relevance 

1.  To what extent are the 
objectives of the ISA² 
programme still 
pertinent in relation to 
the evolving needs and 
problems at both 
national and EU 
levels? 

  Degree of alignment 

  Share of stakeholders 

between stakeholders’ 
perception of needs and 
problems at national 
and sub-national levels 
and the objectives of 
the programme. 
  Degree of alignment 

between stakeholders’ 
perception of needs and 
problems at EU level 
and the objectives of 
the programme. 
  Degree of alignment 

between needs and 
problems originally 
addressed by the 
programme and 
stakeholders’ 
perception of needs and 
problems 

confirming the alignment 
between needs and problems 
addressed by the programme 
and current needs and 
problems. 

  Share of stakeholders 

confirming the alignment 
between the objectives of the 
programme and current needs 
and problems at national and 
sub-national levels. 
  Share of stakeholders 

confirming the alignment 
between the objectives of the 
programme and current needs 
and problems at EU level. 
  Qualitative assessment of the 
alignment between the 
objectives of the programme 
and current needs and 

  Primary information on needs and 
problems from the following 
categories of stakeholders: 
o 
o  Action owners. 
o 

Programme governance. 

Solution Users – European 
Commission. 
Solution Users – Member 
States. 
Stakeholders responsible 
for linked EU 
policies/initiatives. 
Experts. 
Indirect beneficiaries and 
wider public. 

o 

o 

o 
o 

  Secondary information on needs 

and problems from operational 
documents, other official 
documents and relevant literature, 
such as: 
o 

ISA² Decision and 

  Desk research. 
 

Interviews with the following 
categories of stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

o  Stakeholders responsible for 

linked EU policies/initiatives. 
o  Standardisation organisations. 

  Online surveys targeted to the 
following categories of 
stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

135  Note that the  wording of some elements of the  evaluation  framework  mentioned in the  body text of the Staff Working  Document (like the judgment criteria) has been slightly 

revised in some cases to improve understanding. 

121 

 
 
                                                 
Evaluation questions 

Success/judgment criteria 

Indicators 

Data sources 

Data collection / analysis methods 

problems. 

  Qualitative assessment of the 
alignment between needs and 
problems addressed by the 
programme and current needs 
and problems. 

o 

o 
o 

o 

accompanying documents. 
ISA² Rolling Work 
Programme. 
eGovernment factsheets. 
Study on the role of 
eGovernment and 
interoperability in the 
European Semester. 
State of Play of 
Interoperability in Europe. 

o  Data on the digital 

o 

economy from Eurostat. 
Interim review of the DSM 
Strategy. 

o  Stakeholders responsible for 
linked policies/initiatives. 

  Short questionnaire to be 

administered during the ISA2 Mid-
Term Conference. 

  Public consultation. 
  Quantitative assessment of 

responses to interviews and surveys 
(Likert scale). 

  Qualitative assessment of responses 

to interviews and surveys and data 
and information collected via desk 
research. 

2.  How far are the ISA² 

  Degree of alignment 

  Share of stakeholders 

  Primary information actual 

Evaluation criterion #2: Effectiveness 

programme's results in 
the process of 
achieving the 
programme's 
objectives? 

3.  Are there aspects that 
are more or less 
effective than others, 
and if so, what lessons 
can be drawn from 
this? 

between actual and 
expected results and 
objectives of the 
programme.  
Impact of external 
factors on the 
performance of the 
programme. 

 

confirming the alignment 
between actual and expected 
results of the programme. 

  Share of stakeholders 

confirming the alignment 
between the objectives and 
actual results of the 
programme. 

  Measurement of the 

indicators summarising 
the outputs of the 
programme. 

  Degree of alignment 

with principles spelled 
out in Article 4 of the 
ISA2 Decision. 
  Awareness of the 
programme. 

  Share of stakeholders who are 
aware of the programme. 

  Share of stakeholders 

identifying external factors 
contributing to/jeopardising the 
performance of the programme. 

  Share of stakeholders who are 

aware of specific ISA2 
packages/ actions/ solutions. 
  Qualitative assessment of the 

122 

results and contribution to the 
programme’s objectives from the 
following categories of 
stakeholders: 
o 
o  Action owners. 
o 

Programme governance. 

o 

Solution Users – European 
Commission. 
Solution Users – Member 
States. 
Experts. 
Indirect beneficiaries and 
wider public. 
  Secondary information from 

o 
o 

operational documents and other 
official documents, such as: 
o 

ISA² Rolling Work 
Programme 

  Desk research. 
 

Interviews with the following 
categories of stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

o  Standardisation organisations. 

  Online surveys targeted to the 
following categories of 
stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

 
 
Evaluation questions 

Success/judgment criteria 

Indicators 

Data sources 

Data collection / analysis methods 

alignment between objectives, 
expected and actual results of 
the programme. 

  Quantitative assessment of 
performance indicators of 
outputs. 

  Qualitative assessment of the 
alignment with principles 
spelled out in Article 4 of the 
ISA2 Decision. 

o 

o 

ISA² Annual Monitoring 
and Evaluation Reports. 
ISA² Quarterly Monitoring 
Reports. 
ISA² Dashboard. 

o 
o  Documentary evidence on 
funded actions (e.g. 
deliverables, final reports). 

Evaluation criterion #3: Efficiency 

4.  To what extent has the 
programme been cost-
effective? 

5.  Which aspects of the 
programme are the 
most efficient or 
inefficient, especially 
in terms of resources 
mobilised? 
5.1  How is the 
programme 
performing 
relative to the 
planned work and 
budget?   

  Cost-effectiveness 

analysis to assess the 
ratio between allocated 
funds and actual results 
of the programme. 

  Earned value 

management analysis. 

  Efficiency of the 

selection process of the 
actions to be included 
in the Rolling Work 
Programme. 

  Comparison between the costs 
of ISA2 packages and the 
results measured via 
performance indicators. 
  Earned Value, Actual Costs, 
Planned Value, and Schedule 
Performance Index of ISA2 
packages.136 

  Share of stakeholders 

confirming that the selection 
process of the actions is fit-for-
purpose. 

  Assessment of regulatory costs 
linked to the selection process 
of the actions included in the 
Rolling Work Programme. 

  Primary information on costs 

from the following categories of 
stakeholders: 
o 
o  Action owners. 
  Secondary information from 

Programme governance. 

operational documents and other 
official documents, such as: 
o 

ISA² Rolling Work 
Programme 
ISA² Annual Monitoring 
and Evaluation Reports. 
ISA² Quarterly Monitoring 
Reports. 
ISA² Dashboard. 

o 
o  Documentary evidence on 
funded actions (e.g. 

o 

o 

136  The efficiency of the funded actions is already monitored by using the Earned Value Management (EVM) approach.  

123 

  Short questionnaire to be 

administered during the ISA2 Mid-
Term Conference. 

  Public consultation. 
  Quantitative assessment of 

responses to interviews and surveys 
(Likert scale). 

  Qualitative assessment of responses 

to interviews and surveys and data 
and information collected via desk 
research. 

  Multi-criteria analysis. 
  Success stories/lessons learnt. 

  Desk research. 
 

Interviews with the following 
categories of stakeholders: 
o  Programme governance. 
o  Action owners. 

  Online surveys targeted to the 
following categories of 
stakeholders: 
o  Programme governance. 
o  Action owners. 
  Quantitative assessment of 

responses to surveys (Likert Scale). 
  Qualitative assessment of responses 

to interviews and surveys and data 
and information collected via desk 
research. 

  Cost-effectiveness analysis. 

 
 
 
                                                 
Evaluation questions 

Success/judgment criteria 

Indicators 

Data sources 

Data collection / analysis methods 

deliverables, final reports). 

Evaluation criterion #4: Coherence 

6.  To what extent do the 
ISA² actions form part 
of a ‘holistic’ 
approach within the 
framework of the 
programme? (internal 
coherence) 

7.  To what extent is the 
ISA² programme 
coherent with other 
EU interventions that 
have similar 
objectives and with 
global initiatives in the 
same field? (external 
coherence) 

  Degree of coherence 

  Share of stakeholders 

among actions funded 
by the ISA² programme 
(internal coherence). 
  Degree of coherence 

between the 
programme and other 
EU supported 
programmes (external 
coherence). 
o  Focus on CEF, 

SRSP, Horizon 2020 

  Degree of coherence 

between the 
programme and other 
EU policies (external 
coherence). 
o  Focus on DSM, 
Digital Strategy 
(forthcoming), ICT 
standardisation, 
Single digital 
gateway, 
eGovernment Action 
Plan, Tallinn 
Declaration on 
eGovernment. 

  Degree of coherence 

between the 
programme and global 
initiatives in the field 

identifying synergies/overlaps 
between funded actions. 

  Share of stakeholders 

identifying synergies/overlaps 
between the programme and 
other relevant EU 
programmes/policies. 
  Qualitative assessment of 

synergies/overlaps and links 
between funded actions. 

  Qualitative assessment of 

synergies/overlaps between 
objectives of the programme 
and other relevant EU 
programmes/policies. 
  Qualitative assessment of 

synergies/overlaps between 
funded actions and those of 
other relevant EU programmes. 

  Qualitative assessment of 

synergies/overlaps between the 
programme and global 
initiatives in the same field. 
Instances of reuse of results 
delivered by funded actions by 
other actions within the 
programme. 
Instances of reuse of results 
delivered by funded actions by 

 

 

124 

  Primary information on internal 
coherence from the following 
categories of stakeholders: 
o 
o  Action owners. 
o 

Programme governance. 

Solution Users – European 
Commission. 
Solution Users – Member 
States. 

o 

o 

Programme governance. 

  Primary information on external 
coherence from the following 
categories of stakeholders: 
o 
o  Action owners. 
o 

Solution Users – European 
Commission. 
Solution Users – Member 
States. 
Stakeholders responsible 
for linked EU 
policies/initiatives. 
Experts. 
Indirect beneficiaries and 
wider public. 
  Secondary information from 

o 
o 

o 

operational documents and other 
official documents, such as: 
o 

ISA² Decision and 
accompanying documents. 
ISA² Rolling Work 
programme. 

o 

  Standard cost model. 
  Earned value management. 
  Success stories/lessons learnt. 

  Desk research. 
 

Interviews with the following 
categories of stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

o  Stakeholders responsible for 

linked EU policies/initiatives. 
o  Standardisation organisations. 

  Online surveys targeted to the 
following categories of 
stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

o  Stakeholders responsible for 
linked policies/initiatives. 

  Short questionnaire to be 

administered during the ISA2 Mid-
Term Conference. 

  Public consultation. 
  Quantitative assessment of 

responses to interviews and surveys 
(Likert scale). 

 
Evaluation questions 

Success/judgment criteria 

Indicators 

Data sources 

Data collection / analysis methods 

(external coherence). 
o  Focus on OECD 

Digital Government 
and the UNPAN. 

  Level of reuse of 

results of a funded 
action by another 
action within the ISA² 
programme (internal 
coherence). 
  Level of reuse of 

results delivered by 
ISA² actions by other 
EU programmes 
(external coherence). 

other EU programmes. 

o 

o 

ISA² Annual Monitoring 
and Evaluation Reports. 
ISA² Quarterly Monitoring 
Reports. 

  Qualitative assessment of responses 

to interviews and surveys and data 
and information collected via desk 
research. 

o  Documentary evidence on 

  Success stories/lessons learnt. 

o 

o 

funded actions (e.g. official 
deliverables, final reports). 
Legal texts establishing 
other relevant programmes 
accompanying documents.  
Legal texts devising other 
relevant EU policies and 
accompanying documents. 

o  Documents describing 
global initiatives in the 
same field. 

Evaluation criterion #5: EU added value 

8.  What is the additional 

  Achievement of 

  Share of stakeholders 

value resulting from 
the ISA² programme, 
compared to what 
could reasonably have 
been expected from 
Member States acting 
at national, regional 
and/or local levels? 

objectives that could 
not be otherwise 
attained with 
national or sub-
national 
interventions. 
  Achievement of 

objectives at a cost 
lower than what 
could be attained via 
national or sub-
national 
interventions. 
  Achievement in 

terms of cross-border 
interoperability. 
  Contribution to the 

confirming the need for an EU 
intervention to achieve the 
objectives of the programme. 

  Share of stakeholders 
confirming that an EU 
intervention is able to achieve 
the objectives of the 
programme at cost lower than 
costs of national or sub-national 
interventions. 

  Share of stakeholders’ 

providing positive feedback on 
achievements in terms of cross-
border interoperability. 
  Qualitative assessment of the 

contribution to the 
advancement of common EU 

125 

  Primary information on cross-
border interoperability and EU 
added value from the following 
categories of stakeholders: 
o 
o  Action owners. 
o 

Programme governance. 

Solution Users – European 
Commission. 
Solution Users – Member 
States. 
Experts. 
Indirect beneficiaries and 
wider public. 
  Secondary information from 

o 
o 

o 

operational documents and other 
official documents, such as: 
o 

ISA² Decision and 
accompanying documents. 

  Desk research. 
 

Interviews with the following 
categories of stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

o  Standardisation organisations. 

  Online surveys targeted to the 
following categories of 
stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

 
Evaluation questions 

Success/judgment criteria 

Indicators 

Data sources 

Data collection / analysis methods 

advancement of 
common EU 
policies. 

policies. 

  Quantitative assessment of 

indicators summarising cross-
border outputs of the 
programme. 

o 

o 

o 

ISA² Rolling Work 
programme. 
ISA² Annual Monitoring 
and Evaluation Reports. 
ISA² Quarterly Monitoring 
Reports. 

o  Documentary evidence on 

funded actions (e.g. official 
deliverables, final reports). 

9.  How do the ISA² 

  Degree of alignment 

  Share of stakeholders 

  Primary information on user 

Evaluation criterion #6: Utility137 

programme’s actions 
and results, achieved 
and anticipated, 
compare with the 
needs they are 
supposed to 
address?138 

between stakeholders’ 
perception of needs and 
problems at national 
and sub-national levels 
and the results of the 
programme. 

  Degree of alignment 

between stakeholders’ 
perception of needs and 
problems at EU level 
and the results of the 
programme. 

confirming the alignment 
between the results of the 
programme and current needs 
and problems at national and 
sub-national levels. 
  Share of stakeholders 

confirming the alignment 
between the results of the 
programme and current needs 
and problems at EU level. 
  Qualitative assessment of the 

satisfaction and utility from the 
following categories of 
stakeholders: 
o 
o  Action owners. 
o 

Programme governance. 

Solution Users – European 
Commission. 
Solution Users – Member 
States. 
Stakeholders responsible 
for linked EU 
policies/initiatives. 

o 

o 

States. 
  Short questionnaire to be 

administered during the ISA2 Mid-
Term Conference. 

  Public consultation. 
  Quantitative assessment of 

responses to interviews and surveys 
(Likert scale). 

  Qualitative assessment of responses 
to interviews and surveys and of 
data and information collected via 
desk research. 

  Success stories/lessons learnt. 

  Desk research. 
 

Interviews with the following 
categories of stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

o  Stakeholders responsible for 

linked EU policies/initiatives. 
o  Standardisation organisations. 

137  The  utility  criterion  to  some  extent  is  similar  to  the  relevance  criterion  insofar  as  they  both  look  at  stakeholders'  needs.  However,  while  the  relevance  criterion  looks  at  the 
alignment between the objectives of the programme and the current needs and problems experienced by stakeholders, the utility criterion focuses on how the actual results of the 
programme have (or do not have) contributed to meeting stakeholders’ needs. Hence, the utility criterion is a proxy for measuring users’ satisfaction. 

138  In order to draft policy recommendations, feedback on measures to improve the utility of the ISA² programme’s actions has been collected. However, stricto sensu, future measures 

are not part of the interim evaluation, which is mostly a backward-looking exercise. 

126 

 
                                                 
Evaluation questions 

Success/judgment criteria 

Indicators 

Data sources 

Data collection / analysis methods 

  User satisfaction, with 
a breakdown by 
stakeholder group. 

alignment between the results 
of the programme and current 
needs and problems. 
  Quantitative assessment of 
users’ satisfaction (Likert 
Scale). 

  Quantitative assessment of the 
take-up by EU, national and 
sub-national administrations of 
ISA2 outputs. 

o 
o 

Experts. 
Indirect beneficiaries and 
wider public. 
  Secondary information on utility 
from operational documents, 
other official documents and 
relevant literature, such as: 
o  ISA² Rolling Work 

Programme. 

o  ISA² Annual Monitoring and 

Evaluation Reports. 

o  ISA² Quarterly Monitoring 

Reports. 

o  Documentary evidence on 

  Online surveys targeted to the 
following categories of 
stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

o  Stakeholders responsible for 
linked policies/initiatives. 

  Short questionnaire to be 

administered during the ISA2 Mid-
Term Conference 

funded actions (e.g. official 
deliverables, final reports). 

  Public consultation 
  Quantitative assessment of 

responses to interviews and surveys 
(Likert scale) 

  Qualitative assessment of responses 
to interviews and surveys and of 
data and information collected via 
desk research. 

  Success stories/lessons learnt. 

127 

 
 
 
 
 
 
 
 
 
 
 
 
Evaluation questions 

Success/judgment criteria 

Indicators 

Data sources 

Data collection / analysis methods 

Evaluation criterion #7: Sustainability 

10.  To what extent is the 

  Extent to which the 

financial, technical 
and operational 
sustainability of the 
developed solutions – 
maintained and 
operated through the 
ISA² programme –
ensured? 

results achieved by the 
ISA2 programme are 
expected to last if 
funding for actions 
covered by the 
programme would not 
be available in the 
future. 

  Extent to which ‘cost 
recovery’ solutions 
could be introduced. 

  Share of stakeholders expecting 
that results achieved so far 
would last if funding for actions 
covered by the programme 
would not be available in the 
future. 

  Share of actions requiring 
operation and maintenance 
costs to deliver their results. 

  Share of actions requiring 
technical and operational 
support to deliver their results. 

  Share of stakeholders who 
would pay to keep on using 
specific ISA2 solutions. 

  Primary information on 

sustainability from the following 
categories of stakeholders: 
o 
o  Action owners. 
o 

Programme governance. 

Solution Users – European 
Commission. 
Solution Users – Member 
States. 
Experts. 
Indirect beneficiaries and 
wider public. 

o 

o 
o 

  Secondary information on 

sustainability from operational 
documents, other official 
documents and relevant literature, 
such as: 
o 

ISA² Rolling Work 
Programme. 
ISA² Annual Monitoring 
and Evaluation Reports. 
ISA² Quarterly Monitoring 
Reports. 

o 

o 

Source: CEPS final study 

o  Documentary evidence on 

funded actions (e.g. official 
deliverables, final reports). 

128 

  Desk research. 
 

Interviews with the following 
categories of stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

o  Standardisation organisations. 

  Online surveys targeted to the 
following categories of 
stakeholders: 
o  Programme governance. 
o  Action owners. 
o  Solution Users – European 

Commission. 

o  Solution Users – Member 

States. 

  Quantitative assessment of 

responses to interviews and surveys 
(Likert scale). 

  Qualitative assessment of responses 
to interviews and surveys and of 
data and information collected via 
desk research. 

  Quantitative assessment of 

operation and maintenance costs. 

  Success stories/lessons learnt. 

 
 
 
Annex 5: Sample of actions 

The  purpose  of  the  interim  evaluation  is  to  provide  an  overall  assessment  of  the 
performance of ISA2. A sample of actions has therefore been selected to better guide the 
data collection activities.139 More specifically, the evaluation is based on a sample of 20 
of  the  53  actions  included  in  the  2018  Rolling  Work  Programme.  The  following  four 
criteria were used when selecting the sample: 

1.  Action packages: the selected actions should be largely representative of the nine 
ISA2 packages of actions, as they are defined in the Rolling Work Programme.  

2.  New actions and actions continued from ISA: within each action package, the 
selected  actions  should  be  representative  of  two  clusters,  namely  those  actions 
that have been continued from ISA and those actions that have been started under 
ISA2. 

3.  Action start date: to provide the evaluation with enough data, no action started 

in 2018 should be selected.  

4.  Action  end  date:  to  the  extent  possible,  those  actions  that  have  already  ended 

should be included in the sample. 

To build the sample, a number of steps were taken. First, the 53 actions funded by ISA2 
were  grouped  according  to  the  packages  to  which  they  belong.  Second,  within  each 
package, the evaluation team identified those actions that were continued from ISA and 
those that were created under ISA2. Third, a target sample size of 20 actions (i.e. more 
than  one-third  of  the  total)  was  set  to  ensure  the  feasibility  of  the  evaluation  while 
allowing  for  a  comprehensive  sample  of  typical  actions.  Then  the  evaluation  team 
identified the number of actions to be selected in order to reflect: i) the proportions of 
actions from each package relative to the total action population; and ii) the proportion of 
old  and new actions compared to  the total  number of actions per package. The relative 
proportions  were  rounded  off,  taking  into  consideration  the  third  and  fourth  sampling 
criteria, namely the exclusion of actions started in 2018 and, to the extent possible, the 
inclusion of actions that have already ended (Table 7). The final sample is presented in 
Table  8,  which  also  provides  an  overview  of  the  solutions  stemming  from  selected 
actions. Sampled actions have been randomly selected to avoid any selection bias.  

139  Stakeholders  related  to  all  53  actions  were  invited  to  answer  the  online  surveys.  However,  desk 

research (review of documentary evidence) and interviews focused only on sampled actions. 

129 

 
                                                 
Table 7: Number of sampled actions by package and programme 

Package 

Number 
of 
actions 

Number of 
ISA 
actions 

Number of 
ISA2 actions 

Package actions / 
Total actions 
proportion 

Number of 
sampled 
actions 
(total) 

Number of 
sampled 
actions 
(ISA2) 

1. Key and generic interoperability enablers 
2. Semantic interoperability 
3. Access to data/data sharing/open data 
4. Geospatial solutions 
5. eProcurement/ eInvoicing - Supporting instruments 
6. Decision making and legislation - Supporting instruments 
7. EU Policies - Supporting instruments 
8. Supporting instruments for public administrations 
9. Accompanying measures 
TOTAL 
Note: The column ‘Number of ISA actions’ shows the number of actions continued under ISA2 from the programme’s previous editions. The column ‘Number of ISA2 actions’ shows the 
number of new actions under ISA2 that are not a direct continuation of any actions of the previous editions of the programme. 
Source: CEPS final study 

13% 
8% 
13% 
2% 
2% 
19% 
9% 
30% 
4% 
100% 

5 
1 
2 
1 
1 
5 
3 
11 
0 
29 

7 
4 
7 
1 
1 
10 
5 
16 
2 
53 

2 
3 
5 
0 
0 
5 
2 
5 
2 
24 

2 
2 
3 
1 
1 
3 
1 
6 
1 
20 

0 
1 
2 
0 
0 
2 
0 
1 
1 
7 

Number 
of 
sampled 
actions 
(ISA) 
2 
1 
1 
1 
1 
1 
1 
5 
0 
13 

Table 8: Sampled actions and relevant solutions 

Package 

1. Key and generic 
interoperability enablers 

Action Name 

Action 
number 
2016.19  Trusted Exchange 

ISA/ 
ISA2 
ISA 

Platform (e-
TrustEx) 

1. Key and generic 
interoperability enablers 

2016.29  Catalogue of 

ISA 

Services 

Solutions 

Common tool/service: 
Open e-TrustEx 

Common framework: 
Core Public Service 
Vocabulary 
Application Profile 
(CPSV-AP) 

130 

 
  
  
  
  
  
  
Package 

2. Semantic 
interoperability 

2. Semantic 
interoperability 

3. Access to data/data 
sharing/open data 

3. Access to data/data 
sharing/open data 

3. Access to data/data 
sharing/open data 

Action Name 

Action 
number 
2016.07  SEMIC: 

ISA/ 
ISA2 
ISA 

Promoting 
Semantic 
Interoperability 
Amongst the 
European Union 
Member States 

Solutions 

Common framework 
(data models, data 
standards): Core 
vocabularies 

Common 
specification/standard: 
Asset Description 
Metadata Schema 
(ADMS) 

Common 
specification: DCAT 
Application Profile for 
data portals in Europe 
(DCAT-AP), 
GeoDCAT-AP, 
StatDCAT-AP 

Common tool/service: 
VocBench3 

2016.16  Public 

ISA2  Common framework: 

PMKI Core data 
model for 
multilingual 
taxonomies/terminolo
gies 

Study: Big data 
analytics for policy 
making 

Common 
specification/standard: 
Semantic links - core 
dataset with additional 
semantic links 
between different 
language resources (in 
particular having 
EuroVoc as pivot) 
Common tool/service: 
DORIS - stakeholders' 
feedback analysis tool 

Study: Big Data Test 
Infrastructure 

ISA2  Common framework: 
ESS: Statistical 
Production Reference 
Architecture v1.0 

Multilingual 
Knowledge 
Management 
Infrastructure for 
the Digital Single 
Market 

2016.03  Big Data for 

ISA 

Public 
Administrations 

2016.06  Sharing Statistical 

Production and 
Dissemination 
Services and 
Solutions in the 
European 
Statistical System 

2016.18  Development of an 

ISA2  Common tool/service: 

Open Data 
Service, Support 
and Training 
Package in the 
Area of Linked 

Catalogue of data 
visualisation tools 
(part of EU Open 
Data portal) 

131 

 
  
  
  
  
  
  
  
  
  
Package 

Action 
number 

Action Name 

ISA/ 
ISA2 

Solutions 

Open Data, Data 
Visualisation and 
Persistent 
Identification 

4. Geospatial solutions 

2016.10  European Location 

ISA 

Interoperability 
Solutions for e-
Government 
(ELISE) 

Common tool/service: 
Re3gistry 

Common framework: 
EULF Blueprint 

5. eProcurement/ 
eInvoicing - Supporting 
instruments 

2016.05  European Public 

ISA 

Procurement 
Interoperability 
Initiative 

Common tool/service: 
Open e-Prior 

Common tool/service: 
eCertis 

Common tool/service: 
INSPIRE Reference 
Validators and 
Interoperability 
Testing (test 
framework) 

Common 
tool/services: 
Common services, 
pilots and 
applications;  

Common tool/service: 
European Single 
Procurement 
Document (ESPD) 
service as well as a 
data model on the 
ESPD 

6. Decision making and 
legislation - Supporting 
instruments 

6. Decision making and 
legislation - Supporting 
instruments 
6. Decision making and 
legislation - Supporting 
instruments 

2016.23  Legal 

ISA 

interoperability 
(former ICT 
Implications of EU 
Legislation) 

Common tool: Tool 
#27 of the Better 
Regulation Toolbox: 
The digital economy 
and society & ICT 
issues 

Common framework:  
ICT Impact 
Assessment 
Guidelines 

2017.03  REFIT Platform 

ISA2  Common tool/service: 

2017.04 

Inter-Institutional 
Register of 
Delegated Acts 
(RegDel) 

REFIT Platform IT 
Tool 

ISA2  Common tool/service: 
Register of Delegated 
Acts (RegDel) 

132 

 
  
  
  
  
  
  
 
  
  
  
Package 

7. EU Policies - 
Supporting instruments 

Action Name 

Action 
number 
2016.14  European Citizens’ 

ISA/ 
ISA2 
ISA 

Initiatives and 
European 
Parliament 
Elections 

Solutions 

Common tool/service: 
European Parliament 
Crypto Tool 

Common tool/service: 
Online Collection 
Software to support 
European Citizens’ 
Initiatives (OCS for 
ECIs) 
Common tool/service: 
Joinup 

8. Supporting 
instruments for public 
administrations 

8. Supporting 
instruments for public 
administrations 

8. Supporting 
instruments for public 
administrations 

8. Supporting 
instruments for public 
administrations 
8. Supporting 
instruments for public 
administrations 

8. Supporting 
instruments for public 
administrations 

ISA 

2016.20 

Joinup – European 
Collaborative 
Platform and 
Catalogue  
2016.21  National 

Interoperability 
Framework 
Observatory 

ISA 

Common framework: 
NIFO factsheets 

2016.32  European 

ISA 

Interoperability 
Architecture (EIA) 

2016.35  EUSurvey 

ISA 

2016.37 

ISA 

Interoperability 
Maturity 
Assessment of a 
Public Service 
(IMAPS) 

2017.01  Standard-Based 

ISA2 

Archival Data 
Management, 
Exchange and 

Common framework: 
European 
Interoperability 
Reference 
Architecture (EIRA) 
Common tool/service: 
EUSurvey 

Common tool/service: 
Interoperability 
Maturity Assessment 
of a Public Service 
(IMAPS) 
Study: Study on 
Standard-Based 
Archival Data 
Management, 

133 

Common framework: 
State of play of 
interoperability in 
Europe 
Common tool/service: 
CarTool 

Common tool/service: 
DORIS 

Common tool: 
Assessment tool 
offering support for 
the selection of IT 

 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
Package 

Action 
number 

Action Name 

Publication 

9. Accompanying 
measures 

2016.3 

Source: CEPS final study 

Raising 
Interoperability 
Awareness – 
Communication 
Activities 

ISA/ 
ISA2 

ISA2 

Solutions 

Exchange and 
Publication 
 Events organised by 
ISA2 

solutions for archives 
management. 
 Events in which ISA2 
participated 

134 

 
  
  
 
Annex 6: Supporting evidence from desk research 

This annex presents evidence collected from desk research of performance indicators, the rolling work programme, the information available online in the 
webpages dedicated to the individual ISA2 actions and solutions, and additional evidence received from the action owners of the 20 sampled actions. In 
the following section, evidence is presented by evaluation criterion and question. 

Effectiveness 

EQ2: How far are the ISA² programme's results in the process of achieving the programme's objectives? 

EQ3: Are there aspects that are more or less effective than others, and if so, what lessons can be drawn from this? 

Types of users and their geographical distribution 

Table 9: EU Member States using solutions provided by sampled ISA2 actions 

BE 

BG  CZ  DK  DE  EE 

IE 

ADMS 

Core 
Vocabularies 

CPSV-AP 

DCAT-AP 

eCertis 

EIRA 

e-PRIOR 

ESPD* 

Open e-
TrustEx** 
European 
Parliament 
Crypto Tool 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

E
L 

x 

x 

x 

x 

x 

ES 

FR  HR 

IT  CY  LV 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

135 

L
T 

x 

x 

x 

x 

LU  HU  MT  NL  AT  PL  PT  RO 

SL 

SK 

FI 

SE  UK  TOT. 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

3 

7 

6 

7 

x 

23 

7 

1 

23 

28 

28 

x 

x 

x 

x 

x 

x 

x 

x 

 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
BE 

BG  CZ  DK  DE  EE 

IE 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

E
L 

x 

x 

x 

EUSurvey 

GeoDCAT-AP 

IMAPS 

Joinup 

NIFO 

OCS for ECIs 

ES 

FR  HR 

IT  CY  LV 

L
T 

LU  HU  MT  NL  AT  PL  PT  RO 

SL 

SK 

FI 

SE  UK  TOT. 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

x 

1 

1 

2 

26 

26 

1 

Re3gistry 
*Note:  Information  provided  by  action  owner  based  on  the  following  Commission  document  on  the  CEF  eProcurement  Digital  Service  Infrastructure  (November  2018): 
https://ec.europa.eu/inea/sites/inea/files/dsi_fiche_eprocurement.pdf  
**Note: Information provided by action owner. 
Source: CEPS final study, which built on the data available on ISA2 Solutions webpage as of January 2019. 

x 

6 

x 

x 

x 

x 

x 

Table 10: EU institutions using solutions provided by sampled ISA2 actions 

ADMS 
Core 
Vocabularies 
DCAT-AP 
eCertis 
EIRA and 
CarTool 
e-TrustEx 
EUSurvey 
GeoDCAT-AP 
Open e-Prior 
Re3gistry 
RegDel 

European 
Commission 

Council of the 
EU 

European 
Parliament 

Publications 
Office of the EU 
x 

x  

x  
x 

x 

x 
x  
x  
x 
x 
x 

x 

x 

x 

x 

x 

x 

x 
x 

x 

x 

x 
x 

x 

136 

European 
Council 

European 
Central Bank 

TOTAL 

x 

x 

x 
x 

1 

2 

2 
2 

3 

6 
4 
1 
2 
1 
4 

 
  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
VocBench3 
TOTAL 

European 
Commission 
x  
11 

*Note: Information provided by action owner. 
Source: CEPS final study 

Council of the 
EU 

European 
Parliament 

3 

3 

Publications 
Office of the EU 
x 
9 

European 
Council 

European 
Central Bank 

2 

2 

TOTAL 

2 
30 

Table 11: Status of sampled actions and their cross-border feature 

Package 

1. Key and generic 
interoperability enablers 

1. Key and generic 
interoperability enablers 
2. Semantic 
interoperability 

2. Semantic 
interoperability 

3. Access to data/data 
sharing/open data 
3. Access to data/data 
sharing/open data 

Action 
number 
2016.19  Trusted Exchange Platform 

Action Name 

Operational 
solutions 
1 

(e-TrustEx) 

2016.29  Catalogue of Services 

2016.07 

2016.16 

SEMIC: Promoting Semantic 
Interoperability Amongst the 
European Union Member 
States 

Public Multilingual 
Knowledge Management 
Infrastructure for the Digital 
Single Market 

2016.03  Big Data for Public 

2016.06 

Administrations 
Sharing Statistical 
Production and 
Dissemination Services and 
Solutions in the European 
Statistical System 

1 

4 

- 

3 

2 

137 

Out of which cross-
border solution 

1 

1 

2 

- 

3 

2 

Solutions under 
development 
- 

- 

- 

4 

1 

6 

 
 
 
 
 
 
 
Package 

3. Access to data/data 
sharing/open data 

Action Name 

Action 
number 
2016.18  Development of an Open 
Data Service, Support and 
Training Package in the Area 
of Linked Open Data, Data 
Visualisation and Persistent 
Identification 

4. Geospatial solutions 

2016.10  European Location 

5. e-Procurement/ e-
invoicing - Supporting 
instruments 
6. Decision making and 
legislation - Supporting 
instruments 
6. Decision making and 
legislation - Supporting 
instruments 
6. Decision making and 
legislation - Supporting 
instruments 
7. EU Policies - Supporting 
instruments 

Interoperability Solutions for 
e-Government (ELISE) 

2016.05  European Public 

Procurement Interoperability 
Initiative 
2016.23  Legal interoperability 

(former ICT Implications of 
EU Legislation) 
2017.03  REFIT Platform 

2017.04 

Inter-Institutional Register of 
Delegated Acts (RegDel) 

2016.14  European Citizens’ 

Initiatives and European 
Parliament Elections 
Joinup – European 
Collaborative Platform and 
Catalogue  

8. Supporting instruments 
for public administrations 

2016.20 

8. Supporting instruments 
for public administrations 

2016.21  National Interoperability 
Framework Observatory 

Operational 
solutions 

Out of which cross-
border solution 

Solutions under 
development 

1 

4 

3 

- 

- 

- 

2 

1 

3 

1 

4 

3 

2 

- 

1 

2 

1 

3 

138 

1 

1 

5 

- 

1 

- 

- 

- 

- 

 
Package 

8. Supporting instruments 
for public administrations 
8. Supporting instruments 
for public administrations 
8. Supporting instruments 
for public administrations 

Action 
number 
2016.32  European Interoperability 

Action Name 

Architecture (EIA) 

2016.35  EUSurvey 

2016.37 

Interoperability Maturity 
Assessment of a Public 
Service (IMAPS)  

8. Supporting instruments 
for public administrations 

2017.01 

9. Accompanying measures 

2016.3 

Standard-Based Archival 
Data Management, Exchange 
and Publication 
Raising Interoperability 
Awareness – Communication 
Activities 

Operational 
solutions 

Out of which cross-
border solution 

Solutions under 
development 

2 

2 

1 

2 

2 

1 

1 

2 

1 

- 

- 

1 

The full overview of events organised is listed  
in Annex 1: Table 13. 

Source: European Commission’s own elaboration 

TOTALS 

35 

29 

21 

Performance indicators 

Table 12: Overview of performance indicators for sampled ISA2 actions 

Package 

ISA / 
ISA2 

Action 
number 

Solution 

ISA 

2016.19 

e-TrustEx 

1. Key and 
generic 
interoperability 
enablers 

Value 
performance 
indicator 1 
200 

Performance indicator 1 

Public administrations in the 28 
Member States using this solution 

Value 
performance 
indicator 2 
8,800,000 

Performance indicator 2 

Number of documents 
exchanged between the EC, the 
Council, the Member States, 
and companies 

139 

 
 
 
Package 

ISA / 
ISA2 

Action 
number 

Solution 

ISA 

2016.29 

CPSV-AP 

Value 
performance 
indicator 1 
12 

1. Key and 
generic 
interoperability 
enablers 
2. Semantic 
interoperability 

2. Semantic 
interoperability 

2. Semantic 
interoperability 

2. Semantic 
interoperability 

ISA 

2016.07 

ADMS 

ISA 

2016.07 

Core vocabularies 

ISA 

2016.07 

DCAT-AP 

9 

11 

29 

ISA 

2016.07 

VocBench3  

14 

2. Semantic 
interoperability 

ISA2 

2016.16 

2. Semantic 
interoperability 

ISA2 

2016.16 

PMKI Core data 
model for 
multilingual 
taxonomies/ 
terminologies 
Semantic links  

2 journal 
articles; 5 
international 
conference 
proceedings 
4 

Value 
performance 
indicator 2 
2,511 

Performance indicator 2 

Number of downloads on 
Joinup (up until March 2019) 

4,700 

Number of interoperability 
solutions on Joinup described 
using ADMS 

Performance indicator 1 

Public administrations in Member 
States using this solution, 
including cross-border catalogues 

Public administrations (Member 
States, EC), company – using this 
solution 
Public administrations (in the 
Member States and EC) using this 
solution 
Public administrations (in the 
Member States and EC) and their 
data portals, associations, 
universities – using this solution 
Number of public administrations 
(in the Member States and EC), 
universities, institutes, 
international organisations using 
the previous versions of 
VocBench. 
Instances of inclusion in 
academic journals and 
international conferences 

Collaborations with European 
public administrations to establish 
semantic interoperability between 
national language resources and 

140 

 
  
  
  
  
  
  
  
  
  
  
Package 

ISA / 
ISA2 

Action 
number 

Solution 

Value 
performance 
indicator 1 

Performance indicator 1 

Value 
performance 
indicator 2 

Performance indicator 2 

3. Access to 
data/data 
sharing/open 
data 
4. Geospatial 
solutions 
5. eProcurement/ 
eInvoicing - 
Supporting 
instruments 
5. eProcurement/ 
eInvoicing - 
Supporting 
instruments 
5. eProcurement/ 
eInvoicing - 
Supporting 
instruments 
6. Decision 
making and 
legislation - 
Supporting 
instruments 

ISA2 

2016.06 

ESS Service 
Catalogue 

ISA 

2016.10 

Re3gistry 

ISA 

2016.05 

Open e-Prior 

ISA 

2016.05 

eCertis 

10 

13 

70 

44 

ISA 

2016.05 

ESPD 

195,652 

ISA 

2016.23 

280 

Digital screening 
mention in Tool 
#27 of the Better 
Regulation 
Toolbox: The 
digital economy 
and society & ICT 
issues 
ICT Impact 

EuroVoc (3 Member States and 
the EP) 
Number of registered contributors 

Public administrations (Member 
States, EC) using this solution 
Public administrations (EC and 
other EU institutions) using this 
solution 

Public administrations (Member 
States, EU institutions), 
contracting authorities, 
companies – using this solution 
Number of page visits in January 
2019 
(around 4 million visits until 
April 2019) 
Number of Commission 
proposals screened for ICT 
impacts since 2015 

13 

Number of statistical 
organisations listed 

306 

Number of suppliers connected 
via the web portal 

23 

Number of Member States 
implementing the ESPD data 
model;  

6. Decision 

ISA 

2016.23 

13 

Instances of use since 2014 

141 

 
  
  
  
  
  
  
  
  
making and 
legislation - 
Supporting 
instruments 
6. Decision 
making and 
legislation - 
Supporting 
instruments 
6. Decision 
making and 
legislation - 
Supporting 
instruments 
7. EU Policies - 
Supporting 
instruments 

7. EU Policies - 
Supporting 
instruments 
8. Supporting 
instruments for 
public 
administrations 
8. Supporting 
instruments for 
public 
administrations 

Package 

ISA / 
ISA2 

Action 
number 

Solution 

Value 
performance 
indicator 1 

Performance indicator 1 

Value 
performance 
indicator 2 

Performance indicator 2 

Assessment 
Guidelines 

ISA2 

2017.03  REFIT Platform IT 

Tool 

Not yet 
operational. 

ISA2 

2017.04 

RegDel 

144,587 

Number of page views since the 
launch up until 1 February 2019 

2,202 

Number of active subscriptions 

ISA 

2016.14  Online Collection 

38 

Number of ECIs launched using 
the Online Collection Software 

1 

Public administration in the 
Member States 

Software to 
support European 
Citizens' Initiative 
European 
Parliament Crypto 
Tool 
Joinup 

ISA 

2016.14 

ISA 

2016.20 

28 

Number of Member States using 
the solution 

13,440 

Professionals working in the field 
of eGovernment registered on 
Joinup 

ISA 

2016.21 

NIFO 

193 

Number of users for Q3 2018, 
according to ISA2 Dashboard - 
Effectiveness Indicator 
(approximately equal to the 

142 

2,815 

34 

Solutions created within the 
109 Collections (thematic 
collaborative spaces) on Joinup. 

Number of countries covered 
(32 countries covered by NIFO 
factsheets and 34 by the 
Interoperability State of Play 

 
  
  
  
  
  
Value 
performance 
indicator 2 

Performance indicator 2 

reports) 

1,364 

Number of downloads of EIRA 
on Joinup up to February 2019 

68 (in 2017); 
51 (in 2018) 

Public services benchmarked 
(Q3 2017) 

Package 

ISA / 
ISA2 

Action 
number 

Solution 

Value 
performance 
indicator 1 

Performance indicator 1 

8. Supporting 
instruments for 
public 
administrations 
8. Supporting 
instruments for 
public 
administrations 

8. Supporting 
instruments for 
public 
administrations 
8. Supporting 
instruments for 
public 
administrations 

8. Supporting 
instruments for 
public 
administrations 

ISA 

2016.32  EIRA and CarTool 

8 

ISA 

2016.35 

EUSurvey 

17,443 

ISA 

2016.37 

IMAPS 

ISA2 

2017.01 

ISA2 

2017.01 

Study on Standard-
based Archival 
Data Management, 
Exchange, and 
Publication 
Assessment tool 
offering support 
for the selection of 
IT solutions for 
archives 
management 

17 

80 

58 

Source: CEPS final study 

number of members of the NIFO 
community on Joinup) 
Number of public administrations 
in Member States deploying the 
solution 

Number of surveys created (sum 
for 2016, 2017, Q1 and Q3 of 
2018), based on the ISA2 
solutions webpage and the 2018 
Rolling Work Programme. 
Trans-European Systems assessed 
with IMM 

Unique visitors/downloads since 
the publication (2018) 

Unique visitors/downloads since 
the publication (2018) 

143 

 
 
 
  
  
  
  
 
Communication activities 

Table 13: Events funded by ISA2 between 2016 and 2018 

Year 

2016 

2016 
2017 
2017 

2017 

2018 

2018 

2018 

2018 

2018 

Source: CEPS final study 

ISA2 
workshops/conferences 

Location 

Total 
participant
s onsite 

Total remote 
participants 

Number of 
international 
participants 

Number of 
Member 
States 
represented 

From ISA to ISA2 

SEMIC2016* 
Workshop on the EIF 
Sharing and reuse 
Conference 
SEMIC2017 

Open PM2 Conference 
(partially funded by 
ISA2) 
SEMIC 2018 

ISA2CONF18 

Workshop organised as 
part of the European 
Week of Regions and 
Cities 2018 
Workshop organised as 
part of the Inspire 
Conference 

Brussels, 
Belgium 
Rome, Italy 
Thessaloniki 
Lisbon, 
Portugal 
Valletta, 
Malta 
Brussels, 
Belgium 

Sofia, 
Bulgaria 
Brussels, 
Belgium 
Brussels, 
Belgium 

Antwerp, 
Belgium 

300 

206 
N/A 
220 

224 

538 

220 

325 

98 

95 

144 

- 

- 
- 
810 

609 

1,800 
(connections to 
web streaming) 
701 

- 

- 

- 

- 

- 
- 
204 

195 

- 

204 

249 

- 

- 

- 

25 
- 
21 

18 

- 

19 

26 

- 

- 

Number of 
EEA and 
acceding 
countries 
represented 
- 

Number 
of EC 
officials 
(non-
DIGIT) 
- 

3 
- 
0 

2 

- 

2 

7 

- 

- 

- 
- 

- 

120 

- 

- 

 
 
 
 
 
 
  
 
  
 
 
  
 
 
 
Table 14: Events in which ISA2 representatives played an active role 

Year 
2016 
2016 
2016 
2016 
2016 
2016 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 
2017 

Events  
Ljubljana ICT Procurement workshop 
Nordic Digital Day 
Open Source Summit Paris 
Digital Stakeholders Forum 
Digitec16 
Manage IT 2016 
Open Belgium 
SG IT Day 
Digital Day Rome 
ECI Day 2017 
ICT Spring Luxembourg 2017 
Conference Krems 
Digital Assembly 2017 
Semantics Conference 2017 
ICA Conference 2017 
Inspire Conference 2017 
Data for Policy 
DK Architecture Conference 
Jornada sobre Interoperabilidad y Archivo Electronico 
Digitalisseringsmessen 17 
Egov Conference Tallinn, 2017 
3rd Annual Public Sector Transformation Conference 
3rd ELRC Conference 
ICT Proposers Day 2017 
Metaforum 2017 
Informatika v Javni Upravi 

145 

Location 
Ljubljana, Slovenia 
Tallinn, Estonia 
Paris, France 
Brussels, Belgium 
Brussels, Belgium 
Antwerp, Belgium 
Antwerp, Belgium 
Brussels, Belgium 
Rome, Italy 
Brussels, Belgium 
Luxembourg, Luxembourg 
Krems, Austria 
Valletta, Malta 
Amsterdam, The Netherlands 

Strasbourg, France 
London, UK 
Copenhagen, Denmark 
Madrid, Spain 
Odense, Denmark 
Tallinn, Estonia 
Brussels, Belgium 
Brussels, Belgium 
Budapest, Hungary 
Brussels, Belgium 
Brdo, Slovenia 

Total participants 
N/A 
N/A 
180 
150 
500 
N/A 
150 
200 
600 
150 
5,000 
N/A 
5,000 
200 
N/A 
3,000 
200 
N/A 
180 
350 
200 
N/A 
N/A 
150 
N/A 
175 

 
 
Year 
2017 
2017 
2017 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 
2018 

Events  
Paris Open Source Summit 2017 
CEF Conference 
eDemocracy Conference 
Connected Smart Cities Conference 
Flosscon 
GDPR Conference 
Digital Czech Republic 
Interop Summit 2018 
Open Belgium 
RDA Berlin 
IESA 2018 
CNIS2018 
Digital Day 
CEEE Gov Days 2018 
German Federal Level Conference 
Good Governance Conference 2018 
Civil Society Days 2018 
Language Technology Industry Summit 
Egov Conference Tallinn, 2018 
Conference Supervisory reporting for the Digital Age 
Digital Assembly 2018 
ICA Conference 2018 
ICT implications presentation in Vienna 
Inspire Conference 2018 
TOOP Conference 
eGov High Level Conference 2018 
Infofest Montenegro 

146 

Location 
Paris, France 
Brussels, Belgium 
Athens, Greece 
Brussels, Belgium 
Brussels, Belgium 
Berlin, Germany 
Prague, Czech Republic 
Brussels, Belgium 
Louvain La Neuve, Belgium 
Berlin, Germany 
Berlin, Germany 
Madrid, Spain 
Rome, Italy 
Budapest, Hungary 
Berlin, Germany 
Brussels, Belgium 
Brussels, Belgium 
Brussels, Belgium 
Tallinn, Estonia 
Brussels, Belgium 
Helsinki, Finland 
Sofia, Bulgaria 
Vienna, Austria 
Antwerp, Belgium 
Vienna, Austria 
Vienna, Austria 
Podgorica, Montenegro 

Total participants 
150 
150 
100 
500 
200 
150 
250 
150 
250 
200 
250 
300 
300 
150 
120 
650 
200 
200 
300 
150 
125 
5,000 
120 
900 
150 
N/A 
150 

 
Year 
2018 
2018 
2018 
2018 
2018 
2018 
2018  Webinar on Government Transformation: ‘How co-creation will shape 

Events  
European Week of Regions and Cities 2018 
GovTech Summit 2018 
Digitec18 
ICT Vienna 
Paris Open Source Summit 2018 
Symposium on Digital Transformation of the public sector 2018 

2018 

the future of value creation in the public sector’ 
1st CEF eTranslation Conference 

Source: CEPS final study 

Location 
Brussels, Belgium 
Paris, France 
Brussels, Belgium 
Vienna, Austria 
Paris, France 
Belgium 
Belgium 

Brussels, Belgium 

Total participants 
6,000 
3,000 
900 
5,000 
200 
200 
200 

N/A 

147 

 
Alignment with principles spelled out in Article 4(b) of the ISA2 Decision 

Table 15: Contribution of ISA2 actions to the principles listed in Art. 4(b) of the 
ISA2 Decision 

Art. 4(b) Principles 

Subsidiarity and 
proportionality 

User-centricity 

Inclusion and accessibility 

Delivery of public services in 
such a way as to prevent 
digital divide 

Security, respect for privacy 
and data protection 

Multilingualism 

Relevant evidence 
ISA2 aims to contribute to a holistic interoperability landscape by 
creating  solutions  at  EU  level.  The  adoption  of  solutions  by 
European public administrations is voluntary. 

As  part  of the  process  of proposing  actions to  be  included  in  the 
rolling  work  programme,  all  proposed  actions  must  take  user-
centricity into account and describe the ways in which user input 
is integrated into the action and the development of solutions.  
ISA2 uses a number of channels to disseminate information about 
the actions and the solutions developed. These include: i) the ISA2 
website,  which  contains  information  on  each action  and  solution, 
official  documents,  infographics,  videos,  presentations,  training 
material,  and  publications;  ii)  the  Joinup  platform  to  share  the 
solutions  developed  under  ISA2;  iii)  events  organised  by  ISA2  in 
various Member States where promotional material is distributed; 
and  iv)  events  with  ISA2  participation  in  various  Member  States 
where  promotional  material  is  distributed.  In  addition,  solutions 
take  into  account  potential  barriers to  accessibility  and  inclusion, 
detailing such barriers where relevant and providing solutions. An 
example is the ‘EU CAPTCHA’140 action, which aims to address 
the  issues  of  accessibility  of  CAPTCHA  tests  by  exploring 
alternative solutions to improve the user experience. 

As  part  of the  process  of proposing  actions to  be  included  in  the 
rolling  work  programme,  all  the  proposed  actions  must  specify 
their expected contribution to the EU’s higher political priorities, 
such  as  the  Digital  Single  Market.  Where  applicable,  the  rolling 
work  programme  details  how  specific  actions  contribute  to  the 
third  pillar  of  the  Digital  Single  Market  Strategy,  which  also 
includes support for an inclusive digital society. 

Where  appropriate,  the  descriptions  of  actions  included  in  the 
rolling  work  programme  must  specify  the  measures  taken  to 
ensure security, respect for privacy and data protection.  
Examples of ISA2 actions that produce tools and frameworks that 
contribute 
‘PMKI’; 
‘Development  of  an  open  data  service,  support  and  training 
package  in  the  area  of  linked  open  data,  data  visualisation  and 
persistent identification’, and ‘EUSurvey’. 

to  multilingualism 

‘SEMIC’; 

are: 

140  ‘CAPTCHA’  stands  for  ‘Completely  Automated  Public  Turing  Test  to  tell  Computers  and  Humans 
Apart’ and it represents a test that is expected to be difficult for machines to complete correctly, but 
possible for humans to complete correctly. 

148 

 
 
                                                 
Administrative 
simplification and 
modernisation 

Transparency 

Preservation of information 

Openness 

Re-usability and avoidance 
of duplication 

Technological neutrality, 
solutions which, insofar as 
possible, are future-proof, 
and adaptability 

Effectiveness and efficiency 

Source: CEPS final study 

Administrative  simplification  and  modernisation  are  taken  into 
consideration  in  the  rolling  work  programme  of  ISA2  under  the 
section ‘main impact list’. The detailed list of expected impacts for 
each  action  indicates  the  extent  to  which  the  proposed  actions 
contribute to this aspect. 
Various  channels  ensure  the  transparency  of  ISA2  actions.  These 
include: i) the rolling work programme that provides an overview 
of  each  action’s  objectives,  planned  and  developed  solutions, 
expected  impacts,  and  budget;  ii)  the  ISA2  Dashboard  that 
provides  quarterly  updates  on  the  efficiency  and  effectiveness  of 
actions, in terms of costs, earned value management, effectiveness 
targets;  iii)  a  specific  ISA2  webpage  with 
indicators  and 
information on the level of take-up of solutions; and iv) the Joinup 
platform where developed solutions can be accessed.  
The  ‘Library’  section  of  the  ISA2  website  preserves  information 
about  ISA2  in  various  formats  including  presentations,  videos, 
publications, 
training  course 
materials,  press  releases,  articles,  ISA2  work  programmes, 
speeches, and official documents.  
The publication of ISA2 solutions on Joinup ensures that solutions 
are openly available for (potential) users. 

infographics,  posters, 

leaflets, 

As  part  of  the  proposal  process  for  actions  to  be  included  in  the 
rolling  work  programme,  the  descriptions  of  proposed  actions 
specify  (i)  the  extent  to  which  the  action  proposed  reuses  other 
readily  available  solutions  and  (ii)  the  reusability  of  the  action 
outputs. 

among  others, 

landscape’  describes, 

The  rolling  work  programme’s  section  on  ‘Contribution  to  the 
the 
interoperability 
contributions  made  by  actions  to  the  European  Interoperability 
Framework,  aspects  of  which  include  technological  neutrality, 
future-proof  solutions  and  adaptability.  As  such,  this  principle  is 
already taken into account in the proposal phase. 
The ISA2 Dashboard provides quarterly data on the efficiency and 
effectiveness of actions. The historic data can be accessed together 
with the most recent data available. 

149 

 
Efficiency 

EQ4: To what extent has the programme been cost-effective? 

Table 16: Labour costs per hour in euro, services in 2017 

Country 

Belgium 
Bulgaria 
Czech Republic 
Denmark 
Germany  
Estonia 
Ireland 
Greece 
Spain 
France 
Croatia 
Italy 
Cyprus 
Latvia 
Lithuania 
Luxembourg 
Hungary 
Malta 
Netherlands 
Austria 
Poland 
Portugal 
Romania 
Slovenia 
Slovakia 
Finland 
Sweden 
United Kingdom 
Note:  The  table  details  the  hourly  labour  costs  for  the  service  sector  at  Member  State  level.  Such  costs 
include wage and non-wage costs net of subsidies. They do not include vocational training costs or other 
expenditures such as recruitment costs, spending on working clothes, etc.141 
Source: Eurostat. 

€/h 
41.1 
5.2 
11.5 
43.8 
31.5 
12.5 
28.9 
14.3 
20.0 
36.4 
11.4 
27.4 
16.5 
8.8 
8.5 
40.6 
9.3 
13.5 
32.9 
33.5 
9.3 
14.5 
6.7 
17.5 
11.4 
32.0 
40.9 
25.2 

141 

For 

https://ec.europa.eu/eurostat/statistics-
explained/index.php/Hourly_labour_costs#Source_data_for_tables_and_figures_on_this_page_.28MS 

further 

details 

see: 

150 

 
 
 
                                                 
Coherence 

EQ6: To what extent do the ISA² actions form part of a ‘holistic’ approach within the 
framework of the programme? (Internal coherence) 

Table 17: Number of links between sampled actions 

Actions 

Number of other ISA² 
solutions that are used 
by the action 

Number of other ISA² 
actions that use the 
solutions of the action 

1. Key and generic interoperability enablers 

Trusted Exchange Platform (e-TrustEx) 
Catalogue of Services 

4 
2 

2. Semantic Interoperability 

Public Multilingual Knowledge 
Management Infrastructure for the 
Digital Single Market (PMKI) 
SEMIC: Promoting Semantic 
Interoperability Amongst the European 
Union Member States 

1 

5 

3. Access to data/data sharing/open data 

Big Data for Public Administrations 
Sharing Statistical Production and 
Dissemination Services and Solutions in 
the European Statistical System 
Development of an Open Data Service, 
Support and Training Package in the 
Area of Linked Open Data, Data 
Visualisation and Persistent 
Identification 

5 
3 

1 

European Location Interoperability 
Solutions for e-Government (ELISE) 

6 

4. Geospatial Solutions 

5. eProcurement/eInvoicing-Supporting instruments 

European Public Procurement 
Interoperability Initiative 

6** 

6. Decision making and legislation-Supporting instruments 

ICT Implications of EU Legislation 
REFIT Platform 
Inter-Institutional Register of Delegated 
Acts 

6 
0 
1 

7.  EU Policies-Supporting instruments 

European Citizens' Initiatives and 
European Parliament Elections 

0 

8.  Supporting instruments for public administrations 

Joinup - European Collaborative 
Platform and Catalogue 
National Interoperability Framework 
Observatory 
European Interoperability Architecture 
(EIA) 

4 

7 

7 

151 

5 
3 

0 

17 

0 
0 

0 

1* 

4 

0 
2 
1 

0 

14 

0 

9 

 
EUSurvey 
Interoperability Maturity Model 
Standard-Based Archival Data 
Management, Exchange and Publication 

0 
9 
0 

9. Accompanying measures 

0 
2 
1 

Raising Interoperability Awareness - 
Communication Activities 
*Note:  For  the  ‘ELISE’  action,  further  information  was  provided  by  the  action  owner  regarding  the 
number of other ISA² actions that use the solutions of ELISE. 
**Note: For the ‘European Public Procurement Interoperability Initiative’ action, further information was 
provided by the action owner regarding the number of other ISA² solutions that are used by the action. 
Source: CEPS final study 

0 

1 

EQ7: To what extent is the ISA² programme coherent with other EU interventions that 
have similar objectives and with global initiatives in the same field? (External coherence) 

Table 18: Number of links between the sample actions and other EU programmes / 
policies / initiatives 

Actions 

Number of other 
EU programmes / 
policies / initiatives 
that the action relies 
on 

Number of other 
EU programmes / 
policies / initiatives 
that use the 
solution(s) provided 
by the action 

1. Key and generic interoperability enablers 

Trusted Exchange Platform (e-TrustEx) 

Catalogue of Services 

5 

4 

2. Semantic Interoperability 

Public Multilingual Knowledge Management 
Infrastructure for the Digital Single Market 
(PMKI) 

SEMIC: Promoting Semantic Interoperability 
Amongst the European Union Member States 

1 

6 

3. Access to data/data sharing/open data 

Big Data for Public Administrations 

Sharing Statistical Production and Dissemination 
Services and Solutions in the European Statistical 
System 

Development of an Open Data Service, Support 
and Training Package in the Area of Linked Open 
Data, Data Visualisation and Persistent 
Identification 

European Location Interoperability Solutions for 
e-Government (ELISE) 

4. Geospatial Solutions 

0 

12 

3 

3 

5. eProcurement/e-invoicing-Supporting instruments 

7* 

2 

1 

2 

2 

2 

1 

3** 

152 

 
 
European Public Procurement Interoperability 
Initiative 

7 

6. Decision making and legislation-Supporting instruments 

ICT Implications of EU Legislation 

REFIT Platform 

Inter-Institutional Register of Delegated Acts 

3 

0 

0 

European  Citizens' 
Parliament Elections 

7. EU Policies-Supporting instruments 

Initiatives  and  European 

2 

8. Supporting instruments for public administrations 

Joinup - European Collaborative Platform and 
Catalogue 

National Interoperability Framework Observatory 

European Interoperability Architecture (EIA) 

EUSurvey 

Interoperability Maturity Model 

Standard-Based Archival Data Management, 
Exchange and Publication 

1 

0 

3 

1 

0 

0 

9. Accompanying measures 

Raising Interoperability Awareness - 
Communication Activities 
*Note: Additional information was retrieved from the 2019 rolling work programme. 
**Note: Additional information was provided by the actions owners of the ‘ELISE’ action. 
Source: CEPS final study 

1 

0 

2 

1 

2 

0 

1 

1 

1 

0 

1 

4 

1 

Alignment with the DSM Strategy 

Table 19: ISA2 contribution to the DSM Strategy 
Contribution to Pillar I142 of the DSM Strategy 

Action 
number 

Action name 

2016.29 

Catalogue of Services 

2016.07 

SEMIC: Promoting 
Semantic Interoperability 
Amongst the European 
Union Member States 

Source: CEPS final study 

Contribution to Pillar I: Better access for consumers 
and businesses to online goods and services across 
Europe 
Obliges Member States to create Points of Single 
Contact with combined information for business setup 
and other citizen formalities. 
Semantic interoperability is a prerequisite for enacting 
most levels of systems’ interoperability, including the 
once-only principle and ensuring open data. 

142  Access: better access for consumers and businesses to digital goods and services across Europe. 

153 

 
 
                                                 
Contribution to Pillar II143 of the DSM Strategy 

Action 
number 

2016.06 

2016.05 

Action name 

Sharing Statistical 
Production and 
Dissemination Services and 
Solutions in the European 
Statistical System 
European Public 
Procurement Interoperability 
Initiative 

2016.20 

Joinup-European 
Collaborative Platform and 
Catalogue 

Source: CEPS final study 

Contribution to Pillar II: Creating the right 
conditions for digital networks and services to 
flourish 
A reusable solution for the dissemination of statistics for 
use by any administration to reduce the cost of 
dissemination and improve the delivery of data to public 
consumers 

The activities supported by this proposal will help to 
further create or improve standards regarding public 
procurement and encourage Member States to shift 
towards full eProcurement 
Joinup creates a central platform for observatory 
functionalities, collaborative features, and 
interoperability solutions 

Contribution to Pillar III144 of the DSM Strategy 

Action 
number 
2016.19 

2016.16 

Action name 

e-TrustEx 

Public Multilingual 
Knowledge Management 
Infrastructure for the Digital 
Single Market 

2016.15 

FISMA: Financial Data 
Standardization 

2016.18 

Development of an Open 
Data Service, Support, and 
Training Package in the Area 
of Linked Open Data, Data 
Visualisation and Persistent 
Identification 

2016.03 

Big Data for Public 
Administrations 

Contribution to Pillar III: Maximising the growth 
potential of our European Digital Economy 
Acts as a broker in the exchange of data and documents 
to foster greater interaction across heterogeneous 
systems 
The creation of a Public Multilingual Knowledge 
Infrastructure will support EU public administrations in 
creating services that can be accessible and shareable 
independently from the language actually used, as well 
as the SMEs to sell goods and service cross-border in a 
digital single market. 
Looks to work on ICT standardisation to reduce 
administrative costs of legacy and data systems in the 
financial sector. 
Open government data is a core asset for the knowledge-
based economy, since its reuse is a basis for innovative 
information products and services as it is a key enabler 
for transparency, evidence-based decision-making and a 
broader participation in the political discourse. This 
package will enable administrations to enhance their 
data visualisation capacity, to further open up their data 
as well as to increase data quality and interoperability in 
view of better data reusability. 
This action addresses the need to provide the right 
framework conditions for a single market for big data 
and cloud computing as a means for helping to 
accelerate the transition towards a data-driven economy. 

143  Environment: creating the right conditions and a level playing field for digital networks and innovative 

services to flourish. 

144  Economy & Society: maximising the growth potential of the digital economy. 

154 

 
                                                 
Action 
number 
2016.10 

Action name 

European Location 
Interoperability Solutions for 
e-Government (ELISE) 

2016.23 

Legal Interoperability 

2016.32 

2016.21 

European Interoperability 
Architecture (EIA) 
National Interoperability 
Framework Observatory 

2016.35 

EUSurvey 

2017.01 

Standard-Based Archival 
Data Management, Exchange 
and Publication 

2016.30 

Raising Interoperability 
Awareness-Communication 
Activities 

Source: CEPS final study 

Contribution to Pillar III: Maximising the growth 
potential of our European Digital Economy 
This action contributes to the interoperability landscape 
by ensuring that the ‘location’ dimension has an impact, 
adds value and is appropriately addressed within 
solutions across borders and sectors. 
The action has a horizontal value as it can be used for 
the law-making/evaluation of every EU policy. Special 
contribution is indirectly made to the DSM, as the more 
the action is assessing EU legislations the more digital 
and interoperable they become. 
The EIA looks to maintain the EIRA, an interoperability 
reference that is key to the once-only principle 
The EIF and the EIF Implementation Strategy foster 
interoperability and contribute to the DSM. By 
contributing to the implementation of the EIF, this 
action is relevant for the DSM. 
By offering an easy means of collecting opinions and 
information between heterogeneous parties, the 
EUSurvey considerably facilitates the organisation and 
consolidation of any types of 'feedback-based' decision 
By clarifying and supporting technical standards for 
archival management, it will ensure better access to 
digital goods and services and by providing access to 
Commission archives in Open Data format, it will 
generate value, allowing the reuse of this information 
producing new products and services. 
Communicating the ISA² programme and its results 
reinforces the programmes contributions to the DSM, 
the eGovernment action plan, the EIS and the new EIF, 
but also the Open Data Initiative of the European Union 
and INSPIRE, among other. 

Alignment with the Tallinn Declaration on eGovernment 

Box 10: Alignment between ISA2 and the Tallinn Declaration on eGovernment 

The  Tallinn  Declaration  is  a  statement  of  intents  in  which  the  32  signatory  countries  made  a 
commitment to move towards a digital government following five principles: 

1.  Digital-by-default, inclusiveness and accessibility 

2.  Once-only 

3.  Trustworthiness and security 

4.  Openness and transparency 

5.  Interoperability by default 

These principles inform the policy action lines, together with the additional line ‘Horizontal enabling 
policy steps’. Out of these five principles, principles #2, #4 and #5 are specifically reflected in the 
ISA2 programme. In particular:  

155 

 
 
  Principle  #2,  ‘once-only’, 

in  actions  such  as  ‘SEMIC’,  ‘European 
Interoperability Architecture’, and ‘European public procurement interoperability initiative’ 
that support the implementation of the principle. 

is  reflected 

  Principle  #4,  openness  and  transparency,  is  reflected  in  the  attention  paid to  Linked  Open 
like  ‘SEMIC’,  ‘PMKI’,  ‘ELISE’,  ‘Standard-Based  Archival  Data 
Data 
Management, Exchange and Publication’ of ISA2, even though in the Declaration the scope 
is wider and is specifically addressed to the openness of public institutions; and 

in  actions 

  Principle  #5,  interoperability  by  default,  is  directly  linked  to  the  objectives  of  ISA2 

programme.  

Furthermore, the policy action line ‘Horizontal enabling policy steps’ involves some aspects that are 
characteristic of the ISA2 programme, such as the digital transformation of public administrations and 
integration  of  digital  considerations  in  policy  initiatives.  Among  the  many  proposals  of  the 
Declaration, a few are directly coherent with ISA2 objectives:  

 

 

 

 

 

to implement the European Interoperability Framework and the Interoperability Action Plan 
(policy line 5); 

to  consider  strengthening  the  requirements  for  use  of  open  source  solutions  and  standards 
when (re)building of ICT systems and solutions takes place with EU funding (policy line 5); 

to  fully  integrate  digital  considerations  into  existing  and  future  policy  and  regulatory 
initiatives (policy line 6); 

to prepare proposals on the future (post 2020) and sustainability of existing EU-level cross-
border digital service infrastructures and building blocks (policy line 6); and 

to  convene  and  support  the  work  of  groups  of  interested  countries  and  other  parties  to 
exchange practices and develop reference guidelines and standards for taking emerging ICT 
into use in the public administration (policy line 6). 

From  this  standpoint,  there  is  no  contradiction  between  the  Tallinn  Declaration  and  the  ISA2 
programme. Although their scope and aims differ, they proceed in the same direction and share 
common intents. There may be synergies in the above-mentioned points, but they can also turn into 
overlaps and duplications, or even worse, misalignments, if there is no strong coordination. 

While the ISA2 programme is multi-national, the Declaration also affects each country individually. 
Therefore,  initiatives  and projects  may  be  promoted  separately  by  individual  countries to  fulfil  the 
commitments agreed upon by the Member States. The Declaration also includes specific calls to EU 
institutions to enhance interoperability within the EU framework. Overall, this document stresses the 
need  for  more  effort,  both  at  EU  institutions-level  and  at  Member  State-level,  to  ensure  citizen-
centric eGovernment and interoperability.  

Finally,  as  the  annex  of  the  Tallinn  Declaration  highlights  the  importance  of  principles  like  the 
‘Protection  of  personal  data  and  privacy’  and  ‘Incentives  for  digital  service  use’,  better  coherence 
would be ensured if the ISA2 programme began to include some of these principles in its actions. 

156 

 
 
EU added value 

EQ8: What is the additional value resulting from the ISA² programme, compared to what 
could reasonably have been expected from Member States acting at national, regional 
and/or local levels? 

Table 20: Number of participants in the ISA2 Coordination Group meetings (per 
Member State) 

Coordination 
Group Meeting 
13/10/2017 
Presence 
1 
1 
2 

Member State 

Belgium 
Bulgaria 
Czech 
Republic 
Denmark 
Germany 
Estonia 
Ireland 
Spain 
France 
Croatia 
Italy 
Cyprus 
Latvia 
Lithuania 
Luxembourg 
Hungary 
Malta 
Netherlands 
Austria 
Poland 
Portugal 
Romania 
Slovenia 
Slovakia 
Finland 
Sweden 
United 
Kingdom 
Norway 
Iceland 
Montenegro 
Greece 
Source: CEPS final study 

1 
2 
2 
2 
2 
1 
2 
1 
2 
2 
1 
2 
2 
3 
3 
2 
3 
2 
2 
1 
2 
2 
2 
1 

1 
1 
0 
0 

Coordination 
Group Meeting 
19/10/2016 
Presence 
1 
2 
2 

Coordination 
Group Meeting 
24/10/2018 
Presence 
1 
0 
1 

2 
3 
1 
1 
0 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
0 
0 
0 
0 
1 
1 
2 
1 

1 
0 
1 
0 

1 
2 
2 
1 
3 
1 
2 
1 
2 
2 
1 
2 
2 
2 
2 
2 
2 
2 
2 
1 
2 
2 
2 
2 

1 
0 
0 
0 

157 

 
Table 21: Number of participants in the ISA2 Committee meetings (per Member 
State) 

Committee 
Meeting 
02/03/2016 
Presence 
2 
3 
3 

Member 
State 

Belgium 
Bulgaria 
Czech 
Republic 
Denmark 
Germany 
Estonia 
Ireland 
Spain 
France 
Croatia 
Italy 
Cyprus 
Latvia 
Lithuania 
Luxembourg 
Hungary 
Malta 
Netherlands 
Austria 
Poland 
Portugal 
Romania 
Slovenia 
Slovakia 
Finland 
Sweden 
United 
Kingdom 
Norway 
Iceland 
Montenegro 
Greece 
Source: CEPS final study 

1 
3 
5 
1 
3 
1 
2 
2 
2 
2 
3 
3 
2 
2 
3 
3 
2 
2 
2 
2 
2 
3 
1 
3 

2 
0 
0 
0 

Committee 
Meeting 
09/06/2016 
Presence 
1 
2 
3 

Committee 
Meeting 
19/01/17 
Presence 
2 
2 
2 

Committee 
Meeting 
20/06/2018 
Presence 
1 
1 
1 

Committee 
Meeting 
24/01/2018 
Presence 
1 
2 
3 

1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

1 
1 
2 
1 

2 
3 
1 
2 
3 
1 
2 
2 
2 
2 
2 
3 
2 
2 
2 
3 
2 
2 
2 
2 
4 
2 
1 
1 

3 
1 
1 
3 

2 
4 
2 
2 
3 
1 
2 
2 
2 
3 
2 
3 
2 
2 
2 
2 
2 
1 
2 
2 
2 
2 
1 
2 

0 
0 
0 
0 

3 
3 
2 
3 
3 
1 
2 
2 
2 
2 
2 
3 
2 
2 
2 
2 
2 
2 
2 
2 
2 
3 
1 
2 

3 
1 
1 
0 

158 

 
 
 
Table 22: Contribution of sampled actions to cross-border interoperability 

Action Name 

Contribution to cross-border interoperability 

Action 
number 
2016.19 

Package 

1. Key and 
generic 
interoperability 
enablers 

Trusted 
Exchange 
Platform (e-
TrustEx) 

2016.29 

Catalogue of 
Services 

1. Key and 
generic 
interoperability 
enablers 
2. Semantic 
interoperability 

2016.07 

SEMIC: 
Promoting 
Semantic 
Interoperability 
Amongst the 
European 
Union Member 
States 

Public 
Multilingual 
Knowledge 
Management 
Infrastructure 
for the Digital 
Single Market 

2. Semantic 
interoperability 

2016.16 

3. Access to 
data/data 
sharing/open 
data 

2016.03 

Big Data for 
Public 
Administrations 

The e-TrustEx platform is currently used as a key 
element of pan-European messaging 
infrastructures for projects such as: e-PRIOR 
(DIGIT), DECIDE (SG), EDMA (COMP), 
eJustice Portal (JUST), OPOCE (OP), EU-CEG 
(DG SANTE), ESDEN (ESTAT). Around 200 
public institutions across the 28 Member States 
are in scope of these projects (such as national 
parliaments and permanent representations). 
The CPSV-AP is already being used by public 
administrations in Belgium, Italy, Finland, and 
Estonia to create a cross-border federated 
catalogue of public services. 
The proposal will facilitate the cross-border 
interoperability thanks to its inherent support for 
multilingualism. Further cross-border 
interoperability improvements can be expected 
through the alignment of the generic EuroVoc 
thesaurus that covers the EU policy domains with 
specialised EU and national controlled 
vocabularies through VocBench. VocBench is 
already used by public administrations in France, 
Italy, the Netherlands and Scotland and interest 
has been expressed from public administrations in 
Belgium and Slovenia. 
The objective of this action is to support 
enterprises and particularly the language 
technology industry with the implementation of 
the necessary multilingual tools and features in 
order to improve cross-border accessibility of e-
Commerce solutions. The outputs in the form of 
semantic links developed so far contribute to the 
cross-border service interoperability from a 
semantic perspective. 
In 2017, a long list of requirements has been 
collected from different Member States to 
understand their needs in the area of (Big) data 
analytics for policymaking, especially with regard 
to analytics use cases and infrastructure needs. 
The needs have been collected through a 
consultation of the ISA network, through the 
creation of a working group. Member States on 
board so far are the Czech Republic, Estonia, 
Hungary, Malta, Norway, Slovenia, Spain, and 
Portugal. The working group has shown 
significant interest in the action and its outputs. 
Additionally, the action has been presented to the 
ESS Big Data Task Force: the representatives 
have highlighted their interest in the action and 
shown availability to (re-)use its outcomes. 

159 

 
Action Name 

Contribution to cross-border interoperability 

Action 
number 
2016.06 

Package 

3. Access to 
data/data 
sharing/open 
data 

Sharing 
Statistical 
Production and 
Dissemination 
Services and 
Solutions in the 
European 
Statistical 
System 

3. Access to 
data/data 
sharing/open 
data 

4. Geospatial 
solutions 

5. 
eProcurement/ 
eInvoicing - 
Supporting 
instruments 

2016.18  Development 
of an Open 
Data Service, 
Support and 
Training 
Package in the 
Area of Linked 
Open Data, 
Data 
Visualisation 
and Persistent 
Identification 
European 
Location 
Interoperability 
Solutions for e-
Government 
(ELISE) 
European 
Public 
Procurement 
Interoperability 
Initiative 

2016.10 

2016.05 

The development of statistical services includes a 
broad international community. In the ESS, 14 
Member States are actively involved in a Task 
Force and a consortium of six Member States 
(FR, PT, UK, LT, DE, SI) has been set up to 
provide input and take part in the development of 
the guidelines for sharing of statistical services 
and to implement the reuse of developed solutions 
and services with the European Commission. 
In the architecture domain, the ESS reference 
architecture in its current state has been adopted 
by the 28 NSIs CIOs and Heads of Methodology. 
Its upgrading towards greater interoperability 
through more standards and deeper architectural 
guidance is done in collaboration with an ESS EA 
Board involving five Member States. 
Member States can consult and reuse the project 
outputs (knowledge base, trainings, the 
description of the tools and projects included in 
the catalogue of data visualisation tools). 

Road safety data-exchange solutions piloted and 
implemented in Norway and Sweden are being 
rolled-out to five other Member States using CEF 
funds, with more rollouts planned. INSPIRE 
Registry services have 450k accesses per 
quarter.* 
ePrior is used by several EU bodies and some 
components are used by the Belgian 
administration. In particular, in the EU Bodies 
context, economic operators using the ePrior 
system are from various EU countries. For 
eCertis: Roughly 25 services in the EU are 
retrieving data from eCertis, using the CEF eProc 
DSI. Other services will follow. For ESPD 
services: Private and public entities from roughly 
17 Member States have participated in the CEF 
eProc DSI Others are using Structural Funds to 
implement an ESPD service. They all have 
implemented or are currently implementing an 
ESPD service (incl. NO) using the ESPD data 
model or the open source code developed under 
the ISA2 project. In more than 20 Member States, 
there is already at least one ESPD service 
available.  

160 

 
Action Name 

Contribution to cross-border interoperability 

Action 
number 
2017.03 

Package 

6. Decision 
making and 
legislation - 
Supporting 
instruments 

REFIT 
Platform 

7. EU Policies 
- Supporting 
instruments 

2016.14 

2016.20 

8. Supporting 
instruments for 
public 
administrations 

European 
Citizens’ 
Initiatives and 
European 
Parliament 
Elections 
Joinup – 
European 
Collaborative 
Platform and 
Catalogue 

2016.21 

8. Supporting 
instruments for 
public 
administrations 

National 
Interoperability 
Framework 
Observatory 

8. Supporting 
instruments for 
public 
administrations 
8. Supporting 
instruments for 
public 
administrations 

8. Supporting 
instruments for 
public 
administrations 

9. 
Accompanying 
measures 

2016.32 

2016.35 

European 
Interoperability 
Architecture 
(EIA) 
EUSurvey 

2017.01  Standard-Based 
Archival Data 
Management, 
Exchange and 
Publication 
Raising 
Interoperability 
Awareness – 
Communication 
Activities 

2016.3 

The REFIT Platform consists of two Commission 
expert groups: a Government group in which all 
Member States are represented and a Stakeholder 
group with representatives of businesses, social 
partners, civil society organisations in various 
Member States, the Economic and Social 
Committee and the European Committee of the 
Regions. 
ECI-OCS serves the citizens and public 
administrations in all Member States as it 
facilitates the verification of the statements of 
support for legislation. The European Parliament 
Crypto tool is useful to the public administrations 
of all the Member States 
The end-users of Joinup are from different EU 
Member States and countries outside the EU 
(USA, Canada, New Zealand). In addition, several 
national repositories (NL, ES, EL, SL, BE) are 
stored on Joinup, making their national solutions 
available for re-use. 
The outputs of NIFO have already been reused by 
various Member States. The eGovernment 
factsheets are considered as a reference. The state 
of play reports on interoperability in Europe and 
the NIFO factsheets served as a source of input in 
the revision of the EIF and the IAP. 
EIRA has been deployed in EE, NL, DK, ES, CZ 
and PL. 

EUSurvey is available in 23 EU languages, 
facilitating cross-border interoperability. In 2017, 
more than 7,800 surveys have been created with 
the tool, resulting in more than 2.7 million 
contributions. 
The action addresses the semantic interoperability 
issue of how to describe electronic archives by 
means of the identification of existing standards 
for digital archives, facilitating the cross-border 
interoperability of electronic archives. 
This action contributes to the promotion of 
interoperability across the EU. 

*Note: Additional information was provided by the action owners of the ‘ELISE’ action. 
Source: CEPS final study 

161 

 
Alignment with the EIF 

Table 23: ISA2 contribution to the EIF, based on the sample of 20 actions 

Action Name 

Contribution to the EIF 

Package 

1. Key and generic 
interoperability 
enablers 

1. Key and generic 
interoperability 
enablers 

Action 
number 
2016.19 

Trusted Exchange Platform 
(e-TrustEx) 

2016.29 

Catalogue of Services 

2. Semantic 
interoperability 

2016.07 

2. Semantic 
interoperability 

2016.16 

SEMIC: Promoting Semantic 
Interoperability Amongst the 
European Union Member 
States 

Public Multilingual 
Knowledge Management 
Infrastructure for the Digital 
Single Market 

3. Access to 
data/data 
sharing/open data 

2016.03 

Big Data for Public 
Administrations 

e-TrustEx is a platform offered to public administrations at European, national and regional levels 
to undertake secure exchange of natively digital documents or scanned documents from system to 
system via standardised interfaces. The action thus contributes particularly to Recommendation 15 
of the revised EIF, through enabling the secure exchange of documents. 
The action addresses Recommendation 44 of the revised version of the EIF on the catalogue of 
public services. Catalogue of Public services is one of the interoperability enablers for integrated 
public services according to the conceptual model defined by the revised EIF. 
To that end, the action is defining a technical specification (data model) and implementing a set of 
tools to facilitate the creation of catalogue of public services. 
The Action supports the implementation of the EIF and the EIS by promoting semantic 
interoperability, through the definition and use of common specifications, thus contributing 
primarily to Recommendation 16 of the revised EIF. In addition, the action covers the following 
underlying principles of the EIF: Reusability, Multilingualism, Openness, Semantic 
interoperability, Technical interoperability and Standardisation. 
The action meets the recommendations and principles of the new EIF, in particular regarding 
multilingualism, accessibility, administrative simplification, transparency, and reusability of the 
solutions. The creation of a Public Multilingual Knowledge Infrastructure will support EU public 
administrations in creating services that can be accessible and shareable independently from the 
language actually used, as well as allowing SMEs to sell goods and service cross-border in a 
digital single market. 
This action will facilitate the sharing of open data between public administrations through the 
support to the execution of analytics projects on Big Data; increase the transparency of decision-
making in public administrations by supporting knowledge sharing on evidence-based policy-
making practices; support the re-use of open-source data analytics tools developed by Member 
States of EU Institutions; and provide public administrations with the opportunity to test (open 
source) technologies in this domain before making a decision on the technical way forward. 

162 

 
Action Name 

Contribution to the EIF 

Package 

3. Access to 
data/data 
sharing/open data 

Action 
number 
2016.06 

Sharing Statistical 
Production and 
Dissemination Services and 
Solutions in the European 
Statistical System 

3. Access to 
data/data 
sharing/open data 

2016.18 

4. Geospatial 
solutions 

2016.10 

Development of an Open 
Data Service, Support and 
Training Package in the Area 
of Linked Open Data, Data 
Visualisation and Persistent 
Identification 
European Location 
Interoperability Solutions for 
e-Government (ELISE) 

2016.23 

Legal interoperability 
(former ICT Implications of 
EU Legislation) 

The project contributes to the new EIF and the Interoperability Action Plan by 
1) developing, maintaining and promoting interoperable solutions for the production and 
dissemination of statistics by EU public administrations (including the EC) EIF (Focus Area 4); 2) 
developing, maintaining and promoting a) a specification of the EIRA to support better 
interoperability and cooperation for the production and dissemination of Official Statistics in the 
European Statistical System; b) a common infrastructure for the exposure and consumption of 
shared statistical services. (Focus Area 5) 
In addition, the proposal contributes significantly to the realisation of the ESS Vision 2020 
objectives in the domain of sharing tools and improving statistical dissemination. 
The action contributes to the new EIF, namely the interoperability principles: openness, 
transparency, reusability, user–centricity and multilingualism. The action contributes to priorities: 
organisational interoperability (Focus Area 2); sharing of good practices (Focus Area 3); 
governance structure (Focus Area 1) and key enablers: EU open data initiative (Focus Area 4). 

ELISE will deepen the understanding of location interoperability enablers and barriers related to 
the transition towards digital government. As such, it is intended to provide technical assessments 
and recommendations for the new EIF and contribute to the implementation of the Interoperability 
Action Plan, particularly when it comes to Actions 4, 6, 17 & 19. 
The action contributes to all EU policies, as it is about ensuring that EU legislation, no matter 
what the policy area, takes into account interoperability, ICT aspects and related impacts. 
The action implements Recommendation 27 on legal interoperability of the new EIF and the 
interoperability Action Plan action 3 of Focus Area 1 and actions 19 and 20 of Focus Area 5. 

2017.04 

Inter-Institutional Register of 
Delegated Acts (RegDel) 

This action contributes primarily to the transparency principle of the EIF and to Recommendation 
5 of the new EIF by providing a transparent overview of delegated acts. 

163 

6. Decision 
making and 
legislation - 
Supporting 
instruments 
6. Decision 
making and 
legislation - 
Supporting 
instruments 

 
 
Package 

8. Supporting 
instruments for 
public 
administrations 
8. Supporting 
instruments for 
public 
administrations 

8. Supporting 
instruments for 
public 
administrations 
8. Supporting 
instruments for 
public 
administrations 

Action 
number 
2016.20 

Action Name 

Contribution to the EIF 

Joinup – European 
Collaborative Platform and 
Catalogue 

The action facilitates the sharing and reuse of solutions for public administrations and provides the 
stakeholders with the means to collaborate via a collaborative platform, thus contributing to the 
reusability principle of the EIF. 

2016.21 

National Interoperability 
Framework Observatory 

2016.35 

EUSurvey 

This action will fulfil Objectives 4 and 5 of the Action Plan for interoperability. The observatory 
will monitor the implementation of the EIF Recommendations by Member States and the 
achievement of the roadmap of actions proposed in the EIF-IS and AP. It will also provide ad hoc 
and support training to Member States' public administrations to ensure EIF implementation 
across all levels of their national administrations. 
EUSurvey contributes primarily to the multilingualism principle of the EIF. 

2017.01 

Standard-Based Archival 
Data Management, Exchange 
and Publication 

9. Accompanying 
measures 

2016.30 

Raising Interoperability 
Awareness – Communication 
Activities 

Source: CEPS final study 

The action follows the line of the second pillar of EIF (Core interoperability principles: Openness, 
Transparency, Reusability, Technological neutrality and data portability) and of the fourth pillar of 
EIF (Foundation principles for cooperation amongst public administrations: Preservation of 
information, Effectiveness and Efficiency) by proposing the creation of a standard-based approach 
for facilitating the preservation and the exchange of archival information of the public 
administrations in an open, transparent and reusable way. It also aims at creating a set of 
recommendations for the publication of archival information as open data mainly for the usage of 
citizens, focusing thus on the principles of the third pillar of EIF (Principles related to generic user 
needs and expectation: User-centricity, Inclusion and accessibility, Multilingualism). 
This action contributes to the principles of inclusion, accessibility and transparency by 
disseminating information about interoperability and the work of ISA2. 

164