Source: EURLEX
Language: en
Format: md

![european flag](./../../../images/eclogo.jpg)EUROPEAN COMMISSION

Brussels, 23.9.2019

SWD(2019) 1615 final

COMMISSION STAFF WORKING DOCUMENT

Interim evaluation of the ISA² programme

Accompanying the document

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Results of the interim evaluation of the ISA² programme

{COM(2019) 615 final}

Table of contents

1.Introduction

1.1.
   Purpose and scope
   

2.Background to the intervention

2.1.
   Description of the intervention and its objectives
   

2.2.
   Baseline and points of comparison
   

3.Implementation / State of Play

3.1.
   Description of the current situation
   

4.Method

4.1.
   Short description of methodology
   

4.2.
   Limitations and robustness of findings
   

5.Analysis and answers to the evaluation questions

5.1.
   Relevance
   

5.2.
   Effectiveness
   

5.3.
   Efficiency
   

5.4.
   Coherence
   

5.5.
   EU added value
   

5.6.
   Utility
   

5.7.
   Sustainability

6.Conclusions

6.1.
   Context
   

6.2.
   Method
   

6.3.
   Findings
   

6.4.
   Next steps
   

Annex 1: Procedural information

Annex 2: Synopsis report of the consultation activities

Annex 3: Intervention Logic

Annex 4: Evaluation Framework

Annex 5: Sample of actions

Annex 6: Supporting evidence from desk research

Glossary

|  |  |
| --- | --- |
| Term or acronym | Meaning or definition |
| ADMS | Asset Description Metadata Schema |
| CEA | cost-effectiveness analysis |
| CEF | Connecting Europe Facility |
| CEN | European Committee for Standardisation |
| CENELEC | European Committee for Electrotechnical Standardisation |
| CEPS | Centre for European Policy Studies |
| CPSV-AP | Core Public Service Vocabulary Application Profile (CPSV-AP) |
| DCAT-AP | Data Catalogue Vocabulary Application Profile for Data Portals in Europe |
| DG | Directorate-General |
| DG DIGIT | Directorate-General for Informatics |
| DIGIT.D2 | Directorate-General for Informatics, Directorate D Digital Services, Unit D2 Interoperability |
| DSM | Digital Single Market |
| EEA | European Economic Area |
| EFTA | European Free Trade Association |
| EIF | European Interoperability Framework |
| EIRA | European Interoperability Reference Architecture |
| ELISE | European Location Interoperability Solutions for e-Government |
| EQ | evaluation question |
| ERDF | European Regional Development Fund |
| ESF | European Social Fund |
| ESIF | European Structural and Investment Funds |
| ESPD | European Single Procurement Document |
| EU | European Union |
| EVM | earned value management |
| Horizon 2020 | EU framework programme for research and innovation (2014-2020) |
| IAP | Interoperability Action Plan (Annex I to the Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions: European Interoperability Framework — Implementation Strategy. Interoperability Action Plan, Brussels, 23.3.2017, COM(2017) 134 final) |
| ICT | information and communication technology |
| IDA | Programme on Interchange of Data between Administrations |
| IDABC | Programme on Interoperable Delivery of Pan-European eGovernment Services to public administrations, Business and Citizens |
| IMAPS | Interoperability Maturity Assessment of a Public Service |
| Interoperability | As explained in Article 2(1) of the  [ISA2 Decision](http://data.europa.eu/eli/dec/2015/2240/oj) , ‘interoperability’ means the ability of diverse organisations to interact towards mutually beneficial and agreed common goals. It involves the sharing of information and knowledge between the organisations, through their business processes and by means of the exchange of data between their respective ICT systems. |
| ISA | Programme on interoperability solutions for European public administrations |
| ISA2 | Programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens |
| ISA2 actions webpage | <https://ec.europa.eu/isa2/actions_en> |
| ISA2 dashboard | <https://ec.europa.eu/isa2/dashboard/> |
| ISA2 Decision | L 318/1 Decision (EU) 2015/2240 of the European Parliament and of the Council of 25 November 2015 establishing a programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens (ISA2 programme) as a means for modernising the public sector, Brussels 4.12.2015. |
| ISA2 solutions webpage | <https://ec.europa.eu/isa2/solutions_en> |
| ISA2 website | <https://ec.europa.eu/isa2/isa2_en> |
| ISSG | Inter-Service Steering Group |
| IT | information technology |
| JC | judgment criterion |
| Joinup | collaborative platform facilitating the sharing and reuse of IT solutions developed for public administrations |
| NIFO | National Interoperability Framework Observatory |
| OECD | Organisation for Economic Cooperation and Development |
| PMKI | Public Multilingual Knowledge Management Infrastructure for the Digital Single Market |
| REFIT | the European Commission's regulatory fitness and performance programme |
| RegDel | Inter-institutional Register of Delegated Acts |
| SCM | standard cost model |
| SEMIC | The ISA2 programme action that promotes semantic interoperability among EU Member States |
| SPI | schedule performance index |
| SRSP | Structural Reform Support Programme |
| TOOP | the ‘once-only’ principle project, http://www.toop.eu/ |
| UNPAN | United Nations Public Administrations Network |
| VIES | Value Added Tax Information Exchange System |
| W3C | World Wide Web Consortium |

1.Introduction

Purpose and scope

[ISA2](https://ec.europa.eu/isa2/home_en)
 (interoperability solutions for European public administrations, businesses and citizens) is an EU programme that supports the development of digital solutions and common frameworks that enable public administrations, businesses and citizens in Europe to benefit from interoperable cross-border and cross-sector public services.

The programme’s primary beneficiaries are European public administrations at all levels - EU, national, regional and local levels - by reusing the solutions offered by ISA2 they can provide better – more interoperable, user-centric and digital – public services. However, the programme also benefits a wider group of stakeholders, including EU businesses and citizens.

ISA2 is open to EU Member States, other countries of the European Economic Area and candidate countries. In addition to the 28 EU Member States, three other countries take part in the programme: Iceland, Norway (since 2016) and Montenegro (since 2018). The programme encourages cooperation with other non-EU countries and with international organisations and bodies.

The ISA2 programme — established by the 
[ISA2 Decision](https://eur-lex.europa.eu/eli/dec/2015/2240/oj)

[1](#footnote2)
 — has a budget of €130.9 million and runs for five years from 1 January 2016 until 31 December 2020. In line with Article 13(3) of the ISA2 decision, the Commission must carry out an interim evaluation of the ISA2 programme by 30 September 2019. Besides fulfilling this legal obligation, the evaluation aims to improve the implementation of ISA2 and any of its successor programmes.

The evaluation covers ISA2 activities from the programme’s start until the third quarter of 2018 in all participating countries. The evaluation framework builds on the five evaluation criteria stemming from the 
[Commission’s better regulation](https://ec.europa.eu/info/law/law-making-process/planning-and-proposing-law/better-regulation-why-and-how/better-regulation-guidelines-and-toolbox_en)
 requirements: relevance, effectiveness, efficiency, EU added value and coherence. As shown in 
[Box 1](#_Ref9333370)
, it also addresses utility and sustainability in line with the ISA2 Decision.

  

Box 1: Evaluation framework

|  |
| --- |
| The evaluation was based on an evaluation framework composed of seven evaluation criteria and 10 evaluation questions (EQs). The evaluation criteria and the EQs are summarised below:  Relevance refers to the alignment between the programme's objectives and the evolving needs and problems experienced by stakeholders.  EQ1: To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels?  Effectiveness focuses on the extent to which the ISA² programme has achieved the objectives it intended to achieve, and generated the results it intended to produce.  EQ2: How far are the ISA² programme’s results in the process of achieving the programme’s objectives?  EQ3: Are there aspects that are more or less effective than others are, and if so, what lessons can be drawn from this?  Efficiency concerns the minimisation of costs borne by various stakeholders in achieving the objectives/results identified under the ‘effectiveness’ criterion.  EQ.4: To what extent has the programme been cost-effective?  EQ.5: Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised?  EQ5.1: How is the programme performing relative to the planned work and budget?  Coherence is a measure of the degree to which the actions supported by the ISA² programme are consistent with each other (internal coherence) and with the EU policy framework at large and relevant global initiatives (external coherence).  EQ.6: To what extent do the ISA² actions form part of a ‘holistic’ approach within the framework of the programme? (Internal coherence)  EQ.7: To what extent is the ISA² programme coherent with other EU interventions, which have similar objectives and with global initiatives in the same field? (External coherence)  EU added value captures the programme's impacts additional to those that would be achieved if the issues addressed by ISA2 were left solely in the hands of national and sub-national authorities.  EQ.8: What is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels?  Utility refers to (i) the extent to which the results of ISA2 meet stakeholders’ needs and (ii) the level of stakeholder satisfaction with the solutions offered by ISA2.  EQ.9: How do the ISA² programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address?  Sustainability measures the likelihood of the ISA² programme's results lasting beyond its completion.  EQ.10: To what extent is the financial, technical and operational sustainability of the developed solutions — maintained and operated through the ISA² programme — ensured?  For further details, please see  [Annex 4: Evaluation Framework](#_Ref9330490) . |

2.Background to the intervention

Description of the intervention and its objectives

The ISA2 programme’s ultimate objective is to promote the ICT-based modernisation of the public sector in Europe and to facilitate addressing the needs of businesses and citizens, via improved interoperability of European public administrations.

More specifically the programme aims to do the following:

Facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens.

Contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local administration levels.

Promote a holistic approach to interoperability in the EU, by identifying, creating and operating interoperability solutions and facilitating their reuse by European public administrations. This will support the implementation of various EU policies and activities.

By working towards achieving the above objectives, the programme intends to address the problem of existing or emerging ‘electronic barriers that impede the proper functioning of the internal market’
[2](#footnote3)
.

It is important to note that ISA2 is part of a wider policy framework related to the digitalisation of public administrations in the EU. In cooperation with the Member States and the Commission, the ISA2 programme promotes and maintains the 
[European Interoperability Framework (EIF)](http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1524640075337&uri=CELEX:52017DC0134)

[3](#footnote4)
, which has been in place since 2010 and was revised in 2017. The revision of the EIF was called for in the Communication on a 
[Digital Single Market Strategy for Europe](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52015DC0192)

[4](#footnote5)
, which recognised the role of interoperability in cross-border and cross-sector connections between communities, public services and public administrations. Other initiatives related to the digitalisation of public administrations to which ISA2 contributes include:

The 
[eGovernment Action Plans](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52016DC0179)
 (the current version of which runs from 2016 to 2020), which aim to make ‘public administrations and institutions in the EU open, efficient and inclusive, providing borderless, personalised, user-friendly, end-to-end digital public services to all citizens and businesses in the EU’
[5](#footnote6)
.

The 2017 
[Tallinn Declaration on eGovernment](http://ec.europa.eu/newsroom/document.cfm?doc_id=47559)

[6](#footnote7)
, a declaration made by ministers in charge of eGovernment policy across the EU to spell out their commitment to a number of principles, including ‘interoperability by default’, which aims to ensure that public services are designed to work seamlessly within the single market and across organisational silos, relying on the free movement of data and digital services in the EU.

The 
[Connecting Europe Facility (CEF)](http://data.europa.eu/eli/reg/2013/1316/oj)

[7](#footnote8)
 established in 2013 as a funding programme providing support to cross-border interaction between the digital services infrastructures of Member States. Based on interoperability agreements between Member States, CEF Digital provides building blocks that ensure that citizens, businesses and administrations can benefit from seamless digital public services across the EU.

The 
[Single Digital Gateway](http://data.europa.eu/eli/reg/2018/1724/oj)

[8](#footnote9)
, to provide citizens and businesses with online access to information and procedures and to assistance and problem-solving services.

The 
[Structural Reform Support Programme (SRSP)](http://data.europa.eu/eli/reg/2017/825/oj)

[9](#footnote10)
, which provides support to Member States in five main reform areas, including IT capacity building in public administrations.

·Several other EU funding programmes, like Horizon 2020
[10](#footnote11)
 (dedicated to research and innovation), the European Social Fund (ESF)
[11](#footnote12)
 and the European Regional Development Fund (ERDF)
[12](#footnote13)
.

Chapter 5 provides further details of the links between ISA2 and the above-mentioned programmes and initiatives, under the coherence evaluation criterion (section 5.4).

The intervention logic aims to clarify the reasoning followed by EU decision makers when establishing the ISA2 programme. It includes a detailed description of:

·the needs, problems and drivers that the programme intended to address;

·the objectives set out for the programme (following a three-level hierarchy: global objective, specific objectives and operational objectives);

·the inputs or activities of the programme;

·the programme's expected results; and

·the logical links between these components.

The intervention logic elements and links provide a benchmark against which the interim evaluation can assess the programme's achievements in the first half of its period of operation. 
[Figure 1](#_Ref1398290)
 provides an overview of the elements of the intervention logic and the links between them, and further details are provided in 
[Annex 3: Intervention Logic](#_Ref9330511)
.

Figure 1: Intervention logic of the ISA2 programme

![](./../../../resource.html?uri=comnat:SWD_2019_1615_FIN.ENG.xhtml.SWD_2019_1615_FIN_ENG_24002.jpg)

Baseline and points of comparison

ISA2 is the fifth in a series of EU programmes
[13](#footnote14)
 providing and promoting interoperability solutions for public administrations in the EU. ISA2 succeeded the ISA programme, which implemented 51 actions during its 6-year period of operation (from 2010 to 2015) with a total budget of around €164 million
[14](#footnote15)
. The results of ISA represent the main baseline used for the purpose of this interim evaluation.

The end of the ISA programme was marked by a final evaluation
[15](#footnote16)
, which took stock of the programme’s achievements and put forward three recommendations for the future edition of the programme, focusing on the below areas:

communication with public administrations and raising their awareness of interoperability;

a holistic approach to and cross-cutting view of interoperability; and

·cooperation with other EU policies and initiatives.

Although the present interim evaluation focuses on results achieved under ISA2, it also addresses the above-listed recommendations. It is also worth noting that ISA2 aims to achieve its objectives through a number of actions that continue from the previous edition of the programme, as well as through new actions. Where relevant, the impact of ISA on ISA2 was assessed by comparing the performance of actions continued from ISA with that of actions started during ISA2.

  

3.Implementation / State of Play

Description of the current situation

ISA² has been operational from 1 January 2016 and will continue to run until 31 December 2020. It has a total budget of around €131 million. ISA2 consists of nine groups of actions, called packages, which are defined on a yearly basis through an annual revision of the rolling work programme (rolling work programme):

key and generic interoperability enablers;

semantic interoperability
[16](#footnote17)
;

access to data / data sharing / open data;

geospatial solutions;

eProcurement / eInvoicing;

decision-making and legislation;

EU policies — supporting instruments;

supporting instruments for public administrations; and

·accompanying measures.

The 2018 rolling work programme consists of 53 actions, categorised into the nine packages mentioned above. ISA2 managed 39 actions in its first year of operation, 43 actions in 2017, 53 actions in 2018, and is currently managing 54 actions under the 2019 rolling work programme. 
[Table 1](#_Ref9338333)
 provides a breakdown of actions and allocated budget per package and per year. As ISA2 is scheduled to end on 31 December 2020, one additional rolling work programme remains to be adopted. The present evaluation is based on the rolling work programme implemented in 2016, 2017 and 2018, as the 2019 rolling work programme had not yet been adopted when the data collection took place.

Table 1: Overview of allocated budget and number of actions per package from 2016 to 2018

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Package | Allocated budget 2016 (thousand €) | Actions 2016 | Allocated budget 2017 (thousand €) | Actions 2017 | Allocated budget 2018 (thousand €) | Actions 2018 |
| Key and generic interoperability enablers | 4 900 | 6 | 5 407 | 6 | 4 250 | 7 |
| Semantic interoperability | 2 008 | 3 | 1 831 | 3 | 1 503 | 4 |
| Access to data / data sharing / open data package | 2 800 | 5 | 3 548 | 5 | 3 925 | 7 |
| Geospatial solutions | 983 | 1 | 2 240 | 1 | 1 900 | 1 |
| eProcurement / eInvoicing | 2 400 | 1 | 1 445 | 1 | 1 063 | 1 |
| Decision-making and legislation | 2 260 | 6 | 2 608 | 9 | 2 235 | 10 |
| EU policies — supporting instruments | 2 935 | 3 | 3 580 | 3 | 4 820 | 5 |
| Supporting instruments for public administrations | 4 425 | 12 | 3 533 | 13 | 5 315 | 16 |
| Accompanying measures | 1 280 | 2 | 730 | 2 | 1 290 | 2 |
| Total | 23 991 | 39 | 25 545 | 43 | 26 301 | 53 |

Source: ISA2 rolling work programmes (2016, 2017, 2018), indicative planning and financial overview.

The programme in its entirety is overseen by the Interoperability Unit of DG DIGIT (DIGIT.D2), while individual actions are implemented by different Commission services, depending on the thematic scope. The Member States are also involved in programme governance through two channels: the ISA² Committee, the programme's high-level governing body, and the ISA² Coordination Group, a technical body mandated to ensure coherence between the programme’s actions
[17](#footnote18)
. All 28 EU Member States participate in the programme. Beyond the EU, ISA2 has three additional countries — Iceland, Montenegro and Norway — participate in ISA2. In addition, an administrative agreement between DG DIGIT and the Agency for the Development of the Government of Electronic Management and the Information and Knowledge Society of Uruguay has been in place since March 2018
[18](#footnote19)
.

ISA2 actions are selected and implemented through a process consisting of four steps taken each year:

Submission: Commission services, Member States, and other countries participating in ISA2 are invited to submit proposals for actions to be included in the rolling work programme via a call for proposals.

Evaluation and consultation of the ISA2 Committee: The submitted proposals are analysed by DIGIT.D2, which compiles a list of proposals that qualify to be included in the rolling work programme, based on the selection and prioritisation criteria as defined in the ISA2 Decision. The ISA2 Committee gives an opinion on the proposed action list.

Adoption: The annual rolling work programme is adopted by the Commission and the budget is released.

·Implementation: The actions in the adopted annual rolling work programme are implemented by the Commission services in charge.

The programme's overall performance and the progress of each action are frequently recorded through the quarterly and annual monitoring and evaluation reports, which feed into the ISA2 dashboard
[19](#footnote20)
. The dashboard is an online interactive tool that facilitates the dissemination of information about action activities and achievements, and their efficiency, effectiveness and coherence. The efficiency of actions is measured using the earned value management (EVM) analysis (see section 5.3 
[Efficiency](#_Ref9590260)
), effectiveness is presented in terms of performance indicators (see section 5.2 
[Effectiveness](#_Ref9590329)
), and the coherence of actions is mapped using network analysis (see section 5.4 
[Coherence](#_Ref9590394)
). 
[Figure 2](#_Ref9340333)
 illustrates the performance of the ISA2 programme until Q1 2019, as captured by the ISA2 dashboard based on data from the EVM mechanism.

Figure 2: Earned value analysis at programme level

![](./../../../resource.html?uri=comnat:SWD_2019_1615_FIN.ENG.xhtml.SWD_2019_1615_FIN_ENG_24003.jpg)

Note: The cumulated planned value is the sum of the planned values of the programme's different actions for which the EVM is used. The cumulated earned value is the sum of the earned values of the programme's different actions for which the EVM is used. The budget at completion (BAC) is the sum of all undergoing and completed specific contracts financed by the ISA2 budget.

Source: Monitoring team of the ISA2 programme (see also the 
[Efficiency view of the ISA2 Programme](https://ec.europa.eu/isa2/dashboard/programme/efficiency)
).

  

4.Method

Short description of methodology

As a first step, the Commission outlined and agreed on the draft evaluation design (including the intervention logic and the evaluation questions). It then reached out to external experts and asked them to support the evaluation process (see 
[Annex 1](#_Ref9346654)
). The consultant refined the evaluation design and — with the help and under the close monitoring of the Commission — moved to action. First, it collected data, then validated and analysed them, as explained in the following sections. At the end, the consultant summarised its findings in an independent evaluation study (referred to as the CEPS final study in this report)
[20](#footnote21)
.

Data collection

The data collection phase involved a mix of quantitative and qualitative data collection methods and aimed to gather the evidence base to respond to the evaluation questions (EQs) specified in the evaluation framework (
[Annex 4](#_Ref9341586)
). More specifically, data were drawn from two main sources: consultation activities (primary data) and desk research (secondary data).

Primary data

Primary data were collected between 29 November 2018 and 1 March 2019 via the following consultation activities
[21](#footnote22)
:

short questionnaire distributed during the ISA2 Mid-Term Conference and the kick-off workshop;

public consultation;

targeted online surveys;

targeted in-depth interviews; and

·kick-off workshop.

These activities resulted in a total of 129 responses from stakeholders, out of which 14 replies arrived during the public consultation. The low consultation response rate is mainly explained by two reasons:

ISA2 is a technical programme producing solutions addressed mainly to European public administrations. Understanding how the programme works requires specific knowledge in the field. Therefore, direct contact between the average citizen/business and the programme tends to be very limited, and this most probably led to the reduced response rate. In order to increase the participation of citizens/businesses, a short questionnaire with the same questions as those of the public consultation was distributed during the ISA2 Mid-Term Conference, which resulted in 15 additional responses. It is worth noting, however, that businesses and citizens benefit from ISA2 solutions and activities indirectly, through their interactions with public administrations. Thus, a low response rate among these groups of stakeholders does not reflect negatively on the programme.

·Stakeholder groups that are in direct contact with the programme (e.g. solution users, action owners, etc.) were consulted via targeted online surveys and interviews, which covered inter alia the same topics addressed by the public consultation. Therefore, stakeholders belonging to these groups were asked not to respond to the public consultation in order to avoid duplicate answers.

Despite the relatively low response rate, the consultation activities allowed for a wide coverage of the different stakeholders of the ISA2 programme, ranging from actors involved in programme governance to citizens, i.e. the indirect beneficiaries of ISA2. Moreover, on average, the respondents were characterised by a high level of expertise, which contributed to the quality and reliability of primary data collected. The in-depth interviews and targeted surveys provided particularly useful input to the interim evaluation. They were tailored to stakeholders closely related to the programme and allowed them to provide extensive details, thus making it possible for the evaluation to get deep insights into the subject areas covered. In general, primary data contributed to the assessment of all evaluation criteria. Overall, the answers collected are consistent across stakeholder groups.

For more details, please consult 
[Annex 2: Synopsis report of the consultation activities](#_Ref9351660)
, which presents a breakdown of responses by consultation activity and stakeholder category along with respondent characteristics.

Chapter 5 of this report presents the results of the consultation activities using mainly bar charts with averages. The results of the public and targeted consultation activities are combined and presented in aggregated form throughout the evaluation. For each question, average scores were computed for each stakeholder group that provided feedback on the specific question. Averages do not take into account those respondents who answered ‘don’t know/no opinion’ to the question under analysis. The data labels of the bar charts show the average score first, then the number of respondents in brackets.

Secondary data

The desk research reviewed the following resources:

publicly available documents and data sources, including the ISA2 annual rolling work programmes, the ISA2 dashboard, and the webpages dedicated to actions and solutions; and

·additional operational documents including monitoring and evaluation reports, overviews of communication activities, and lists of participants in the ISA2 Committee and Coordination Group.

[Annex 6](#_Ref9581764)
 provides an overview of the supporting evidence collected from desk research.

Note that, in order to better guide the data collection activities, a sample of 20 actions was selected from the total of 53 actions funded by the ISA2 programme up to 2018. The criteria used for sampling and the sampled actions are presented in 
[Annex 5:](#_Ref9348102)
[Sample of actions](#_Ref9348102)
.

The desk research, review of the programme's operational documents and targeted in-depth interviews focused on the 20 sampled actions. However, stakeholders involved in all sampled actions had the opportunity to respond to either the targeted online surveys or the public consultation, to participate in the kick-off workshop and to complete the short questionnaire distributed during the ISA2 Mid-Term Conference.

The data collection was complemented by an expert assessment of the programme, performed by four technical interoperability experts who were part of the consultant’s evaluation team.

Data validation

The collected data were validated via triangulation in order to ensure the robustness of evidence. The Commission’s Better Regulation Toolbox defines triangulation as ‘the application and combination of several research methodologies in the study of the same phenomenon’
[22](#footnote23)
. In fact, for all evaluation criteria and questions, data were collected from multiple sources and via at least two different data collection methods (e.g. interviews, targeted questionnaire, public consultation, desk research).

Quantitative methods of data analysis

Besides the statistical analysis of data collected via consultation activities, desk research and the qualitative analysis of open responses provided by consulted stakeholders and information contained in documentary evidence, three specific quantitative methods were used in order to evaluate the programme's efficiency:

The standard cost model (SCM)
[23](#footnote24)
 is a method of assessing administrative costs imposed by rules or policies inter alia on businesses and public administrations. It is based on the identification of the basic components of a rule, the information obligations, whose costs for the addressees can be measured and quantified. An information obligation is a specific duty to gather, process or submit information to the public authority or a third party. The SCM was used to measure the costs borne by action owners in preparing and submitting proposals for ISA2 actions.

Cost-effectiveness analysis (CEA)
[24](#footnote25)
 is a method of assessing the merits of a policy in an interim and ex post evaluation setting. In a nutshell, CEA measures the value-for-money of past policies, i.e. the amount of benefit generated by unitary costs. Costs are measured in monetary terms, whereas effectiveness is measured in ‘natural units’, and the unit of account varies depending on the nature of the problem addressed (e.g. the number of users of key and generic interoperability enablers). The heterogeneity of performance indicators available for ISA2 actions makes it difficult to draw conclusions about the programme's overall cost-effectiveness and thus using CEA was only possible for certain action packages and indicators, as described in Chapter 5.

·In line with the PM2 methodology developed by the Commission
[25](#footnote26)
, the earned value management (EVM) and earned schedule (ES) methods are currently used to monitor and assess the efficiency of the ISA2 programme. EVM is a project management technique that helps determine work progress against a given baseline, so that costs, time, and scope of a certain activity are constantly tracked. In the context of the ISA2 programme, efficiency is assessed at action and programme levels. The implementation of EVM requires managers to calculate the earned value, i.e. a quantification of the ‘worth’ of the work done to date, and the actual costs, i.e. the executed budget for achieving the work, and to compare them with the planned value of such activity. This allows for a better understanding of the programme's performance. The ES is an extension of the EVM method that deepens the level of analysis to a ‘units of time’ layer. In the framework of the ISA2 programme, tailored versions of the EVM and ES approaches are adopted
[26](#footnote27)
.

Limitations and robustness of findings

The 
[consultation strategy](https://ec.europa.eu/isa2/sites/isa/files/consultation_strategy_isa2_interim_evaluation.pdf)

[27](#footnote28)
 and the data collection plan put forward in the Evaluation Framework (
[Annex 4](#_Ref9341586)
) were fully implemented. Therefore, the evaluation was able to draw robust conclusions for all EQs based on both primary and secondary data. Nevertheless, some emphasis needs to be placed on caveats that may have an impact on the main findings of the interim evaluation as well as on future evaluation exercises:

The timing of this evaluation is compliant with Article 13 of the ISA2 Decision, which requires the Commission to present an interim evaluation to the European Parliament and the Council by 30 September 2019. Given this constraint, the evaluation confined the analysis to actions funded during the programme's first three years (2016, 2017 and 2018). In particular, some actions under evaluation are still ongoing; hence, their outputs and outcomes can only be recorded to some extent. On a more general note, as discussed in the intervention logic (
[Annex 3](#_Ref9341762)
), impacts occur over the long term and it is difficult to capture them in an interim evaluation. In principle, in the coming years ISA2 actions may see improvements
[28](#footnote29)
 in the way they achieve the programme’s objectives and deliver the expected results.

Only action owners had access to the contact details of their solution users due to confidentiality and data protection reasons. Therefore, action owners were requested to: i) invite users to complete the relevant targeted online survey; and ii) share with DIGIT.D2 the details of those users who gave their consent to participate in interviews. This two-step approach may have limited the number of answers received from solution users. In addition, some action owners explained that they were unable to contact their users for the purpose of evaluation, as the users’ consent to be involved in consultation activities was never collected. Against this background, ISA2 may consider requesting its users to provide their consent to be contacted by Commission staff and contractors for evaluation purposes in the future.

5.Analysis and answers to the evaluation questions

In line with the Evaluation Framework (
[Annex 4](#_Ref9341586)
), the interim evaluation focused on seven evaluation criteria: relevance, effectiveness, efficiency, coherence, EU added value, utility and sustainability. The below chapters present the following for each of them: a) the pre-defined evaluation questions (EQ) and related judgment criteria; b) the results of the analysis fed by the collected data; and finally c) the answer to the EQ.

Relevance

Evaluation framework for 'relevance'

Relevance refers to the alignment between the programme's objectives and the evolving needs and problems
[29](#footnote30)
 experienced by stakeholders. As shown in 
[Box 1](#_Ref9333370)
, the interim evaluation translated this criterion into the following EQ:

EQ1:To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels?

To answer this question, the evaluation assessed two main judgment criteria (JCs), by relying on data collected via consultation activities as well as on expert assessment:

JC1:Alignment between needs and problems addressed by the programme and current needs and problems experienced at the EU, national and subnational level; and

JC2:Alignment between the objectives of the programme and current needs and problems experienced at the EU, national and subnational level.

Analysis

JC1: Alignment between original and current needs and problems

The rapidly evolving nature of ICT and interoperability begs the question as to whether the needs and problems originally targeted by the programme are the same as the current needs and problems experienced by stakeholders.

[Figure 3](#_Ref9504117)
 shows the average scores derived from the answers received on this question, per stakeholder group. Overall, more than 70 % of respondents consider that the three original needs, as presented in 
[Figure 3](#_Ref9504117)
, are still relevant to a high extent or to the fullest extent (93 out of 128 respondents for the first need, 99 out of 127 respondents for the second need, and 94 out of 125 respondents for the third need). The persistence of the problem of administrative e-barriers originally identified by the programme is confirmed as well. Two-thirds of the respondents (79 out of 118) consider this problem still relevant to a high extent or the fullest extent, in comparison to almost 13 % (15 out of 118) who believe the problem is relevant only to a limited extent or not at all.

Figure 3: Extent to which needs and problems originally addressed by ISA2 are currently experienced by European public administrations, businesses and/or citizens (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

There are some additional needs and problems that the programme could aim to address, as pointed out by 39 % of the consulted stakeholders (46 out of 129). More specifically, several respondents converged on the following needs and problems related to interoperability of public services that European public administrations, businesses and/or citizens currently experience:

EU level:

The need for a more binding legal framework for interoperability
[30](#footnote31)
, including the need for legally binding standards or the general need for a more binding approach to interoperability in the EU.

The need for a more prescriptive approach to designing interoperable public services (‘interoperability by design’). It is worth noting, however, that the revised EIF adopted in 2017 explicitly mentions ‘interoperability by design’ as the conceptual model for interoperable public services
[31](#footnote32)
. Therefore, this need is taken into account through the wider framework of the EIF.

The need to improve the way administrations communicate with one another, given the different forms of organisation and functioning of public administrations in the Member States.

The need to share best practices.

·The need to account for new developments such as blockchain, privacy-by-design and self-sovereign identities
[32](#footnote33)
, which are changing the interoperability landscape.

National/subnational level:

Resource constraints, such as shortage of qualified IT staff, experienced by national and local public administrations.

The different political priorities among Member States hindering a consistent approach to interoperability in the EU.

·The limited awareness of ISA2 and other initiatives related to interoperability, especially at the regional and local levels
[33](#footnote34)
.

JC2: Alignment between needs and problems and the programme's objectives

After ensuring that all the original needs and problems are still relevant and identifying additional needs and problems experienced by stakeholders, the evaluation checked the degree to which the programme's objectives address these needs and problems.

The majority of consulted stakeholders agree that, through its general objective
[34](#footnote35)
, ISA2 addresses the main needs and problems identified in the field of interoperability of digital public services at the time the programme was set up, either to a high extent or to the fullest extent (
[Figure 4](#_Ref9507001)
). This result holds true across all stakeholder groups. Over 80 % of respondents indicate that, by achieving its general objective, ISA2 can address the original needs to a high extent or to the fullest extent (103 out of 125 respondents for the first need, 103 out of 123 respondents for the second need, and 103 out of 124 respondents for the third need). When it comes to the problem of administrative e-barriers, approximately two-thirds of respondents (77 out of 116) agree that ISA2 can address the problem through its general objective, while only 11 % (13 out of 116) believe that the programme can only address the problem to a limited extent or not at all. These results are also confirmed by the independent assessment carried out by technical experts, who pointed out that ISA2 highlights the topic of interoperability and helps build a community of professionals in this area, thus addressing the needs and problems identified in the intervention logic.

Figure 4: Extent to which achieving ISA2 objectives contributes to addressing the needs and problems originally addressed by the programme (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

When it comes to the new needs and problems identified by consulted stakeholders, on average, respondents believe that the programme, through its general objective, could address them at least to some extent (
[Figure 5](#_Ref9513170)
). However, three out of the six respondents responsible for linked EU initiatives who indicated additional needs and problems consider that the general objective of ISA2 cannot address the additional needs and problems, or can only address them to a limited extent, as some of the issues mentioned fall outside the programme's scope (for instance, Member States' different political priorities).

Figure 5: Extent to which achieving ISA2 objectives contributes to addressing additional (current) needs and problems identified by consulted stakeholders (number of respondents by stakeholder category)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent; (DK/NO) do not know/no opinion.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Answer to the EQ on 'relevance'

EQ1: To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels?

The original needs and problems that the programme intended to address are still fully relevant and the objectives of ISA2 are pertinent to addressing them. In addition, consulted stakeholders have identified a number of new needs and problems related to the interoperability of digital public services. Some of these fall outside the programme's scope (such as resource constraints or different political priorities at national level). Others are already being addressed by existing ISA2 actions (for example, the 'Innovative Public Services' action provides support for identifying the innovation potential and framing conditions of emerging disruptive technologies). Finally, ISA² could (better) address some of these additional needs and problems by considering the below measures for future actions and future editions of the programme:

Increasing awareness of interoperability beyond national administrations by fostering more collaboration and exchanges of views with regional and local administrations. The recently launched Interoperability Academy action could help with this by developing training materials tailored to the needs of regional and local governments.

Performing studies to assess the costs and benefits of implementing ISA2 solutions. The findings of such studies could feed into promotional activities, targeting the sub-national layer of public administrations as well.

Improving the sharing of best practices among public administrations, academia and interested professionals thus raising awareness of interoperability and increasing the take-up of ISA2 solutions across Member States.

Relying more on potential influencers (like researchers, teachers or committed public servants) when communicating about interoperability. This could eventually create a community of interoperability ambassadors.

Building advisory capabilities around interoperability. These would help interested stakeholders to pick the appropriate interoperability solutions that match their needs, and would at the same time provide support services and technical assistance for effective implementation.

·Assessing the rationale and impacts of a possible binding interoperability instrument. Such assessment should rely on the findings of the final evaluation of the ISA2 programme and on the evidence that will be gathered when the implementation of the EIF is evaluated in 2021.

Effectiveness

Evaluation framework for 'effectiveness'

Effectiveness focuses on the extent to which the programme has achieved its objectives and generated the expected results. Analysing the effectiveness of ISA2 requires answers to two EQs:

EQ2:How far are the ISA² programme’s results in the process of achieving the programme’s objectives?

EQ3:Are there aspects that are more or less effective than others are, and if so, what lessons can be drawn from this?

To answer these questions, the evaluation applied the following judgment criteria (JCs):

JC1:Alignment between the programme's actual results, objectives and expected results;

JC2:Impact of external factors on the programme's performance;

JC3:Awareness of the programme;

JC4:Performance indicators; and

JC5:Alignment with principles specified in Article 4(b) of the ISA2 Decision.

The analysis relies on data collected via consultation activities, a desk review and an expert assessment.

Analysis

JC1: Alignment between the actual results, objectives and expected results of ISA2

The focus of this judgment criterion falls on the solutions developed by ISA2, as they represent the most tangible results achieved so far. 
[Figure 6](#_Ref9584015)
 shows the extent to which the programme's actual results are aligned with its objectives, according to the consulted stakeholders. 45 % of respondents (49 out of 109) agree that ISA2 solutions have contributed to achieving the programme's general objective to a high extent or to the fullest extent. In contrast, only 15 % of respondents (16 out of 109) believe that ISA2 solutions have made either no contribution or a limited contribution. The positive feedback is higher when it comes to the programme's specific objectives. For instance:

More than 56 % of respondents (59 out of 106) confirm that the solutions developed have contributed to developing, maintaining and promoting a holistic approach to interoperability in the Union (specific objective 1) to a high extent or to the fullest extent;

·More than 56 % of respondents (60 out of 108) indicate that the solutions developed have contributed to facilitating the reuse of interoperability solutions by European public administrations (specific objective 5) to a high extent or to the fullest extent.

However, respondents to the consultation activities express a lower level of agreement regarding the contribution to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels of public administration (specific objective 3). Only 34 % of respondents (36 out of 106) consider that ISA2 solutions have contributed to this objective to a high extent or to the fullest extent, while 22 % (23 out of 106) believe that there has been either no contribution or only a limited contribution. This is aligned with the assessment of the EU added value criterion (see 
[Figure 17](#_Ref9755929)
 in section 5.5), which shows that Member States’ initiatives tend to perform relatively better when it comes to achieving specific objective 3.

Figure 6: Extent to which ISA2 solutions contributed to achieving the programme’s objectives (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

The desk review of secondary data also shows good progress towards the achievement of ISA2 objectives. The 20 sampled actions have already produced 35 solutions, and a further 21 solutions are under development. Most of the operational solutions are used across the 28 Member States and across different EU institutions, thus contributing to a coherent approach to interoperability in the EU (
[Annex 6](#_Ref9581764)
: 
[Table 9](#_Ref11679417)
, 
[Table 10](#_Ref9587614)
 and 
[Table 11](#_Ref11679432)
).

The Interinstitutional Register for Delegated Acts
[35](#footnote36)
 is a good example that illustrates how the various ISA2 solutions help achieve the programme’ objectives. It was set up under the ‘RegDel’ action
[36](#footnote37)
 in order to respond to the joint commitment of the Commission, the Council and the European Parliament to facilitate access to delegated acts through a dedicated register. By doing so, the register also ‘supports the implementation of Union policies and activities’ as envisaged by specific objective 4 of the ISA2 programme. Launched in December 2017, the solution has amassed 2202 active subscriptions and over 144 000 views (up to 1 February 2019)
[37](#footnote38)
.

Another example is the European Interoperability Reference Architecture (EIRA©)
[38](#footnote39)
, which helps classify and organise interoperability building blocks used in the delivery of digital public services. EIRA facilitates the reuse of interoperability solutions by European public administrations (specific 5 of ISA2) and is appreciated by its users, as shown in 
[Box 2](#_Ref11681623)
.

Box 2: EIRA user testimonial

![](./../../../resource.html?uri=comnat:SWD_2019_1615_FIN.ENG.xhtml.SWD_2019_1615_FIN_ENG_24004.jpg)

Source: ISA2 communication team

Other examples include the ‘Joinup’ solution
[39](#footnote40)
 and events such as the ‘Sharing and Reuse Conference’
[40](#footnote41)
 that facilitate access to and the reuse of interoperability solutions, as well as the exchange of best practices.

Turning to the operational objectives (
[Figure 7](#_Ref9586777)
), the highest achievements of ISA2 solutions pertain to the following ones:

Operational objective 7 – 68 % of respondents (42 out of 62) confirmed that the existing solutions have contributed to supporting and promoting the maintenance, updating, promotion and monitoring of the implementation of the EIS, the EIF and the EIRA to a high extent or to the fullest extent, and only 1 % (1 out of 62) believe that there was a limited contribution. One good example is the European Interoperability Architecture (EIA), which has developed a conceptual reference architecture that is already deployed in eight Member States. Moreover, the European Interoperability Reference Architecture (EIRA) solution that is available on the ‘Joinup’ platform has been downloaded 1 364 times so far (
[Annex 6](#_Ref9581764)
: 
[Table 12](#_Ref9587649)
). In addition, the National Interoperability Framework Observatory contributes to this objective by assessing interoperability developments in Member States and national alignment with the EIF.

·Operational objective 9 — according to 68 % of respondents (50 out of 73), ISA2 solutions have contributed to supporting and promoting the maintenance and publication of a platform allowing access to and collaboration with regard to best practices, functioning as a means of raising awareness and disseminating available solutions to a high extent or to the fullest extent, compared to 11 % (8 out of 73) who believe that the contribution has been limited. The ‘Joinup’ action, by maintaining and further developing the collaborative platform with the same name, is directly related to this operational objective. The platform counts over 13 000 members, namely professionals working in the field of eGovernment. It allows members to work together on over 109 thematic working spaces, called ‘collections’ (
[Annex 6](#_Ref9581764)
: 
[Table 12](#_Ref9587649)
).

Based on the feedback received, there are two areas for potential improvement:

·Operational objective 4 — only 23 % of respondents (15 out of 64) believe that ISA2 solutions have contributed to supporting and promoting the identification of legislation gaps, at Union and national level, that hamper cross-border or cross-sector interoperability between European public administrations to a high extent or to the fullest extent.

·Operational objective 5 — only 25 % of respondents (17 out of 68) indicate that ISA2 solutions have contributed to supporting and promoting the development of mechanisms that measure and quantify the benefits of interoperability solutions including methodologies for assessing cost-savings to a high extent or to the fullest extent.

The fact that the ‘Legal Interoperability’ action, responsible for the advancement of these two operational objectives, has until now mainly focused on the Commission’s internal policy-making process, may explain the lower awareness levels among the programme’s stakeholders.

Figure 7: Extent to which ISA2 solutions contributed to achieving the programme’s operational objectives (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

As the programme has only just passed its midway point, it is reasonable to assume that its actual results might not fully match the expected results, since actions are ongoing and solutions will continue to be developed until the programme ends in 2020. Experts confirmed this assumption, pointing out that ISA2 solutions have contributed to, for instance, ‘a coherent interoperability landscape in the Union’, but more time is needed for the expected results to be fully realised. The feedback received as part of the consultation activities provides further evidence of this (
[Figure 8](#_Ref9589478)
).

Figure 8: Extent to which ISA2 solutions achieved the programme's expected results (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

The developed solutions are mostly aligned with the expected result of a coherent interoperability landscape in the EU based on a holistic approach to interoperability. In fact, around 50 % of respondents (37 out of 77) believe that ISA2 solutions have achieved this expected result to a high extent or to the fullest extent, in comparison to 8 % (6 out of 77) who argue that there has been limited achievement or no achievement in this area. The achievement of expected results is further assessed below, when the performance indicators are presented.

Turning to areas of potential improvement, only 30 % of respondents (23 out of 76) consider that ISA2 solutions led to improved efficiency and productivity of the European public administrations. Interestingly, during the interviews some respondents pointed out that it is difficult to reply to this specific question, as no specific studies have been carried out on the efficiency and productivity effects of ISA2 solutions, although such effects are expected. Future actions may aim to rectify this and ensure that impacts generated by ISA2 solutions for public administrations are identified and quantified.

JC2: Impact of external factors on the programme's performance

In line with the Evaluation Framework, assessing the programme's effectiveness also implies considering the positive and negative impacts of external factors on the programme's overall performance.

To this end, consulted stakeholders were asked to identify and rank the external factors that on the one hand contribute to and on the other hand jeopardise the programme's performance. There is a clear pattern in the answers of all groups consulted on this matter (programme governance stakeholders, action owners and solution users). The calls for common standards and frameworks from public administrations are considered particularly beneficial to the programme's performance (
[Figure 9](#_Ref9590820)
), as they can guide the development of new solutions to address the issues raised by public administrations, i.e. the direct beneficiaries of the solutions. Most respondents, 81 % (63 out of 75), believe that these calls contribute to the performance of ISA2 to a high extent or to the fullest extent. This also shows that interoperability is among the priorities of European public administrations.

Figure 9: Extent to which the following external factors contribute to the performance of ISA2 (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

On the other hand, institutional complexity is the external factor seen as having the greatest negative impact on the programme's performance (
[Figure 10](#_Ref9591037)
). 78 % of respondents (64 out of 82) believe that this factor jeopardises the programme's performance to a high extent or to the fullest extent. Institutional complexity comes in the form of multiple layers of governance and differences between national public administrations in various Member States as well as between local public administrations within Member States, all of which can cause coordination issues. Technical experts confirmed this result by identifying institutional and legal complexity as the factors that most jeopardise the programme’s performance.

Figure 10: Extent to which the following external factors jeopardise the performance of ISA2 (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

JC3: Awareness of the programme

When analysing the feedback collected via the consultation activities, each stakeholder group's level of awareness of interoperability issues and, more specifically, of the ISA2 programme, must be taken into consideration. As illustrated in 
[Annex 2](#_Ref9349519)
 (
[Figure 30](#_Ref9325154)
 and 
[Figure 31](#_Ref9325186)
), the consulted stakeholders were asked to assess their knowledge of digital public services and interoperability as well as their knowledge of ISA2. On average, the results show that most respondents have expert knowledge of interoperability and of ISA2. 71 % of respondents (91 out of 128) are familiar with digital public services and interoperability to a high extent or to the fullest extent, in comparison to 11 % (14 out of 128) who either are not familiar with the field or are familiar with it only to a limited extent. In addition, 63 % (81 out of 128) of respondents are familiar with the ISA2 programme to a high extent or to the fullest extent, compared to 20 % (25 out of 128) who report only a low degree of familiarity or no familiarity with the programme.

One factor that explains these positive awareness results is the ISA2 programme’s strong commitment to ‘enhance information and increase awareness with regard to the programme and its benefits’, as stated in Article 3 of the ISA2 Decision. In fact, raising public administrations’ awareness of interoperability was one of the three recommendations stemming from the final evaluation of the ISA programme
[41](#footnote42)
. The ISA2 programme addressed this by adopting its communication strategy and engagement plan in 2017
[42](#footnote43)
 and by increasing its presence at conferences and other events, as shown in 
[Box 3](#_Ref9594886)
.

The relatively low participation in the interim evaluation’s consultation activities and the need to better liaise with regional and local administrations, identified under the 
[Relevance](#_Ref9608063)
 criterion (see section 5.1), call for continued and even improved awareness raising efforts.

JC4: Performance indicators

In line with the Evaluation Framework, the desk research collected data on several performance indicators (
[Annex 6](#_Ref9581764)
: 
[Table 12](#_Ref9587649)
) in order to assess the extent to which the programme’s operational objectives had been achieved and to complement the data gathered via consultation activities.

When talking about performance indicators, it is important to distinguish between different categories of ISA2 actions:

actions continued from ISA and actions started during ISA2; and

·actions already completed at the time of the interim evaluation and actions still ongoing.

The different stages of an action imply different levels of achievement of objectives and expected results, as some solutions are operational while others are in the early proof-of-concept stage. This should be kept in mind when looking at the performance indicators’ results (
[Box 4](#_Ref9596697)
).

Data show that the 20 sampled actions have developed 35 solutions so far, out of which 10 emerged under ISA2. As shown in 
[Table 2](#_Ref9594678)
, the solutions fall into one of the following four categories: common tools/services, common frameworks, common specifications/standards, and studies.

Table 2: Overview of solutions developed under the sampled actions, by type

|  |  |  |
| --- | --- | --- |
| Type of solution | Total number of solutions | New solutions developed by ISA2 |
| Common tools/services | 20 | 4 |
| Common frameworks | 9 | 2 |
| Common specifications/standards | 3 | 1 |
| Studies | 3 | 3 |
| TOTAL | 35 | 10 |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Aside from these four solution types associated with ISA2 actions, there is also a particular case represented by the ‘Raising Interoperability Awareness — Communication Activities’ action, which is part of the ‘Accompanying measures’ package. There are no solutions associated with this action, and its outputs are of a specific nature, namely events organised or events to which ISA2 representatives actively contributed. 
[Box 3](#_Ref9594886)
 presents an overview of these communication activities.

Box 3: Communication activities

|  |
| --- |
| From 2016 to the end of 2018, ten major events were organised as part of the ‘Raising Interoperability Awareness — Communication Activities’ action, with an average of 211 participants per event. The events were organised in six different countries: Belgium, Bulgaria, Greece, Italy, Malta, and Portugal, thus ensuring that a wide pool of professionals was reached.  In addition, ISA2 representatives participated in 60 events organised in 18 Member States and one EU candidate country (Montenegro) between 2016 and 2018. The number of participants in these events ranged from 125 to 6 000, thus ensuring that information about the programme was disseminated to numerous stakeholders.  [Tables 13](#_Ref9595195)  and  [14](#_Ref9595263)  in  [Annex 6](#_Ref9581764)  present an overview of these events. The ISA2 website also lists past and upcoming events [43](#footnote44) . |

An array of metrics are in use to assess the effectiveness of solutions, for example, the number of users, instances of use, downloads and page views. When these performance indicators are assessed in a more in-depth way, it is clear that they support the feedback received from consulted stakeholders that the programme is on track to achieving the expected results. However, it is also clear that more time is needed to fully achieve these results. This pattern is also confirmed when the performance indicators for actions continued from ISA are compared with those for actions started during ISA2, as detailed in 
[Box 4](#_Ref9596697)
.

Box 4: Comparing solutions and performance indicators of actions continued from ISA and actions started during ISA2

|  |
| --- |
| The pattern that emerges when the performance indicators associated with actions continued from ISA and those started during ISA2 are assessed can be illustrated using the two sampled actions that are part of the semantic interoperability package:  ·‘Public Multilingual Knowledge Management Infrastructure’ (PMKI) for the Digital Single Market (initiated during ISA2); and  ·‘SEMIC: Promoting semantic interoperability amongst the European Union Member States’ (a continuation from the action with the same name that was in place during ISA).  The ‘PMKI’ action has so far provided two solutions: i) a common framework in the form of a core data model for multilingual taxonomies/terminologies; and ii) a common specification in the form of semantic links between different language resources. The most suitable performance indicator associated with the core data model is measured by ‘instances of inclusion in academic journals and international conference’. Since the model was published in 2017, two scientific papers based on it have been published in international journals, as part of the action. In addition, the work carried out on the model has been presented at seven international conferences. With regard to the semantic links, the main performance indicator is measured by instances of collaboration with European public administrations with a view to establishing semantic interoperability between national language resources and EuroVoc [44](#footnote45) . Four collaborations have been set up so far.  In comparison, for the ‘SEMIC’ action, which has been in place longer than PMKI, the relevant performance indicators are the number of solution users, in particular the number of public administrations using the solutions. For the four solutions that are part of SEMIC, the numbers of users are as follows:  ·the ‘Asset Description Metadata Schema’ (ADMS) is used in eight public administrations in the Member States and the Commission, and by one company. In addition, the solution has been used to describe approximately 4 700 interoperability solutions on Joinup;  ·the ‘Core Vocabularies’ are used in 11 public administrations in the Member States and in the Commission;  ·the ‘Data Catalogue Vocabulary Application Profile for Data Portals in Europe’ (DCAT-AP) is used by 29 public administrations in the Member States and the Commission, and by associations and universities; and  ·‘VocBench3’, which also falls under SEMIC [45](#footnote46)  and previous versions of which are used in 14 public administrations in the Member States and the Commission, and by universities, institutes, and international organisations.  An analysis of the solutions developed under the PMKI and SEMIC actions suggests that the performance indicators assessing actions continued from ISA tend to show a higher degree of maturity and take-up than those developed under ISA2. Therefore, it is clear that solutions need time to show fully-fledged results. This is a natural conclusion, and since there is still over one year and a half left until the ISA2 programme concludes, there is still time for solutions to be further developed. |

JC5: Alignment with principles specified in Article 4(b) of the ISA2 Decision

The evaluation of effectiveness also requires an assessment of the programme’s compliance with the principles listed in Article 4(b) of the ISA2 Decision. The rolling work programme provides details on how the ISA2 actions should consider these principles. The overall judgment stemming from the desk analysis of the rolling work programme is positive (
[Annex 6](#_Ref9581764)
: 
[Table 15](#_Ref9596232)
). Supporting examples are provided by examining the contribution of ISA2 actions to the implementation of the EIF, as the principles in question also fall under the scope of the EIF (
[Annex 6](#_Ref9581764)
: 
[Table 23](#_Ref9597000)
).

Answer to the EQs on ‘effectiveness’

EQ2: How far are the ISA² programme’s results in the process of achieving the programme’s objectives?

The results achieved by ISA2 so far are aligned with the programme’s objectives. However, they do not yet fully match the expected results, as most of the actions are still ongoing and solutions are still being developed. More time is needed in order to achieve all the expected results. This is confirmed by the assessment of key performance indicators and a comparison between actions continued from ISA and actions started under ISA2.

External factors could improve but also jeopardise the way in which the programme achieves its objectives and delivers its results. The main findings hold true across the different stakeholder groups and are further confirmed by expert assessments. The call for common standards and frameworks from public administrations is an external factor that contributes to the programme’s performance. In contrast, institutional complexity could harm the achievement of cross-border and cross-sectoral interoperability.

EQ3: Are there aspects that are more or less effective than others are, and if so, what lessons can be drawn from this?

While in general the programme is on track to achieving its objectives, it performs relatively less well when it comes to developing a more effective, simplified and user-friendly e-administration at the national, regional and local levels of public administration. This is an area where Member States can in fact have a greater impact. National initiatives related to the ICT modernisation of the public sector can also improve the overall performance of ISA2 by increasing the capacity for the take-up of ISA2 solutions.

Regarding the take-up rate of solutions, the evaluation found that the actions that continued from previous editions of the programme have produced solutions that are now more widely used than solutions resulting from newly established ISA2 actions.

As regards the specific request of Article 13(5) of the ISA2 Decision to assess the programme’s compliance with the principles set out in Article 4 of this decision, the evaluation found that the rolling work programme process effectively ensures that the ISA2 actions take into consideration the core principles listed in the ISA2 Decision.

When it comes to communication and awareness-raising activities, the evaluation results suggest that ISA2 duly acted upon the recommendation of the final ISA evaluation and increased general awareness of the programme. Still, some aspects of ISA2 could benefit from more promotion. For example, as legal interoperability is specifically mentioned among the operational objectives, the solutions developed for assessing the ICT and interoperability impacts of legislation could be more prominently featured in order to improve awareness among stakeholders. Moreover, ISA2 could better stress the benefits of implementing certain interoperability solutions. To this end, it could run studies that quantify the impact of interoperability solutions on the efficiency and productivity of public administrations. Such studies would improve the conditions for assessing the programme’s effectiveness and could have positive spillover effects on dissemination and communication activities: the monetary benefits deriving from ISA2 solutions would be better emphasised, thus providing a better basis for stakeholder support of future EU efforts promoting interoperability. Finally, as noted under section 5.1 
[Relevance](#_Ref9609437)
, ISA2 communication activities should better target the sub-national layers of administrations.

Efficiency

Evaluation framework for ‘efficiency’

Efficiency relates to the extent to which the programme’s objectives are achieved at a minimum cost. This evaluation criterion includes two EQs and one sub-question:

EQ4:To what extent has the programme been cost-effective?

EQ5:Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised?

EQ5.1:
   How is the programme performing relative to the planned work and budget?

To answer these questions, the evaluation focused on three judgment criteria, relying on data collected from the ISA2 dashboard, the monitoring and evaluation reports and consultation activities:

JC1:Efficiency of the selection process of the actions to be included in the rolling work programme;

JC2:Cost-effectiveness analysis to assess the ratio between allocated funds and actual results of the programme; and

JC3:Earned value management analysis.

Analysis

JC1: Efficiency of the action selection process

The ISA2 programme is implemented in four phases: i) submission; ii) evaluation; iii) adoption; and iv) implementation (further details are provided in Chapter 3). The process to select ISA2 actions includes the first three phases, i.e. submission, evaluation, and adoption. Note that this section does not assess the selection of contractors who will implement the actions, as this process follows the general European public procurement rules.

Several elements of the ‘submission’ phase need to be explained here. First, it is important to note that, in order to prepare and submit a proposal to be included in the ISA2 rolling work programme, the applicant must: i) find out about the call for proposals; ii) study the documents of the call for proposals and understand the rules and procedures; iii) prepare a concise description of the proposed action in compliance with the ‘work programme entry template’; iv) collect the required internal authorisations to submit the proposal; and v) submit the proposal by e-mail. Second, it is important to distinguish between two types of proposal, namely proposals for new actions and proposals for continuing actions that have already started. This distinction is relevant when it comes to assessing the overall workload required for submitting a proposal.

The efficiency of the action selection process (which includes the submission, evaluation and adoption phases) is assessed based on feedback received from programme governance representatives and action owners, as they are the stakeholder groups directly involved in the process. Most respondents consider the process fit-for-purpose at least to some extent: 16 out of 19 consulted programme governance stakeholders and 21 out of 23 consulted action owners (
[Figure 11](#_Ref9675404)
). However, 13 out of the 42 consulted stakeholders indicated that some measures could still be taken to further streamline the process. The measures mentioned by several stakeholders include:

Simplifying the rolling work programme, which can be done by:

oMaking the template more flexible to account for action specificities. This would ensure better input of information, which would make the process of drafting the proposals more efficient, and would also make it easier for anyone seeking specific information about ISA2 actions to read and understand the work programme.

oExtending the duration of a rolling work programme to cover more than one year, thus allowing for multiannual actions. In principle, actions to be run on a multiannual basis can be proposed, but the budget is allocated on an annual basis. Therefore, action owners must submit an updated proposal every year as part of the rolling work programme in order to access funding.

oSetting up an electronic data input system rather than full text editing.

·Simplifying the overall process by launching thematic calls for actions, thus ensuring that the selection is more objective-driven.

Figure 11: Extent to which the action selection process is fit-for-purpose (number of respondents by stakeholder category)

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

As regards the submission phase of the selection process, 26 out of the 42 consulted action owners and programme governance representatives indicated that they had prepared and submitted proposals. Out of these 26 respondents, 21 also specified the amount of time they spent on preparing the proposal. The answers provided vary between 1 to 30 person-days, with the average being 9 person-days. The large variation can be primarily explained by the differences in the efforts required to submit a proposal for a new action versus the amount of effort required to renew a proposal for an existing action.

As regards the time needed for submission, 
[Figure 12](#_Ref9675447)
 shows a clear concentration of answers: 14 of the respondents indicated that they finalised their proposal in 1 to 6 person-days. However, four respondents indicated that 20 or more person-days were needed. This distribution of answers and the qualitative inputs provided by some of the respondents during the consultation activities makes it possible to differentiate between preparing a renewed proposal and a brand-new proposal. Renewing a proposal for an action that has already been included in a previous rolling work programme tends to require between 1 and 6 person-days at most, while preparing a new proposal takes significantly longer, between 10 and 30 person-days. In most cases, stakeholders prepare the proposals themselves, without relying on external contractors.

Additional time is also required to merge similar proposals, upon request by the Commission team in charge of shortlisting the proposals to be funded.

Figure 12: Distribution of the number of days needed to prepare and submit a proposal for an ISA2 action (number of respondents)

Note: The histogram was constructed using a bin size of 2 days, which grouped the answers of respondents in intervals of 2 days. This bin size was deemed most appropriate after reviewing the answers provided.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

The time spent to prepare and submit a proposal for an action to be included in the rolling work programme, whether it is a new or a renewed proposal, can be used to measure regulatory costs by applying the standard cost model
[46](#footnote47)
, presented in the Better Regulation Toolbox
[47](#footnote48)
. First, person-days are converted into person-hours by assuming that each person-day corresponds to eight hours
[48](#footnote49)
. Then, the overall number of hours is multiplied by a standard tariff, i.e. the 2017 hourly labour cost for the service sector at the Member State level (see 
[Annex 6](#_Ref9581764)
: 
[Table 22](#_Ref9675970)
)
[49](#footnote50)
.

Two respondents indicated that they relied on external service providers to prepare the proposal. In this case, regulatory costs would also include out-of-pocket expenses incurred for such services. 
[Table 3](#_Ref9686590)
 presents the average regulatory costs both taking into account the fees related to external service providers and without accounting for these additional fees. The average respondent faced regulatory costs of approximately €1 077 for renewing a proposal and €5 651 for preparing a new proposal, without accounting for fees incurred when relying on external service providers. Considering the two instances when external service providers were used in the preparation of proposals, the average regulatory cost for a renewed proposal amounted to €1 138 and for a new proposal the average regulatory cost was €6 590. At the same time, the average funding allocated to an ISA2 action between 2016 and 2018 amounted to €1 217,566
[50](#footnote51)
. The costs incurred to prepare a proposal are thus very small, ranging between 0.09 % and 0.5 % of the potential funding that could be allocated to the proposal once accepted.

Table 3: Average regulatory costs incurred by respondents who prepared and submitted proposals for ISA2 actions

|  |  |  |  |
| --- | --- | --- | --- |
|  | Average regulatory cost, including external services (€) | Average regulatory cost without external services (€) | Average funding allocated per action (€) |
| Renewed proposal | 1 138 | 1 077 | 1 217,566 |
| New proposal | 6 590 | 5 651 |  |

Note: The calculations are based on a sample of 21 respondents who specified the time spent on preparing proposals.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

One respondent mentioned that the reason they relied on external service providers in this process was the need to tailor the proposal to the requirements of the rolling work programme template. As such, one potential cost-reduction method could be the simplification of the rolling work programme template, thus further corroborating the measures proposed to streamline the action selection process.

JC2: Cost-effectiveness analysis

Based on the performance indicators discussed in section 5.2 on effectiveness and on data related to the actual costs of actions, a cost-effectiveness analysis
[51](#footnote52)
 could be performed. However, it is important to note the limitations of this method. First, the heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness (
[Box 5](#_Ref9691206)
 shows the variety of ISA2 performance indicators). Consequently, computing an aggregate value that is representative of the programme or action packages is quite difficult. Second, such data are only partially available, as some ISA2 solutions are still under development. In addition, as concluded in section 5.2, it is expected that the values of the performance indicators will improve over time, as solutions tend to require longer periods to fully show their results. Third, data on effectiveness are not yet available for a limited number of solutions developed so far.

Against this background, the cost-effectiveness analysis can be exemplified by using some of the ISA2 packages that rely on similar performance indicators. Two packages fit this profile: ‘Key and generic interoperability enablers’ and ‘eProcurement / eInvoicing — supporting instruments’. The sampled actions included in both packages use the number of public administrations (at EU, national and regional levels) that use the solutions generated by the actions as a performance indicator. 
[Table 4](#_Ref9690327)
 presents the estimated average cost per solution user based on the available data on the number of solution users (public administrations) and the actual costs of the action between 2016 and Q3 2018. For the ‘Key and generic interoperability enablers’ package, the average actual cost
[52](#footnote53)
 per public administration using the solutions that are part of this package amounts to €19 420. For the ‘eProcurement / eInvoicing — supporting instruments’ package, the average actual cost amounts to €25 870. Note that these average costs are calculated per public administration using the solution rather than per single end-user. Each administration serves millions of end-users (citizens/businesses); hence, costs per end-user are estimated to be low.

Table 4: Costs per user of ISA2 solutions

|  |  |  |  |
| --- | --- | --- | --- |
| Package | Actual costs of sampled actions in the package (€) | Number of public administrations using the solutions of the package | Average actual cost (€) |
| Key and generic interoperability enablers | 4 117,069.81 | 212 | 19 420 |
| eProcurement / eInvoicing — supporting instruments | 3 544,135.98 | 137 | 25 870 |

Note: The calculations are based on the sampled actions of the indicated ISA2 packages over the period 2016 and Q3 2018. A full overview of the number of users and other performance indicators is presented in 
[Annex 6](#_Ref9581764)
: 
[Table 12](#_Ref9587649)
. The overview of costs per package is shown in 
[Figure 13](#_Ref9692490)
.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

The appraisal of the programme’s cost-effectiveness could be made more consistent by ensuring that monitoring and evaluation reports quantify some performance indicators common to all actions. Examples of such indicators include the number of EU public administrations using a solution, the number of public administrations participating in pilot projects, a consistent approach to reporting the number of downloads from the ‘Joinup’ platform for solutions that can be downloaded, and/or the number of single users per solution.

Box 5: Summary of the main costs and benefits identified

|  |
| --- |
| Costs. The main costs associated with the intervention fall into two categories: regulatory costs of the selection process for ISA2 funded actions (see  [Table 3](#_Ref9686590) ) and the costs per user of ISA2 solutions (see  [Table 4](#_Ref9690327) ).  Benefits. In turn, the benefits generated are expressed in the form of performance indicators attached to each solution developed as part of ISA2 actions, which can be measured using a variety of metrics. The benefits generated by the programme can thus be summarised as follows (the list is not exhaustive):  ·280 Commission proposals have been screened for ICT impacts since 2015;  ·13 440 professionals working in the field of eGovernment are registered on ‘Joinup’ in order to access interoperability solutions and collaborate with one another;  ·over 8.8 million documents have been exchanged so far between the Commission, the Council, the Member States, and companies using the ‘e-TrustEx’ platform;  ·the ‘European Single Procurement Document’ (ESPD) service [53](#footnote54)  has been visited around 4 million times (until April 2019); and  ·the ‘Core Public Service Vocabulary-Application Profile’ (CPSV-AP) solution has been downloaded from ‘Joinup’ over 2 500 times.  Additional performance indicators, collected as part of the desk review of secondary data, are included in  [Annex 6](#_Ref9581764) :  [Table 12](#_Ref9587649) . |

JC3: Earned value management (EVM) analysis

EQ5 and EQ5.1 focus on progress made in implementing the programme based on the EVM analysis. This analysis helps determine progress of work against a given baseline, so that costs, time, and scope of a certain activity are constantly tracked. To support this, the ISA2 dashboard provides quarterly data on the earned value, planned value, and actual costs of each monitored action
[54](#footnote55)
. As regards the sample of 20 actions grouped in the respective packages, 
[Figure 13](#_Ref9692490)
 presents the data at the end of Q3 2018 for the earned and planned value, as well as for the actual costs recorded. The earned value is close to equalling the planned value, which shows that the work is progressing as expected. Even where a bigger gap exists (e.g. in the ‘Key and generic interoperability enablers’ package), the indicators are expected to converge at the end of the year as the ISA2 budget year closes.

Figure 13: Earned value, actual costs and planned value sampled actions, by package (Q3 2018)
[55](#footnote56)

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

The data available also make it possible to perform a historical analysis of work progress for each package of actions. Progress can also be shown in the form of a schedule performance index (SPI), which is the ratio of earned value to planned value (
[Figure 14](#_Ref9692983)
). An SPI value above one indicates that the item analysed is ahead of schedule, while an SPI value below one suggests that the item analysed is behind schedule. Since the beginning of the programme and up until Q3 of 2018, the SPIs of the sampled actions grouped by packages have shown fluctuations, but now tend to converge on one, although values below one are still reported for almost all packages. These data show that the ‘Semantic interoperability’ and ‘Supporting instruments for public administrations’ packages are on track, while the others are close to achieving the planned level of work. Given that some actions started later on in the programme, as is the case of the ‘Geospatial solutions’ package, data for the EVM analysis are not always available from Q2 of 2016, but only from a later point in time.

SPI values above two were reported for the ‘Key and generic interoperability enablers’ package in Q2 and Q3 of 2016. This indicates that the work on the package was ahead of schedule in that period. While this result might seem surprising at first, it can be explained by the fact that some of the actions involved started under ISA and continued under ISA2. For these actions, processes had already been established during the previous programme, so work under ISA2 could start much faster than planned, resulting in a higher SPI value at the beginning of the process. In fact, both of the actions sampled from the ‘Key and generic interoperability enablers’ package (‘Trusted Exchange Platform e-TrustEx’ and ‘Catalogue of Services’) had been launched under the ISA programme.

Figure 14: Schedule performance indices of the sampled actions grouped by packages

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Answer to the ‘efficiency’ EQs

EQ.4: To what extent has the programme been cost-effective?

The heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness. For those packages where it was possible to apply cost-effectiveness techniques, costs per end-user (e.g. business, citizens, etc.) have been estimated as low. In this respect, monitoring and evaluation reports could converge towards some common metrics allowing for a more consistent appraisal of cost-effectiveness. For example, the agreed performance indicators could comprise the number of EU public administrations using a given solution, the number of downloads from the ‘Joinup’ platform for downloadable solutions, and/or the number of single users per solution.

EQ5: Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised?
  
   EQ5.1: How is the programme performing relative to the planned work and budget?

The process of selecting actions to be funded by ISA2 is considered relatively efficient and fit for purpose. On average, six person-days at most are required to prepare and submit a proposal for actions that had already been included in previous rolling work programmes. On the other hand, between 10 and 30 person-days are required to prepare a proposal for a new action. The costs incurred in preparing a proposal are relatively small, ranging between 0.09 % and 0.5 % of the potential funds that could be allocated to the proposal once accepted. Nonetheless, this process could be further streamlined by simplifying the rolling work programme and launching thematic calls for actions, thus ensuring that the selection process is driven by objectives.

When looking at progress made in the programme’s implementation, it is clear that all packages are either on track or close to achieving the planned level of work. As some actions started during ISA and continued under ISA2, some processes had already been established during the previous programme. Therefore, work on these actions under ISA2 could start faster than planned.

Coherence

Evaluation framework for ‘coherence’

Article 13(4) of the ISA2 Decision requires that the coherence of ISA2 actions is assessed; this is referred to as the programme’s ‘internal coherence’. In addition, Article 13(6) pays specific attention to the programme’s ‘external coherence’ in order to ‘identify potential overlaps, examine coherence with areas for improvement, and verify synergies with other Union initiatives, in particular with the CEF’. The coherence criterion translates into two EQs:

EQ6:To what extent do the ISA2 actions form part of a ‘holistic’ approach within the framework of the programme? (Internal coherence)

EQ7:To what extent is the ISA2 programme coherent with other EU interventions, which have similar objectives and with global initiatives in the same field? (External coherence)

To assess both the internal and the external coherence of the programme, findings from desk research on links between ISA2 actions and the relationship between ISA2 and other relevant EU programmes/policies/initiatives were used in addition to the information provided by stakeholders. In this context, the following judgment criteria are considered:

JC1:Internal coherence:

-Degree of coherence among actions funded by the ISA² programme;

-Level of reuse of results of a funded action by another action within the ISA² programme;

JC2:External coherence:

-Degree of coherence between the programme and other EU supported programmes;

-Level of reuse of results delivered by ISA2 actions by other EU programmes;

-Degree of coherence between the programme and other EU policies; and

-Degree of coherence between the programme and global initiatives in the field.

Analysis

JC1: Internal coherence

Given that the ISA2 programme currently manages 53 actions
[56](#footnote57)
, it is important to ensure that they work together to create benefits and that work is not duplicated. The stakeholder groups consulted on this matter (programme governance, action owners, solution users) tend to agree that ISA2 actions generate synergies (
[Figure 15](#_Ref9701062)
). At the same time, most stakeholders indicate only limited overlaps between ISA2 actions. Stakeholder feedback on synergies and overlaps, as two sides of the same coin, paints a positive picture of the internal coherence of the ISA2 programme.

Only 4 % of respondents (2 out of 53) identify a high degree of overlaps between ISA2 actions, while the vast majority, 77 % (41 out of 53), consider overlaps as either not existing or being limited. In addition, respondents point to the existence of synergies: 41 % of respondents (27 out of 66) identify synergies as existing to a high extent or to the fullest extent and 17 % (11 out of 66) identify either no synergies or limited synergies. It is relevant to note that a significant number of solution users indicated that they were unaware of either synergies or overlaps between ISA2 actions. This feedback can be explained by stakeholders’ various levels of knowledge about ISA2 packages. As not all stakeholders are familiar with all ISA2 packages, they might not be in a suitable position to identify potential synergies or overlaps.

Figure 15: Extent to which there are synergies / overlaps between ISA2 actions (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

The review of documentary evidence confirms the results of the consultation activities, particularly regarding the existence of synergies between ISA2 actions. The indicator of the ‘network’ of relationships between the programme’s actions
[57](#footnote58)
, as shown on the ISA2 dashboard, is particularly relevant here. The quantitative assessment of the data provided online for the sample of 20 actions under assessment shows (see 
[Annex 6](#_Ref9581764)
: 
[Table 17](#_Ref9701848)
) that there are multiple instances of internal reuse of ISA2 solutions, with actions like ‘SEMIC’ and ‘Joinup’ playing a particularly important role in the system.

At the same time, some actions tend to be less central in the network. This is the case particularly for the ‘EUSurvey’
[58](#footnote59)
 and ‘European Citizens’ Initiative and European Parliament Election’
[59](#footnote60)
 actions. The specificity of these actions makes it more likely that their solutions are reused by other programmes or initiatives, rather than by other ISA2 actions.

Throughout the consultation activities, additional inputs were collected from respondents concerning synergies between actions and examples of instances when results delivered by an ISA2 action were used by another action (see 
[Box 6](#_Ref9703039)
).

Box 6: Examples of internal coherence between ISA2 actions

|  |
| --- |
| ·The ‘DCAT-AP’ specification, developed under the ‘SEMIC’ action, has been reused in the development of the ‘CPSV-AP’ under the ‘Catalogue of Services’ action. ‘Access to base registries’ reuses ‘Core Vocabularies’, ‘DCAT-AP’ (both part of the ‘SEMIC’ action) and ‘EIRA’ (part of the ‘European Interoperability Architecture’ action). In addition, ‘SEMIC’ solutions have also been used by the ‘ELISE’ action.  ·The ‘Re3gistry’ solution, part of the ‘ELISE’ action, is used to share codes developed for ‘Core Vocabularies’ (part of the ‘SEMIC’ action).  ·The platform for secure file exchange ‘e-TrustEx’ (part of the action with the same name) is used in the area of eProcurement by the ‘European Public Procurement Interoperability Initiative’ action, in particular by the ‘e-Prior’ solution developed as part of the action. ‘E-Prior’ is a module built based on ‘e-TrustEx’.  ·As regards synergies with the ‘Joinup’ platform, there is a particularly strong relationship between the ‘Sharing and Reuse’ action and ‘Joinup’. The former aims to support public administrations in sharing collaborative development, reusing IT solutions and promoting best practices; the latter is a collaborative platform providing the means to achieve this.  ·The ‘National Interoperability Framework Observatory’ (NIFO) action is reusing some of the information from the ‘Access to base registries’ action in order to compile the eGovernment factsheets. |

JC2: External coherence

In order to assess the external coherence of ISA2, stakeholders were asked to assess the level of synergies and overlaps existing between ISA2 and four EU programmes identified as relevant for the field of digitalisation of public administrations and interoperability: i) CEF
[60](#footnote61)
, ii) ESF
[61](#footnote62)
, iii) Horizon 2020
[62](#footnote63)
 and iv) SRSP
[63](#footnote64)
. The results are presented in 
[Figure 16](#_Ref9703935)
.

Figure 16: Extent to which there are synergies / overlaps between ISA2 and other relevant EU programmes (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Out of the four programmes, respondents to the consultation activities identify the CEF as having the highest level of synergies with ISA2. According to 78 % of respondents (51 out of 65), ISA2 and CEF have a high level or a very high level of synergies. As an example, both programmes contribute to the Commission’s open data policy: the DCAT-AP solution of ISA2 provides a metadata specification used by open data portals across the EU and CEF ensures funding for the public open data infrastructure that aims to increase the reuse of open public data in the EU. However, 18 % of respondents (10 out of 57) also note the existence of overlaps to a high extent or to the fullest extent, which could be addressed by better delineating the scope of the concerned solutions.

When it comes to the relationship between Horizon 2020 and ISA2, half of the respondents (31 out of 63) identify synergies to a high extent or to the fullest extent, while only 12 % (6 out of 51) indicate the existence of overlaps to a high extent or to the fullest extent. The situation is similar for SRSP, with 52 % of respondents (22 out of 42) pointing out that there are synergies between ISA2 and SRSP either to a high extent or the fullest extent, and only 6 % (2 out of 32) indicating a high or very high degree of overlaps.

At the opposite end of the spectrum, out of the four programmes listed above, the ESF seems to be the programme with the lowest level of coherence with ISA2. Only 26 % of respondents (6 out of 23) indicate the existence of synergies between the ESF and ISA2 to a high extent or to the fullest extent.

Respondents also pointed to synergies and overlaps with other EU initiatives. More specifically, three additional EU programmes and initiatives were mentioned: the European Regional Development Fund (ERDF)
[64](#footnote65)
, corporate IT governance
[65](#footnote66)
 and the Single Digital Gateway
[66](#footnote67)
. This feedback is discussed in the points below.

While the synergies between the ESF and ISA2 appear to be limited, two respondents pointed to the existence of a high synergy level between the ERDF and ISA2. At the same time, one respondent indicated that there could be overlaps between ISA2 and ERDF, possibly due to a lack of coordination. More specifically, national IT systems developed using ERDF funds could be incompatible with ISA2 solutions, thus preventing the adoption of these solutions.

A high degree of complementarity was also identified between the Commission’s corporate IT governance and ISA2, as explained by one respondent. The corporate IT governance team maintains that the Commission’s proposed IT investments take into consideration the reuse of IT solutions that have already been developed. In this sense, the work of this team supports synergies between ISA2 and other initiatives, ensuring that ISA2 solutions and any other IT solutions developed so far are reused in the future. In addition, the team helps mitigate the risk of developing very similar IT solutions under different spending programmes, which would create overlaps.

·One respondent also mentioned the existence of synergies with the Single Digital Gateway (SDG). In fact, this is clear given that ISA2 implemented an action dedicated to the interoperability requirements of the SDG in 2018
[67](#footnote68)
. ISA2 helped detail the IT architecture of the SDG and identify functional, technical and semantic interoperability challenges in its implementation. Further ISA2 solutions like ‘eForms’
[68](#footnote69)
 and ‘e-Documents’
[69](#footnote70)
 also support the SDG’s implementation.

[Table 18](#_Ref9706698)
 in 
[Annex 6](#_Ref9581764)
 shows that ISA2 actions have multiple links outside of the programme, as they rely on other EU programmes / policies / initiatives to deliver their results and their solutions are used by other programmes / policies / initiatives.

Beyond examining alignment with the initiatives mentioned above, it is worth analysing the degree of alignment of ISA2 with other initiatives or broader policies, like the Digital Single Market Strategy, the eGovernment Action Plan, ICT standardisation or the Tallinn Declaration on eGovernment.

When it comes to the Digital Single Market (DSM) Strategy
[70](#footnote71)
, the desk review of the rolling work plan reveals that ISA2 contributes to all three of the strategy’s pillars (see 
[Annex 6](#_Ref9581764)
: 
[Table 19](#_Ref9707503)
): (i) better access for consumers and businesses to digital goods and services across Europe; (ii) creating the right conditions and a level playing field for digital networks and innovative services to flourish; and (iii) maximising the growth potential of the digital economy. Based on the 20 sampled actions, ISA2 in particular contributes to the strategy’s third pillar through:

the revision and implementation of the new European Interoperability Framework, which is explicitly mentioned in the DSM strategy;

work on supporting the implementation of the once-only principle, carried out as part of several actions such as ‘SEMIC’ and ‘European Interoperability Architecture’; and

·support for the implementation of the Single Digital Gateway.

Together with the launch of the DSM strategy for Europe, a new eGovernment Action Plan was announced for 2016-2020, aiming to ‘remove existing digital barriers to the Digital Single Market and to prevent further fragmentation arising in the context of the modernisation of public administrations’
[71](#footnote72)
. The ways in which ISA2 contributed to implementing the DSM strategy also directly relate to the eGovernment Action Plan. In particular, the revised version of the EIF and the ISA2 actions monitoring its implementation in the Member States (‘EIF implementation and governance models’ and ‘NIFO’) address Action 4 of the eGovernment Action Plan. Furthermore, ISA2 is also contributing to Actions 1 and 6 through its work in the field of eProcurement. For instance, the ‘European public procurement interoperability initiative’ action facilitates the implementation of eProcurement in European public administrations and supports the EU’s TOOP
[72](#footnote73)
 project by adjusting the necessary tools for the once-only principle.

In the field of ICT standardisation, the Rolling Plan for ICT Standardisation
[73](#footnote74)
 provides a good overview of the EU’s main initiatives and priorities in this area. The Commission drafts this plan in collaboration with the European Multi-Stakeholder Platform on ICT Standardisation, which includes representatives of the Member States, EFTA countries, European and international standardisation organisations, industry, SMEs and consumers. The 2018 Rolling Plan on ICT Standardisation points out two main areas with highly relevant synergies between ISA2 and ICT standardisation:

1.eGovernment. The Rolling Plan emphasises that in order to leverage the applicability of technical specifications developed under ISA2, one solution would be to suggest using them as standards by the relevant standardisation organisations for the specifications in question. In this respect, it is worth noting that W3C, an international standardisation forum, has discussed some ISA2 solutions (like the ‘ADMS’ specification
[74](#footnote75)
) and has published a W3C note
[75](#footnote76)
.

2.Public Sector Information, Open Data and Big Data. The Rolling Plan also mentions the usefulness of introducing a common standard for referencing open data in European open data portals. The ‘DCAT Application Profile’
[76](#footnote77)
, an ISA2 solution that is already used by several open data portals across the EU, stands out as a candidate for this purpose. The Rolling Plan invites the CEN ‘to support and assist the DCAT-AP standardisation process’. 
  
There are several potential issues with standardisation though. More specifically, the evaluation found that the coherence between intellectual property rights for ISA2 solutions and CEN/CENELEC standards needs to be clarified and resolved before the standardisation system could be fully implemented
[77](#footnote78)
.

Finally, in order to gain a comprehensive understanding of EU-level coherence, it is also useful to analyse the alignment of ISA2 with the priorities set at the highest political level in the Tallinn Declaration on eGovernment
[78](#footnote79)
. The assessment in 
[Box 10](#_Ref2603037)
 under 
[Annex 6](#_Ref9581764)
 concludes that there is no contradiction between the Tallinn Declaration and the ISA2 programme: even though they have different scope and aims, they proceed in the same direction and share common goals. However, strong coordination is required between them to build synergies and avoid overlaps and duplication.

Answer to the ‘coherence’ EQs

EQ6: To what extent do the ISA² actions form part of a ‘holistic’ approach within the framework of the programme? (Internal coherence)

ISA2 actions are characterised by substantial synergies with each other and limited overlaps. Stakeholder feedback on synergies and overlaps shows a positive picture of the internal coherence of the ISA2 programme. There are multiple instances of internal reuse of ISA2 solutions, with actions like ‘SEMIC’ and ‘Joinup’ playing a particularly important role in the system. At the same time, some actions tend to be less central within the network. This is for instance the case of the ‘EUSurvey’ and ‘European Citizens’ Initiative and European Parliament Election’ actions and can be explained by the particular nature of these actions.

This strong internal coherence is in line with the second recommendation of the final ISA evaluation, which called for taking a holistic approach on interoperability within ISA2.

EQ7: To what extent is the ISA² programme coherent with other EU interventions, which have similar objectives and with global initiatives in the same field? (External coherence)

When it comes to external coherence, ISA2 followed the third recommendation of the final ISA evaluation and developed close cooperation with other EU policies and initiatives. Respondents to the consultation activities identified the highest level of synergies between ISA2 and CEF. They also pointed out the synergies with ‘other’ EU initiatives such as the ERDF, corporate IT governance and the Single Digital Gateway. Synergies also exist between ISA2 and some broader EU initiatives and policies, such as the Digital Single Market strategy, the eGovernment Action Plan, the Tallinn Declaration on eGovernment, and the Rolling Plan for ICT standardisation.

In conclusion, ISA2 actions have multiple links outside of the programme, as they rely on other EU programmes / policies / initiatives to deliver their results and their solutions are used by other programmes / policies / initiatives. This further corroborates the positive feedback on the programme’s external coherence.

EU added value

Evaluation framework for EU added value

The EU added value captures the programme's impacts additional to those that would be achieved if the issues addressed by ISA2 were left solely in the hands of national and sub-national authorities. This evaluation criterion seeks to answer the following EQ:

EQ8:What is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels?

To answer this question, the evaluation analysed the following judgment criteria based on both primary and secondary data:

JC1:Achievement of objectives that could not be otherwise attained with national or sub-national interventions;

JC2:Achievement of objectives at a cost lower than what could be attained via national or sub-national interventions;

JC3:Achievements in terms of cross-border interoperability; and

JC4:The contribution to the advancement of common EU policies.

Analysis

JC1: Achievement of objectives that could not have been achieved without EU-level intervention

Most respondents believe that national or sub-national interventions would not be able to achieve the general and specific objectives in the field of interoperability that ISA2 is pursuing, or that they would achieve only be able to achieve them to a limited extent (
[Figure 17](#_Ref9755929)
). This observation is particularly evident when it comes to developing, maintaining and promoting a holistic approach to interoperability in the Union (specific objective 1), as 83 % of respondents (95 out of 115) believe that national or sub-national interventions would not be able to achieve this objective or would achieve it only to a limited extent. Additionally, 72 % of respondents (83 out of 116) point to limited achievements or no achievement from national or sub-national initiatives with regard to facilitating the efficient and effective electronic cross-border or cross-sector interaction between European public administrations, and between European public administrations and businesses and citizens (specific objective 2). These examples show that the coordination provided by a unique EU-level programme plays an important role in improving overall interoperability among European public administrations. The technical experts supporting the evaluation have unanimously confirmed this conclusion.

Figure 17: Extent to which national or sub-national interventions would be able to achieve the ISA2 objectives without the programme (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

There is, however, one specific area in which national or sub-national interventions would be able to contribute in the absence of an EU-level programme. A quarter of respondents (29 out of 115) believe that national or sub-national interventions would be able to contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels (specific objective 3) to a high extent or to the fullest extent. This result shows how well EU and national initiatives complement each other, since specific objective 3 is the one where the ISA2 programme shows the worst performance in terms of effectiveness (see 
[Figure 6](#_Ref9584015)
 in section 5.2).

It is worth emphasising that while national initiatives related to the ICT modernisation of the public sector cannot achieve the objectives pursued by ISA2, they are still very important to improving the programme’s performance as they increase the capacity for the take-up of ISA2 solutions. Initiatives pursued by Member States in this field show that interoperability is a priority, which encourages working together with ISA2. In addition, from the point of view of ICT development, national initiatives put Member States in a better position to use the solutions provided by ISA2.

JC2: Achievement of objectives at a lower cost

In addition, 83 % of respondents (91 out of 109) emphasised that ISA2 is probably or definitely able to achieve its objectives at a lower cost than comparable national or sub-national initiatives (
[Figure 18](#_Ref9756631)
). The main stakeholders of ISA2 appreciate the economies of scale that this EU-level programme generates.

Figure 18: Ability of ISA2 to achieve the relevant objectives at a lower cost than comparable national or sub-national interventions (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

JC3: Achievements in terms of cross-border interoperability

ISA2 focuses on providing both cross-border and cross-sectoral interoperability solutions. From the point of view of EU added value, the contribution to cross-border interoperability is particularly relevant, as it shows the extent to which the programme works towards achieving a holistic approach to interoperability in the EU.

Over half of the respondents (50 out of 93) believe that ISA2 has contributed to a high extent or to the fullest extent to improving cross-border interoperability in the EU. This result holds true across the consulted stakeholder groups, in comparison to less than 6 % (6 out of 93) who only see a limited contribution in this area (
[Figure 19](#_Ref9780893)
).

Figure 19: Extent to which ISA2 contributed to improving cross-border interoperability in the EU (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Several respondents also emphasised the following aspects:

Awareness: ISA2 raises awareness of interoperability in the EU and helps put the topic on national agendas; without this, little progress would be made in the field.

·Organisational contribution: ISA2 brings people together, creating networks and helping national organisations meet their counterparts in different countries, thus facilitating exchanges between Member States.

Desk research also supports the two examples mentioned above. First, ISA2 is committed to raising awareness about interoperability through its communication activities presented in 
[Box 3](#_Ref9594886)
. Second, it contributes to the creation of a network of officials working on interoperability by holding regular ISA2 Committee and ISA2 Coordination Group meetings, which bring together representatives of Member States responsible for this topic. There were five ISA2 Committee meetings and three Coordination Group meetings between 2016 and 2018. In most of them, all 28 Member States were represented by at least one participant (with a maximum of three participants per meeting per Member State). In addition, Norway was represented in all Coordination Group meetings and in four of the five Committee meetings, Iceland was represented in three Committee meetings and one Coordination Group meeting, and finally Montenegrin representatives were present in three Committee meetings and one Coordination Group meeting (see 
[Annex 6](#_Ref9581764)
: 
[Table 20](#_Ref9781618)
 and 
[Table 21](#_Ref9781624)
).

The fact that countries beyond the EU have either become members of ISA2 or cooperate with the programme shows that the programme’s benefits do not stop at the EU borders. An additional example is the administrative cooperation between DG DIGIT and the Agency for the Development of the Government of Electronic Management and the Information and Knowledge Society of Uruguay, officially started on 26 March 2018 and consisting of delegation visits, exchanges of experts, workshops, and sharing of best practices.

Further evidence comes from analysing the cross-border interoperability angle of the 20 sampled ISA2 actions: 29 out of the 35 operational solutions developed by these actions have a cross-border feature (
[Annex 6](#_Ref9581764)
: 
[Table 11](#_Ref11679432)
). One example is the IMAPS solution
[79](#footnote80)
, which facilitates the assessment of public services at the local, national, and EU level from the point of view of their interoperability maturity. In practice, it consists of a compact and user-friendly self-assessment carried out through the IMAPS online survey. This 30-minute questionnaire looks at the context of the public service, how the service is delivered, how it consumes existing services and how it is managed. On completion of the IMAPS survey, the results give tailored recommendations and confidential guidance for improvement. 68 public administrations completed the assessment in 2017 and 51 in 2018 — at local, national and EU level. The solution has thus had a cross-border impact, since the assessments are directed at public administrations all across the EU as confirmed by the user testimonial in 
[Box 7](#_Ref11683292)
.

Box 7: IMAPS user testimonial

![](./../../../resource.html?uri=comnat:SWD_2019_1615_FIN.ENG.xhtml.SWD_2019_1615_FIN_ENG_24005.jpg)

Source: ISA2 Communication Team

Further key achievements include (
[Annex 6](#_Ref9581764)
: 
[Table 22](#_Ref9675970)
):

The ‘e-TrustEx’ platform is a key enabler of secure information exchange for seven pan-European projects in which over 200 public administrations from all Member States participate.

The ‘Big Data for Public Administrations’ action created a working group to assess the needs in the area of big data analytics for policy making with the participation of the Czech Republic, Estonia, Hungary, Malta, Norway, Slovenia, Spain, and Portugal.

Several actions foster multilingualism:

othrough their work on providing common frameworks, the actions on semantic interoperability contribute to addressing multilingual challenges to the cross-border interaction of European public administrations; one example is the translation of the six ‘Core Vocabularies’ into 23 official EU languages
[80](#footnote81)
;

otools have been developed by ISA2 with the specific purpose of being ready for use in all official EU languages; one example is ‘EUSurvey’; and

oCPSV-AP is used, for instance, in Finland and Estonia to create cross-border federated catalogues of public services.

One potential improvement would be to further increase the programme’s focus on multilingualism by providing more information about ISA2 in different EU official languages, as the current website is only available in English.

JC4: Contribution to the advancement of common EU policies

Half of the respondents (44 out of 84) indicate that ISA2 has contributed to a high extent or to the fullest extent to the advancement of common EU policies or initiatives (
[Figure 20](#_Ref9784065)
). For instance, stakeholders noted that ISA2 plays a central role in the implementation of the EIF by abiding by its principles and tracking its implementation. The review of secondary data provides additional evidence supporting this. The Communication on the revised EIF
[81](#footnote82)
 is accompanied by an ‘Interoperability Action Plan, which sets out five focus areas and 22 actions to be undertaken in order to tackle existing interoperability issues during the 2017-2020 period. The Communication is also accompanied by 47 specific recommendations for improving interoperability in the EU. ISA2 actions can be clearly linked to the focus areas, actions, and recommendations outlined in the revised EIF (see 
[Annex 6](#_Ref9581764)
: 
[Table 23](#_Ref9597000)
).

Figure 20: Extent to which ISA2 contributed to the advancement of common EU policies (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

By contributing to the implementation of the revised EIF, ISA2 is also contributing to the establishment of the Digital Single Market, as explained in section 5.4 on external coherence. The technical experts involved in the evaluation note that the programme has increased attention paid to the importance of interoperability and cross-border e-services for sustaining the Digital Single Market. In this way, the programme fully meets the aims of advancing common shared policies within the EU, as stated in the Tallinn Declaration.

Answer to the ‘EU added value’ EQ

EQ8: What is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels?

ISA2 ensures EU-level coordination, which is necessary for improving overall interoperability among European public administrations. Without the programme, national and/or sub-national interventions would not be able to achieve the interoperability-specific objectives that ISA2 is aiming at. In addition, ISA2 is able to achieve these objectives at a lower cost than comparable national or sub-national initiatives. These conclusions hold true across the different stakeholder groups and are further corroborated by expert assessment.

Nonetheless, national or sub-national interventions would be able to contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels even if there was no EU-level programme. This is the specific objective where the ISA2 programme appears to be less effective, thus showing how well EU and national initiatives complement each other. National initiatives on the ICT modernisation of the public sector can even improve the performance of ISA2 by increasing capacity for the take-up of ISA2 solutions.

ISA2 also creates additional value by improving cross-border interoperability in the EU, as well as advancing common EU policies. As regards the former, ISA2 raises awareness of interoperability across EU Member States and helps put the topic on national agendas. It also brings people together, thus creating networks, helping national organisations meet their counterparts in different countries and facilitating exchanges between Member States in the field of interoperability. As regards the latter, ISA2 plays a central role in implementing the EIF and supports the establishment of the Digital Single Market.

Utility

Evaluation framework for ‘utility’

Article 13(4) of the ISA2 Decision requires an assessment of the programme’s utility. Utility refers to (i) the extent to which the results of ISA2 meet stakeholders’ needs; and (ii) the level of stakeholder satisfaction with ISA2 solutions. This criterion translates into the following EQ:

EQ9:How do the ISA2 programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address?

The assessment of this EQ is based on primary data collected via consultation activities as well as on evidence gathered via desk research, and relies on the following judgment criteria:

JC1:Alignment between stakeholders’ perception of needs and problems at EU, national and sub-national levels and the results of the programme; and

JC2:User satisfaction, with a breakdown by stakeholder group.

Analysis

JC1: Alignment between EU- and Member State-level needs and problems and the programme’s results

Respondents from both EU and Member State public administrations confirm that the solutions developed or maintained by ISA2 have contributed to addressing the original needs and problems identified in the field of interoperability (
[Figure 21](#_Ref9793703)
 and 
[Figure 22](#_Ref9793789)
)
[82](#footnote83)
. Overall, 42 % of respondents at the EU level (38 out of 91) and 46 % of respondents at the Member State level (13 out of 28) consider that the solutions have contributed to addressing the needs and problems originally covered by the programme to a high extent or to the fullest extent. In comparison, only 5 % of EU-level respondents (5 out of 91) and 11 % of Member State-level respondents (3 out of 28) see a limited or no contribution.

Figure 21: Extent to which ISA2 solutions contributed to addressing the needs and problems originally covered by the programme: EU-level respondents (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Figure 22: Extent to which ISA2 solutions contributed to addressing the needs and problem originally covered by the programme: respondents from the Member States (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

As the programme is still ongoing, the ways in which solutions meet user needs may still improve. It is expected that the take-up rate of solutions will increase over time, especially for the programme’s newer solutions, a conclusion that was explained in section 5.2 on effectiveness. With a higher adoption rate of ISA2 solutions across European public administrations, the needs and problems would be increasingly better addressed, thus leading to higher user satisfaction.

The feedback from consultation activities is further supported by secondary data. When it comes to the need for cooperation among public administrations with the aim to enable more efficient and secure public services, one example of the contribution of ISA2 is the “Access to Base Registries” action. The action has developed and is constantly updating a “Catalogue of Solutions”
[83](#footnote84)
 for base registries, facilitating interconnection and cooperation between public administrations. In addition, a set of guidelines for addressing potential challenges has been created.
[84](#footnote85)
 Both outputs help implement the once-only principle thus increasing the efficiency and user-friendliness of public services. Another example of an ISA2 action fostering cooperation in order to enhance efficiency in the public sector is the “Communication and information resource centre for administrations, businesses and citizens (CIRCABC)” action and tool
[85](#footnote86)
. CIRCABC enables the efficient collaboration between EU institutions, European public administrations and other stakeholders, by offering a free-of-charge, trustworthy tool. Feedback is constantly collected from users in order to improve the way the tool meets their needs.

As regards the need for exchanging information among public administrations to fulfil legal requirements or political commitments, the “Open e-TrustEx” solution, for instance, provides a platform for the secure exchange of information between public administrations in the EU and in the Member States. It is used in seven pan-European projects, including the e-Prior Open Source e-Procurement platform
[86](#footnote87)
 and the European eJustice Portal
[87](#footnote88)
, contributing to cross-border interoperability. Another example is the “Inter-institutional register of delegated acts (RegDel)”
[88](#footnote89)
, which directly responds to the commitment of the Commission, Council, and European Parliament to increase transparency and traceability of the delegated acts process.
[89](#footnote90)
 

Turning to the need for sharing and reusing information among public administrations to increase administrative efficiency and cut red tape for citizens and businesses, the “Joinup” platform can be mentioned. It provides a collaborative space where solutions can be shared between stakeholders. Additionally, the platform also hosts a number of national repositories where five Member States have made national solutions available to other interested parties. An additional example in this regard is the “Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification” action, which contributes to the sharing of best practices, by providing a set of reusable tools for data visualisation and data manipulation, available on the EU Open Data Portal.
[90](#footnote91)

As noted in section 5.1 on relevance, several respondents identified additional needs and problems to those explicitly mentioned in the programme’s intervention logic. In this respect, 27 % of EU-level respondents (11 out of 41) and 50 % of Member State-level respondents (6 out of 12) indicated that solutions have contributed to addressing these additional needs and problems to a high extent or to the fullest extent. In comparison, 22 % of EU-level respondents (9 out of 41) and 25 % of Member State-level respondents (3 out of 12) believe that this contribution has been either limited or absent (
[Figure 23](#_Ref9795041)
 and 
[Figure 24](#_Ref9795045)
).

Figure 23: Extent to which ISA2 solutions contributed to addressing additional needs and problems identified by consulted stakeholders: EU-level respondents (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Figure 24: Extent to which ISA2 solutions contributed to addressing additional needs and problems identified by consulted stakeholders: respondents from the Member States (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

JC2: Stakeholder satisfaction

The feedback received from respondents regarding user satisfaction tends to be positive (
[Figure 25](#_Ref9795454)
). More specifically, half of the respondents (57 out of 110) indicate that they are satisfied with the way solutions meet user needs to a high extent or to the fullest extent, while only 6 % (7 out of 110) indicate limited satisfaction.

Figure 25: Extent to which ISA2 solutions are meeting user needs in the fields of cross-border and cross-sectoral interoperability of digital public services (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Respondents also provided feedback on measures that could be taken to improve the programme’s utility. This feedback centres on the following aspects and shows significant complementarities with the measures proposed to improve the relevance of ISA2 (see section 5.1):

Best practices and user support. An increased focus on sharing best practices and providing support to users would have a beneficial effect on user satisfaction.

More focus on already existing solutions. According to the feedback from the consultation activities, another way to better address user needs would be to focus more on the already existing solutions and on improving their quality. This would improve the way in which user needs are considered in the further development of a smaller group of actions.

Promotion. Respondents indicated that they would like to see more activities promoting ISA2 at the Member State level in order to increase the level of take-up of solutions. An increased level of take-up would contribute to addressing the needs identified in the field of interoperability, thus improving overall satisfaction levels. Furthermore, feedback received from standardisation organisations shows that more awareness is needed among practitioners as well, particularly in the field of standards. Organising workshops targeted to specific groups of professionals could be useful. Formal presentations made at the plenary sessions of standards development organisations could also provide more exposure.

Co-creation. Another measure could be to involve users not only in the testing phase of solutions, but also in the design phase, and to establish a co-creation process. This includes more extensive on-boarding services (ensuring that users are provided with extensive information about the developed solutions), an increased focus on the sustainability of (re)user communities, as well as the integration of different solutions and services into one service, thus making it easier for users to navigate the landscape of available solutions.

·Commitment. The ISA2 Committee could reinforce the results of the programme by using it to set clear commitments to interoperability in the future. Member States could show their commitment to using the solutions stemming from ISA2 actions beyond simply participating in the selection of actions for the rolling work programme.

The ISA2 programme’s communication team and action owners already make an effort to capture users’ feedback and information on additional needs related to existing solutions. Their outreach activities made it possible to put together a number of user testimonials
[91](#footnote92)
, which support the promotion of solutions. For example, the user story presented in 
[Box 8](#_Ref11685349)
 encourages the use of Joinup. In the underlying detailed questionnaire, the interviewee shared her ideas about possible improvements, including more active marketing of the solution and the need to keep information on the Joinup platform up-to-date.

Box 8: ‘Joinup’ user testimonial

![](./../../../resource.html?uri=comnat:SWD_2019_1615_FIN.ENG.xhtml.SWD_2019_1615_FIN_ENG_24006.jpg)

Source: ISA2 Communication Team

Answer to the ‘utility’ EQ

EQ9: How do the ISA² programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address?

According to consulted stakeholders, solutions developed or maintained by ISA2 have contributed to addressing the original needs and problems identified in the field of interoperability. As the programme is still ongoing, the way solutions meet user needs may still improve. In particular, these needs and problems are expected to be better addressed due to more extensive adoption of ISA2 solutions across European public administrations.

In line with the conclusion drawn on the relevance criterion (section 5.1), whereas some of the new needs and problems experienced by consulted stakeholders are addressed by the programme, some others fall beyond its scope as it is currently defined. Therefore, the solutions developed so far cannot respond to these particular needs and problems.

The feedback received from respondents in terms of user satisfaction tends to be positive. Nevertheless, both primary and secondary data show that additional measures could increase the programme’s utility. For example, ISA2 could place more emphasis on providing support to users of its existing solutions. Moreover, it could already involve users in the design phase of future solutions, thus increasing quality and user satisfaction.

Sustainability

Evaluation framework for ‘sustainability’

Sustainability is the final evaluation criterion that has to be assessed as part of the interim evaluation according to Article 13 of the ISA2 Decision. Sustainability measures the likelihood that the results of the ISA² programme last beyond its completion. The below EQ addresses this criterion:

EQ10:To what extent is the financial, technical and operational sustainability of the developed solutions — maintained and operated through the ISA² programme — ensured?

The evidence base for assessing the sustainability of ISA2 consists of primary data collected via the consultation activities, complemented by expert assessment. The evaluation relies on the following judgment criteria:

JC1:Extent to which the results achieved by the ISA2 programme are expected to last if funding for the actions covered by the programme would not be available in the future; and

JC2:Extent to which ‘cost recovery’ solutions could be introduced.

Analysis

JC1: Results would last if funding for actions would not be available in the future

This part of the assessment relies on the hypothetical scenario in which, after the completion of ISA2, no other similar programme is implemented at the EU level. Overall, the consulted stakeholders tend to have a positive outlook on the sustainability of results achieved so far (
[Figure 26](#_Ref9797930)
), with 79 % of respondents (66 out of 84) believing that the programme results probably or definitely would last in the absence of future funding for the actions currently supported by ISA2.

Figure 26: Likelihood that results achieved so far would last if funding for actions covered by the programme would not be available in the future (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Although the global results of the programme are expected to last (as the progress made in the field of interoperability will not be undone), certain issues may arise when considering the specificities of different solutions and future steps towards interoperability. In fact, 65 % of respondents (55 out of 85) consider that operations and maintenance costs
[92](#footnote93)
 required to run the existing solutions would have a highly or fully negative effect on the ability of solutions to deliver their results once the programme ends. In addition, 56 % (48 out of 85) hold the same opinion about the technical and operational support
[93](#footnote94)
 needed for the solutions (
[Figure 27](#_Ref9798096)
).

Beyond these two factors, respondents indicated other aspects that would have a negative effect on the solutions once ISA2 came to an end and no other programme replaced it:

Lack of further development for existing solutions. Given the rapid pace of changes in the field of ICT, solutions that do not benefit from continued development support and updating face the risk of becoming obsolete over time. This factor emerged from the consultation activities and was mentioned by the technical experts as well
[94](#footnote95)
.

·Lack of coordination. In the absence of a single programme overseeing the development of solutions, future changes to solutions might lead to diverging results between public administrations or bilateral solutions between countries and a large variety of outcomes. This would be contrary to the goal of achieving a holistic landscape of interoperability in the EU.

·Dissemination and communication. ISA2 also plays an important role in promoting interoperability and creating a space for discussions on the topic. Without such a programme, the interest in a common approach to interoperability would be jeopardised.

Figure 27: Extent to which financial, technical and operational aspects would prevent ISA2 solutions from delivering their results if the programme was terminated and not replaced (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

The elements presented above explain why 87 % of respondents (75 out of 86) believe that the achievement of the ISA2 general objective would — either probably or definitely — be jeopardised should ISA2 be terminated without being replaced (
[Figure 28](#_Ref9801658)
). This confirms that the ISA2 programme plays a central role in improving the interoperability landscape in the EU by promoting the ICT-based modernisation of the public sector in Europe and making it possible to address the needs of businesses and citizens via improved interoperability of European public administrations.

Figure 28: Likelihood that the ISA2 general objective would be jeopardised if the programme was terminated and not replaced (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

JC2: Cost recovery

Given the adverse effects of operational and maintenance costs as well technical and operational support, stakeholders were asked to assess the feasibility of introducing a fee for users in order to gain access to ISA2 solutions, in case funding would no longer be provided for the actions currently carried out by ISA2. Over half of the respondents (47 out of 79) would probably or definitely not pay such a fee, in comparison to 40 % of respondents (32 out of 79) who indicated that they would probably or definitely pay a fee in order to use the solutions (
[Figure 29](#_Ref9800611)
). The main reason underlying these responses is that users are accustomed to having access to ISA2 solutions free of charge. Changing this to a pay-for-access system may lead users to search for other solutions that are open source and free of charge, depending on the level of the fee requested. Nevertheless, for some of the more mature solutions, a fee could be acceptable if the level of take-up is already quite high, making the solution indispensable to some extent.

Figure 29: Likelihood that users would agree to pay a fee to continue accessing ISA2 solutions (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Answer to the ‘coherence’ EQs

EQ10: To what extent is the financial, technical and operational sustainability of the developed solutions - maintained and operated through the ISA² programme - ensured?

Overall, consulted stakeholders tend to have a positive view of the sustainability of results achieved so far. However, while the programme’s global results are expected to last (as the progress made in the field of interoperability will not be lost), certain issues may arise when considering the specificities of different solutions and future steps towards improved interoperability.

In fact, there are solutions that require both operational and maintenance costs as well as technical and operational support, which would render the solutions unable to deliver further results if the programme was terminated. Additional obstacles to the programme’s sustainability include the lack of further development for existing solutions; the lack of coordination between national administrations; and the limited dissemination of information and communication about the interoperability of digital public services. Consequently, ISA2 should investigate possibilities to increase the sustainability of the results of its existing solutions. It could assess the feasibility and costs of various sustainability measures and invest in them. The programme could for example consider transferring some of its solutions to open source communities or encouraging companies to build services around free ISA2 solutions under the European Union Public Licence (EUPL)
[95](#footnote96)
.

To conclude, the findings suggest, that the ICT-based modernisation of the public sector in Europe and the possibility to address the needs of businesses and citizens via improved interoperability of European public administrations — the general objective of ISA2 — would be jeopardised in the absence of ISA2 or any similar EU initiative. The consulted stakeholders’ reluctance to pay for ISA2 solutions makes this conclusion even more probable.

6.Conclusions

Context

ISA2 is a European funding programme offering interoperability solutions and common frameworks for European public administrations, businesses and citizens. It is the fifth in a series of European instruments fostering interoperability in the European public sector since 1995. It follows the ISA programme, which terminated in 2015, and covers the 2016-2020 period.

ISA2 was established by Decision (EU) 2015/2240 of the European Parliament and of the Council (‘ISA2 Decision’), which legally requires the European Commission to carry out an interim evaluation of the programme by 30 September 2019, and to communicate the results.

Method

Work on the evaluation started in Q2 2018 with the involvement of interested Commission services, who discussed and outlined the evaluation design. External experts supported the evaluation process as well by finalising the evaluation design, collecting, validating and analysing data, and summarising their findings in an independent evaluation study
[96](#footnote97)
.

The agreed evaluation framework (
[Annex 4](#_Ref9341586)
) covered all seven evaluation criteria identified by the ISA2 Decision, namely: relevance, effectiveness, efficiency, coherence, EU added value, utility and sustainability. It also described the evaluation questions and related judgment criteria for each evaluation criterion, together with the data sources and data collection methods. For data collection, the evaluation relied on both primary data gathered through stakeholder consultation activities and secondary data collected via desk research between November 2018 and March 2019 (Chapter 4).

Although the evaluation was able to draw robust conclusions for all evaluation criteria based on the collected data, there were two limitations:

1) Timing. As this is an interim evaluation, the analysis covered only the first three years of the programme (2016, 2017 and 2018). The ongoing ISA2 actions are expected to produce further results in the coming years, which will influence the programme’s overall achievement of objectives.

2) Reach. ISA2 is a very technical programme, and mainly involves interoperability experts. This explains the rather low participation in the stakeholder consultation activities. Another reason stems from the stringent confidentiality and data protection rules: without their specific consent, the evaluation team could approach ISA2 solution users only through the action owners acting as intermediaries.

Findings

Based on data collected from 129 consulted stakeholders, extensive desk research, and expert assessments, the interim evaluation confirms that ISA2 performs well in all the evaluation criteria. Nonetheless, some additional measures could be taken to improve the performance of ISA2 and any future EU programme dedicated to interoperability; these are presented below.

Relevance. The original needs and problems that the programme intended to address are still fully relevant and the objectives of ISA2 address them. Some stakeholders, however, experience a number of additional needs (like a more prescriptive approach to designing interoperable public services) and problems (including resource constraints), which ISA2 can address only partially. One additional need that deserves special attention is the request to ensure more collaboration and exchanges with regional and local administrations in order to increase awareness of interoperability and the take-up of ISA2 solutions at the sub-national level.

Effectiveness. The results that ISA2 has achieved so far are aligned with its objectives. The 20 sampled actions that were evaluated have so far generated 35 solutions, some of which are used by all 28 EU Member States as well as by EU institutions. Nevertheless, the programme has not realised all the expected results yet, because most of its actions are still ongoing and solutions are still being developed. The programme’s duration also influences the take-up rate of solutions. Actions continued from previous editions of the programme have produced solutions that are now more widely used than solutions resulting from actions started under ISA2.

There is one specific ISA2 objective where the evaluation found that the programme on its own is less effective: the development of a more effective, simplified and user-friendly e-administration at the national, regional and local level. Here Member States can in fact have a great impact, and complement the EU-level initiatives on interoperability and digitalisation.

As regards external factors, they have a mixed impact on ISA2. They can improve but also jeopardise the way in which the programme achieves its objectives and delivers its results.

When it comes to communication and awareness-raising activities, the evaluation results suggest that ISA2 duly acted upon the recommendation of the final ISA evaluation and increased the general awareness of the programme through its dedicated communication efforts. Still, some aspects of ISA2 could benefit from more promotion.

Efficiency. The programme’s implementation is progressing as planned, all actions being either on track or close to achieving the planned level of work. Nevertheless, the heterogeneity of performance indicators makes it difficult to draw conclusions about its overall cost-effectiveness. For those ISA2 packages where it was possible to aggregate performance indicators for different actions, costs per end-user (e.g. businesses, citizens) have been estimated as very low.

The selection process for actions funded by ISA2 is considered relatively efficient. Nonetheless, it could be further improved by simplifying the rolling work programme (e.g. making the template more flexible) and launching thematic calls for actions, thus ensuring that the selection process is driven by objectives.

Coherence. Substantial synergies and limited overlaps among ISA2 actions paint a positive picture of the programme’s internal coherence. This strong internal coherence is in line with the second recommendation of the final ISA evaluation, which called for taking a holistic approach to interoperability within ISA2.

ISA2 also followed the final third recommendation of the final ISA evaluation, on external coherence, and worked closely with other EU policies and initiatives. It established synergies with the CEF, Horizon 2020 and the Structural Reform Support Programme. ISA2 also interacts with initiatives such as the Single Digital Gateway, Digital Single Market Strategy, eGovernment Action Plan, Tallinn Declaration on eGovernment, and the Rolling Plan for ICT standardisation. Nonetheless, some overlaps were identified between specific solutions under ISA2 and the CEF; these could be addressed by better delineating the scope of each solution.

EU added value. ISA2 ensures EU-level coordination, which is necessary for improving overall interoperability among European public administrations. Without the programme, national and/or sub-national interventions would not be able to achieve the interoperability-specific objectives that ISA2 aims to achieve. In addition, ISA2 is able to achieve these objectives at a lower cost than comparable national or sub-national initiatives.

ISA2 also creates additional value by improving cross-border interoperability in the EU, as well as advancing common EU policies: it plays a central role in the implementation of the EIF and supports the establishment of the Digital Single Market.

Utility. User satisfaction with ISA2 solutions tends to be positive, with only 7 out of 110 stakeholders reporting ‘limited satisfaction’. As the programme is ongoing, it is expected that the take-up of ISA2 solutions will increase, thus improving the way solutions meet user needs and, in turn, overall user satisfaction. However, ISA2 could place more emphasis on providing support to the users of its existing solutions. Moreover, it could involve users already in the design phase of future solutions, which would increase both quality and user satisfaction.

Sustainability. The consulted stakeholders have mixed views on the sustainability of ISA2 solutions. While most of them believe that the programme results would last in the absence of future funding, they also recognise that ISA2 solutions may not be able to deliver their results if the related maintenance costs and support needs were not covered.

Stakeholders agree, however, that ISA2 plays a central role in improving the interoperability landscape in the EU and that its absence would jeopardise the efforts of European public administrations to improve interoperability and foster the ICT-based modernisation of the public sector in Europe.

Next steps

The interim evaluation identified several potential measures that may improve the performance of ISA2 and its successor instruments (see 
[Box 9](#_Ref9868697)
). It is up to the programme’s management to assess the feasibility of these actions and put forward specific recommendations, which the Commission could then put into practice in the future.

Box 9: Overview of identified measures that would improve the programme’s performance

|  |
| --- |
| ·increase awareness of interoperability beyond national administrations by fostering more collaboration and exchanges with regional and local administrations;  ·perform studies to assess the costs and benefits of implementing ISA2 solutions and measure their impact on the efficiency and productivity of public administrations;  ·improve the sharing of best practices among public administrations;  ·rely more on potential influencers when communicating about interoperability;  ·build advisory capabilities around interoperability;  ·assess the rationale and impacts of a possible binding interoperability instrument;  ·continue and even increase communication on and promotion of interoperability by  targeting the sub-national layers of administrations too;  ·simplify the ISA2 rolling work programme;  ·launch thematic calls for actions to ensure that the selection process is driven by objectives;  ·define some common metrics to allow for a more consistent appraisal of cost-effectiveness;  ·clarify the scope of each solution to avoid any overlaps;  ·build synergies and avoid overlaps with the Tallinn Declaration on eGovernment;  ·assess any potential obstacles to ISA2 technical specifications that may arise with regard to the intellectual property rights policy for CEN/CENELEC standards;  ·increase the programme’s focus on multilingualism by providing more information about ISA2 in different official EU languages;  ·provide more support to the users of existing solutions;  ·involve users already in the design phase of future solutions;  ·assess the feasibility and costs of various sustainability measures and invest in them; and  ·transfer some ISA2 solutions to open source communities or encourage companies to build services around free ISA2 solutions under the European Union Public Licence. |

Annex 1: Procedural information

1.Lead DG, Decide Planning/CWP references

DG Informatics (DIGIT), PLAN/2018/2654.

2.Organisation and timing

The Commission announced this interim evaluation in an 
[evaluation and fitness check roadmap](https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-2768206_en)

[97](#footnote98)
 on 29 May 2018.

Following the invitation by the Director-General of DIGIT in May 2018, the first Inter-service Steering Group (ISSG) meeting took place on 8 June 2018, with participants from DG CNECT, DIGIT, EMPL, FISMA, GROW, OP, SG, TAXUD and the JRC. The ISSG met five times between June 2018 and June 2019, as follows:

|  |  |
| --- | --- |
| Period | Activity |
| June 2018 – October 2018 | Involvement of external consultants |
| November 2018 – May 2019 :  oNovember 2018 – March 2019  oFebruary 2019 – May 2019 | Running the evaluation process:  oData collection  oData analysis and synthesis |
| June 2019 | Preparation of the Commission report for inter-service consultation and adoption |

3.Exceptions to the better regulation guidelines

Not applicable

4.Consultation of the RSB (if applicable)

Not applicable

5.Evidence, sources and quality

Evidence & sources

Data have been drawn from two main sources:

  

1) Primary data were collected through the below consultation activities:

|  |  |
| --- | --- |
| Consultation activity | Timeframe |
| ISA2 Mid-Term Conference short questionnaire: A short questionnaire based on the public consultation questionnaire was distributed during the ISA2 Mid-Term Conference and during the Kick-off workshop. | 29 - 30 November 2018 |
| Kick-off workshop: The workshop was to present the interim evaluation and the planned consultation activities to the different stakeholder groups of the ISA2 programme. | 30 November 2018 |
| Public consultation: This activity gave all relevant parties, primarily general public, businesses, stakeholders’ associations and public administrations, the opportunity to express their opinions and views on the achievements of the ISA2 programme. The public consultation was available in 23 EU languages. | 7 December 2018 – 1 March 2019 (12 weeks) |
| Targeted online surveys: Five online surveys were made available on EUSurvey, targeting the following stakeholder groups: programme governance, action owners, solution users, stakeholders responsible for linked EU policies/initiatives, and standardisation organisations. | 14 December 2018 – 23 January 2019 (the surveys remained open after the initial deadline of 18 January 2018 to accommodate some last-minute requests from stakeholders) |
| Targeted in-depth interviews: Interviews were conducted with stakeholders from the five groups targeted by the online surveys: programme governance, action owners, solution users, stakeholders responsible for linked EU policies/initiatives, and a standardisation expert. The interviews, based on questionnaires very similar to those used for the online surveys, facilitated the collection of additional qualitative data to complement data collected via such surveys. | 12 December 2018 – 23 January 2019 (several interviews were conducted after the initial deadline set on 18 January 2018 as some stakeholders were not available to be interviewed at an earlier time) |

2)Secondary data were gathered by reviewing:

|  |
| --- |
|  |
| The annual rolling work programmes of ISA2 |
| The ISA2 Dashboard |
| The Europa webpages on ISA2 actions and solutions |
| Monitoring and evaluation reports: 2016 (annual report), 2017 (quarterly reports and annual report), and 2018 (the reports for the first two quarters of 2018) |
| Overview of communication activities of events organised by ISA2 between 2016 and 2018 (including information on the location and the number of participants) and events in which ISA2 was represented (including information on the location of the events) |
| Lists of participants in ISA2 Committee meetings and in ISA2 Coordination Group meetings between 2016 and 2018 |
| Performance indicators directly shared by action owners of the sampled actions |

  

Quality

To support the ISA2 interim evaluation, in September 2018 the Commission tasked the Centre for European Policy Studies (CEPS) with delivering an independent evaluation study by May 2019. The ISSG members closely monitored the work of CEPS. They discussed the consultant’s various deliverables at the ISSG meetings and also commented on the documents in writing. The ISSG accepted CEPS final evaluation study by written procedure on 24 May 2019. This final ‘sign off’ by the ISSG also built on the quality assessment results, which concluded that the external contractor's study and work met the required quality standards.

Annex 2: Synopsis report of the consultation activities

Introduction

In line with the 
[consultation strategy](https://ec.europa.eu/isa2/sites/isa/files/consultation_strategy_isa2_interim_evaluation.pdf)

[98](#footnote99)
, consultation activities were carried out to collect stakeholders’ views and opinions, thus contributing to the evidence base needed to achieve the two main objectives of the interim evaluation of the ISA2 programme:

Evaluating the implementation of the programme by considering seven evaluation criteria:

oRelevance ‒ the alignment between the programme’s objectives and the current needs and problems experienced by stakeholders;

oEffectiveness ‒ the extent to which the programme has achieved its objectives;

oEfficiency ‒ the extent to which the programme’s objectives are achieved at a minimum cost;

oCoherence ‒ the alignment between the programme and comparable EU initiatives as well as the overall EU policy framework;

oEU added value - the additional impacts generated by the programme, as opposed to leaving the subject matter in the hands of Member States;

oUtility ‒ the extent to which the programme meets stakeholders’ needs; and

oSustainability ‒ the likelihood that the programme’s results will last beyond its completion.

·Identifying areas of potential improvement and devising recommendations to improve the programme’s performance and bring more value to its stakeholders.

Feedback received on the evaluation roadmap

Before the interim evaluation was launched, one stakeholder provided feedback on the ISA2 evaluation roadmap93, noting that the ‘initiative is good, well planned, and funded’, but also that ‘it could have benefitted from more advertising’
[99](#footnote100)
. This input was received in June 2018.

  

Types of consultation activities conducted

A mix of consultation activities was organised during the evaluation process to reach a variety of the programme's stakeholders. The following seven stakeholder groups were targeted:

Programme governance: Commission representatives from DIGIT’s Interoperability unit and members of the ISA2 Committee and Coordination Group;

Action owners: Commission representatives in charge of specific actions defined under ISA2;

Solution users: Commission and Member State representatives who are using/reusing ISA2 solutions;

Stakeholders responsible for linked EU policies/initiatives: Commission representatives who are in charge of EU initiatives linked to ISA2, and who are thus in a position to provide feedback, on the external coherence of the programme among other things;

·Experts: experts in IT and eGovernment able to perform an expert assessment of the programme and of the issues and developments in the field of interoperability;

Standardisation organisations: Representatives of standardisation organisations who can provide feedback on ISA2 activities related to standards and technical specifications; and

·Indirect beneficiaries and the wider public: Other programme beneficiaries or parties who are involved in or affected by the ISA2 programme in any way, including the private sector and the general public.

Between 29 November 2018 and 1 March 2019, five consultation activities were carried out in accordance with the Commission’s better regulation requirements:

Targeted online surveys (14.12.2018 – 23.1.2019)

Five online surveys were made available on EUSurvey, targeting the following stakeholder groups: programme governance, action owners, solution users, stakeholders responsible for linked EU policies/initiatives, and standardisation organisations.

The online surveys were initially set to close on 18 January 2019, but the deadline was extended to 23 January in order to accommodate some last-minute participants. Altogether 80 responses were collected through these surveys.

Targeted in-depth interviews (12.12.2018 – 23.1.2019)

Interviews were held with the same five stakeholder groups as the online surveys: programme governance, action owners, solution users, stakeholders responsible for linked EU policies/initiatives, and a standardisation expert. The interviews were based on questionnaires that were very similar to those used for the online surveys. They helped collect additional qualitative data from 20 stakeholders complementing the data collected from the surveys.

The in-depth interviews were initially supposed to run until 18 January 2019. However, as some stakeholders were not available in the original time-frame interviews continued until 23 January 2019.

Public consultation (7.12.2018 – 1.3.2019)

This activity gave all relevant parties — mainly the public, businesses, stakeholders' associations and public administrations, the opportunity to express their views on the ISA2 programme's achievements. Experts could participate in the public consultation and state their level of expertise in the programme and in the interoperability of digital public services. The public consultation
[100](#footnote101)
 was available on EUSurvey in 23 EU languages and yielded 14 replies. This low response rate is due to two main reasons. First, ISA2 is a technical programme producing solutions addressed mainly to European public administrations; therefore, the direct contact between the average citizen/business and the programme is limited. Second, stakeholder groups that are in direct contact with the programme (e.g. solution users, action owners, etc.) were consulted via targeted online surveys and interviews.

ISA2 Mid-Term Conference short questionnaire (29 – 30.11.2018)

A short questionnaire based on the public consultation questionnaire was distributed during the ISA2 Mid-Term Conference (29 November 2018)
[101](#footnote102)
 and during the Kick-off workshop (30 November 2018) of the interim evaluation. 15 stakeholders shared their views on the programme through this channel.

Kick-off workshop

The purpose of the kick-off workshop of 30 November 2018 was to present the interim evaluation and the planned consultation activities to the ISA2 programme’s stakeholder groups
[102](#footnote103)
. There were 40 participants in the workshop.

Methodology and tools used to process data

The consultation questionnaires were mainly based on ‘Likert scale’ questions, meaning that respondents were asked to give their opinion on a scale from (1) to (5) or from (1) to (4), depending on the type of question:

1 - not at all; 2 - to a limited extent; 3 - to some extent; 4 - to a high extent; 5 - to the fullest extent; or

·1 – definitely would not; 2 – probably would not; 3 – probably would; 4 – definitely would.

This approach helped the comparison of answers between respondents. In addition to the Likert scale questions, a number of open-ended questions were included to gather more in-depth information from consulted stakeholders. In addition, the interviews allowed more detailed feedback to be collected.

Characteristics of respondents

With a total of 129 respondents, the various consultations reached all of the following groups: programme governance, action owners, solution users, stakeholders responsible for linked EU policies / initiatives, standardisation organisations and the wider public (
[Table 5](#_Ref9324386)
). In addition, a team of technical experts assessed the programme and their conclusions are presented throughout the evaluation report.

Note that the results of the public and targeted consultation activities have been combined and analysed jointly in this report.

Table 5: Number of consulted stakeholders by consultation activity and stakeholder category

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Stakeholder  category | In-depth interview | Online survey | Public consultation | Short question-naire | TOTAL |
| Programme Governance | 4 | 15 | - | - | 19 |
| Action owners | 3 | 20 | - | - | 23 |
| Solution users | 6 | 37 | - | - | 43 |
| Stakeholders related to linked EU policies/ programmes/ initiatives | 6 | 4 | - | - | 10 |
| Standardisation organisations | 1 | 4 | - | - | 5 |
| Wider public | - | - | 14 | 15 | 1 |
| TOTAL | 20 | 80 | 14 | 15 | 129 |

Note: Regarding the ‘Expert’ stakeholder group, four technical experts independently assessed the programme. In addition, most of the respondents from the consulted stakeholder groups are experts in interoperability and ISA2, as shown in 
[Figure 30](#_Ref9325154)
 and 
[Figure 31](#_Ref9325186)
.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

  

The consulted stakeholders claimed to have high levels of expertise both on the interoperability of public services (
[Figure 30](#_Ref9325154)
) and on the ISA2 programme as a whole (
[Figure 31](#_Ref9325186)
).

Figure 30 Knowledge of digital public services and interoperability (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Figure 31 Knowledge of ISA2 (average score of answers, number of respondents)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

One slight exception is solution users who reported a relatively more limited knowledge of the entire programme compared to the other stakeholder groups consulted. Even so, most respondents are familiar with at least 5 packages of the 9 ISA2 packages of actions (
[Figure 32](#_Ref9325258)
).

Figure 32 Knowledge of ISA2 action packages (number of respondents familiar with a given number of packages)
[103](#footnote104)
 

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Regarding geographical distribution, solution users from various Member States provided input in the in-depth interviews and the online surveys:

At EU level, solution users who participated in the consultations work in EU institutions located in a number of EU countries: Belgium, Czechia, France, Hungary, Italy, Luxembourg, Poland, Portugal, and Sweden;

·At national and regional levels, solution users (including respondents who answered as individuals) who provided feedback come from the following 13 countries: Belgium, Bulgaria, Croatia, Czechia, France, Germany, Greece, Hungary, Italy, Luxembourg, the Netherlands, Portugal, and Sweden.

Feedback on the public consultation was provided by individuals in six Member States (Belgium, France, Germany, Greece, Spain, and the United Kingdom) and by public administrations from three Member States (Czechia, Greece, and Spain).

  

Results

The consultation activities revealed a general consensus among the different stakeholder groups consulted and confirmed that ISA2 performs well in all the seven evaluation criteria under investigation. The feedback from the consultation activities is summarised as follows:

Relevance

The respondents to the consultation activities emphasised that ISA2 is fully relevant with respect to the evolving needs and problems in the field of interoperability.

The majority of respondents indicate that: i) the needs and problems originally addressed by the programme are still experienced by European public administrations, businesses and/or the wider public; and ii) by achieving its general objective, ISA2 can address the needs and problems identified when the programme was established.

Respondents pointed out that there are other needs and problems currently experienced by stakeholders in the field, namely:

the need for a more binding legal framework for interoperability and a more prescriptive approach to designing interoperable public services;

the need to improve the way administrations communicate with one another;

the need to share best practices;

the resource constraints experienced by national and local public administrations;

the different political priorities of Member States that prevent a consistent approach to interoperability in the EU; and

·the limited awareness of ISA2 and other interoperability-related initiatives, especially at regional and local level.

In general, respondents to the consultation activities believe that the programme, through its general objective, the programme could address these new needs and problems to some extent, as some of the issues mentioned fall outside the scope of the programme.

Effectiveness

The results achieved so far by ISA2 are aligned with the programme’s objectives according to the consulted stakeholders, in particular when it comes to:

developing, maintaining and promoting a holistic approach to interoperability in the EU; and

·facilitating the reuse of interoperability solutions by European public administrations.

However, the results to date still fall short of the expected results, as most of the actions are ongoing and solutions are still being developed. For instance, respondents emphasised that relatively more could be done to achieve the expected result of improved efficiency and productivity of the European public administrations.

External factors could improve but also jeopardise the way in which the programme achieves its objectives and delivers its results. The vast majority of respondents confirmed that the calls for common standards and frameworks from public administrations are important external factors contributing to the performance of the programme. In contrast, institutional complexity in the form of multiple layers of governance and differences between national public administrations from various Member States and local public administrations within Member States could lead to coordination issues that hamper cross-border and cross-sectoral interoperability.

Efficiency

Respondents from the programme governance and action owners gave feedback on the efficiency of the process for selecting actions funded by ISA2. The process is considered relatively efficient and fit-for-purpose by the respondents from these two stakeholder groups. However, some measures could be taken to further streamline the selection process, such as:

simplifying the rolling work programme; and

·simplifying the overall process by launching thematic calls for actions.

Respondents from the same two groups also provided information about the amount of time they spent on preparing and submitting a proposal for an action to be considered by the ISA2 programme. On average, six person-days are required to renew a proposal for actions already included in previous rolling work programme. In contrast, between 10 and 30 person-days are required to prepare and submit a proposal for a new action.

Coherence

The consultation activities focused on both the internal coherence of the programme, i.e. the synergies or overlaps existing between the ISA2 actions, and its external coherence, namely the synergies or overlaps between ISA2 and other EU initiatives, policies, or programmes.

Internal coherence

ISA2 actions are characterised by substantial synergies with each other and limited overlaps. During the consultation activities, respondents provided examples of synergies to support their assessment. Examples mentioned include:

solutions developed under the ‘SEMIC’ actions are reused by the ‘Catalogue of Services’ and the ‘Access to Base Registries’ actions;

the ‘Re3gistry’ solution, part of the ‘ELISE’ action, is used to share the codes for the ‘Core Vocabularies’, part of the ‘SEMIC’ action; and

·The solutions developed as part of the ‘e-TrustEx’ actions are reused by the action ‘European Public Procurement Interoperability Initiative’.

External coherence

On external coherence, respondents to the consultation activities identified most synergy between ISA2 and CEF. In contrast, of the EU programmes interacting with ISA2, the ESF tends to have the least synergies. Respondents indicated that Horizon 2020 (in particular the work related to the once-only principle) and the SRSP (through the support given to Member States to align and integrate with cross-border services) also have some synergies with ISA2.

Respondents also pointed at synergies and overlaps with other EU initiatives such as ERDF, Corporate IT Governance and the Single Digital Gateway.

EU added value

Most respondents believe that national or sub-national interventions could not achieve or could only slightly achieve the general and specific interoperability objectives that ISA2 is pursuing. The consulted stakeholders consider that ISA2 plays an important role in ensuring a level of coordination that improves interoperability among European public administrations.

Respondents also emphasised that ISA2 can achieve its objectives at a lower cost than comparable national or sub-national initiatives. Most also believe that ISA2 has helped improve cross-border interoperability in the EU. Examples frequently mentioned by stakeholders are the work done in the field of semantic interoperability and the fact that ISA2 raises awareness about interoperability in the EU, without which little progress would be made in the field.

When it comes to advancing common EU policies, consulted stakeholders noted that ISA2 plays a central role in implementing the EIF (including its 2017 revision) by abiding by its principles and tracking its implementation. Stakeholders emphasised that ISA2 has also contributed to specific areas of broader EU policy, namely the Digital Single Market.

Utility

Respondents in both EU-level and national public administrations confirm that the solutions developed or maintained by ISA2 have contributed to addressing the original needs and problems identified in the field of interoperability. Respondents emphasised that while some of the new needs and problems experienced in the field are addressed by the solutions provided by ISA2, others, such as changing political priorities in the Member States, fall beyond the scope of the programme as it is currently defined.

The feedback received from respondents in terms of user satisfaction tends to be positive. Nevertheless, a number of measures were identified to increase the utility of the programme, namely:

placing more emphasis on sharing best practices and providing support to users;

establishing a co-creation process with users;

improving the quality of existing solutions by better considering user needs;

ensuring the Member States’ commitment to using ISA2 solutions; and

·strengthening the promotion of ISA2 solutions at Member State level as well as among specific groups of professionals (e.g. standards development organisations).

Sustainability

Overall, consulted stakeholders have a positive view of the sustainability of results achieved so far. Nevertheless, while the programme’s global results are expected to last (as the progress made in the field of interoperability will not be lost), certain issues may arise when considering the specific aspects of different solutions and future steps towards increased interoperability. In fact, there are solutions requiring both operations and maintenance costs as well as technical and operational support, which would be unable to deliver further results if the programme terminated. Additional obstacles to the sustainability of the programme include:

lack of development for existing solutions;

lack of coordination between European public administrations; and

·limited dissemination and communication related to interoperability.

Furthermore, introducing a fee for users to gain access to ISA2 solutions is not considered a feasible solution, especially in the event that funding would no longer be provided for the actions currently carried out by ISA2. Finally, the vast majority of respondents, from all stakeholder groups, stressed that the ICT-based modernisation of the public sector in Europe and the possibility to address the needs of businesses and the public via improved interoperability of European public administrations would be jeopardised, if ISA2 were terminated without any similar EU programme being established in its place.

Figures presenting the results of the consultation activities per evaluation criterion

Relevance

[Extent to which needs and problems originally addressed by ISA](#_Ref14383468)
[2](#_Ref14383468)
[are currently experienced by European public administrations, businesses and/or citizens (average score of answers, number of respondents)](#_Ref14383468)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

  

Extent to which achieving ISA2 objectives contributes to addressing the needs and problems originally addressed by the programme (average score of answers, number of respondents)

![](./../../../resource.html?uri=comnat:SWD_2019_1615_FIN.ENG.xhtml.SWD_2019_1615_FIN_ENG_24007.jpg)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent.

Note: Averages do not account for respondents answering ‘don’t know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which achieving ISA2 objectives contributes to addressing additional (current) needs and problems identified by consulted stakeholders (number of respondents by stakeholder category)

![](./../../../resource.html?uri=comnat:SWD_2019_1615_FIN.ENG.xhtml.SWD_2019_1615_FIN_ENG_24008.jpg)

Score: (1) not at all; (2) to a limited extent; (3) to some extent; (4) to a high extent; or (5) to the fullest extent; (DK/NO) do not know/no opinion.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Effectiveness

Extent to which ISA2 solutions contributed to achieving the programme’s objectives (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)
 Extent to which ISA2 solutions contributed to achieving the programme’s operational objectives (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which ISA2 solutions achieved the programme's expected results (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which the following external factors contribute to the performance of ISA2 (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which the following external factors jeopardise the performance of ISA2 (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Efficiency

Extent to which the action selection process is fit-for-purpose (number of respondents by stakeholder category)

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Distribution of the number of days needed to prepare and submit a proposal for an ISA2 action (number of respondents)

Note: The histogram was constructed using a bin size of 2 days, which grouped the answers of respondents in intervals of 2 days. This bin size was deemed most appropriate after reviewing the answers provided.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Earned value, actual costs and planned value sampled actions, by package (Q3 2018)
[104](#footnote105)

![](./../../../resource.html?uri=comnat:SWD_2019_1615_FIN.ENG.xhtml.SWD_2019_1615_FIN_ENG_24009.jpg)

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Schedule performance indices of the sampled actions grouped by packages

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

  

Coherence

Extent to which there are synergies / overlaps between ISA2 actions (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which there are synergies / overlaps between ISA2 and other relevant EU programmes (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

EU added value

Extent to which national or sub-national interventions would be able to achieve the ISA2 objectives without the programme (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Ability of ISA2 to achieve the relevant objectives at a lower cost than comparable national or sub-national interventions (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which ISA2 contributed to improving cross-border interoperability in the EU (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which ISA2 contributed to the advancement of common EU policies (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

  

Utility

Extent to which ISA2 solutions contributed to addressing the needs and problems originally covered by the programme: EU-level respondents (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which ISA2 solutions contributed to addressing the needs and problem originally covered by the programme: respondents from the Member States (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which ISA2 solutions contributed to addressing additional needs and problems identified by consulted stakeholders: EU-level respondents (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which ISA2 solutions contributed to addressing additional needs and problems identified by consulted stakeholders: respondents from the Member States (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which ISA2 solutions are meeting user needs in the fields of cross-border and cross-sectoral interoperability of digital public services (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

  

Sustainability

Likelihood that results achieved so far would last if funding for actions covered by the programme would not be available in the future (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Extent to which financial, technical and operational aspects would prevent ISA2 solutions from delivering their results if the programme was terminated and not replaced (average score of answers, number of respondents)

Score: 1 — not at all; 2 — to a limited extent; 3 — to some extent; 4 — to a high extent; 5 — to the fullest extent.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Likelihood that the ISA2 general objective would be jeopardised if the programme was terminated and not replaced (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Likelihood that users would agree to pay a fee to continue accessing ISA2 solutions (average score of answers, number of respondents)

Score: 1 — definitely would not; 2 — probably would not; 3 — probably would; 4 — definitely would.

Note: Averages do not account for respondents answering ‘do not know/no opinion’ (DK/NO).

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Annex 3: Intervention Logic

This Annex presents the ISA2 programme’s intervention logic, which comprises the rationale, the inputs, and the expected results of the intervention. This step aims to clarify the logic followed by EU decision-makers when establishing the ISA2 programme. The intervention logic includes a detailed description of the needs, problems and drivers that the programme intended to address, the programme’s objectives (with a three-level hierarchy: global, specific and operational objectives), its inputs/activities, its expected outputs, outcomes and impacts and the logical links between these components.

1.The rationale for the intervention: ‘why did the EU establish the ISA² programme?’

The intervention logic starts by understanding the rationale of the ISA² programme. At this stage, the reasons for its establishment were identified by (i) assessing the needs and problems that it aims to address; and (ii) outlining EU decision-makers’ main objectives.

Needs and problems
[105](#footnote106)

As highlighted by the Digital Agenda for Europe
[106](#footnote107)
, interoperability plays a key role in maximising the social and economic potential of ICT. Against this background, three specific needs can be identified in the field of interoperability
[107](#footnote108)
:

the need for cooperation among public administrations to enable more efficient and secure public services;

the need for exchanging information among public administrations to fulfil legal requirements or political commitments; and

·the need for sharing and reusing information among public administrations to increase administrative efficiency and cut red tape for businesses and the public.

The Digital Single Market (DMS) Strategy
[108](#footnote109)
 confirmed these needs later on by recognising interoperability as a prerequisite for ‘efficient connections across borders, between communities and between public services and authorities’.

When it comes to problems, the ISA2 proposal
[109](#footnote110)
 outlines one main issue, namely the existence of administrative e-barriers leading to a fragmented market.

This particular problem is affected by the following eight drivers (as pointed out in a series of documents including the ISA2 proposal, the Final Evaluation of the ISA programme
[110](#footnote111)
, the Commission Staff Working Document accompanying the DSM Strategy
[111](#footnote112)
):

limited governance and coordination among Commission DGs and between Member States and the Commission on programmes and initiatives related to interoperability;

limited cooperation among dispersed institutional entities;

resource constraints in relation to interoperability;

internal organisational complexity;

limited ICT architectures and tools enabling interoperability;

limited availability of common frameworks, guidelines, and specifications;

limited awareness of the benefits of interoperability; and

·cultural fragmentation.
[112](#footnote113)

Objectives

The ISA2 objectives are presented in a hierarchical order, where the achievement of lower level objectives is normally a pre-condition for attaining the higher-level ones.

In this respect, the following three levels of objectives have been identified:

general objectives, concerning the overall rationale of an intervention and its longer-term and more diffuse effects, i.e. pertaining to the questions: ‘why has the ISA² programme been set up?’ and ‘what ultimate goal was it expected to contribute to?’;

specific objectives, providing a basis for assessing an intervention in relation to the short-term or medium-term results that occur at the level of direct beneficiaries/recipients of the intervention; and

·operational objectives, providing a basis for assessing an intervention in relation to its direct outputs, i.e. ‘what is directly produced/supplied during the ISA² programme’s implementation?’.

In concrete terms, the ISA2 programme’s objectives are the following:

General objective:

·To promote the ICT-based modernisation of the public sector in Europe and to facilitate addressing the needs of businesses and citizens via improved interoperability of European public administrations, thus contributing to the completion of the Digital Single Market and, ultimately, to economic growth and the global competitiveness of the European Union.
[113](#footnote114)
 

Specific objectives:

Specific objective #1: To develop, maintain and promote a holistic approach to interoperability in the Union in order to eliminate fragmentation in the interoperability landscape in the Union (Article 1(a) of the ISA2 Decision).

Specific objective #2: To facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations on the one hand, and between European public administrations and businesses and citizens on the other (Article 1(b) of the ISA2 Decision).

Specific objective #3: To contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local levels of public administration (Article 1(b) of the ISA2 Decision).

Specific objective #4: To identify, create and operate interoperability solutions supporting the implementation of Union policies and activities (Article 1(c) of the ISA2 Decision).

·Specific objective #5: To facilitate the reuse of interoperability solutions by European public administrations (Article 1(d) of the ISA2 Decision).

Operational objectives:

The operational objectives of the ISA2 programme are listed in the first column of 
[Table 6](#_Ref9329003)
. The additional columns are relevant for the programme’s expected results (see below for further details).

Table 6: Overview of operational objectives and expected outputs of the ISA² programme 

|  |  |  |
| --- | --- | --- |
| Operational objectives | Expected outputs | Performance indicators |
| Operational objective (1):  To support and promote the assessment, improvement, operation and re-use of existing cross-border or cross-sector interoperability solutions and common frameworks (Article 3 (a) of the ISA2 Decision) | ·Reuse of interoperable solutions and common frameworks  ·Reuse of best practices  ·Continuity in the delivery of interoperable solutions  ·Professional delivery of a coherent portfolio of interoperable solutions and common frameworks for European public administrations  ·Improved services and infrastructures  ·Improved tools  ·Continuity in the tools provision | ·Number of common services and generic tools used by European public administrations  ·Adoption of common frameworks by European public administrations  ·Users uptake  ·User satisfaction  ·Number of new sectors covered by existing solutions (cross-sector / cross-border dimension) |
| Operational objective (2):  To support and promote the development, establishment, bringing to maturity, operation and re-use of new cross-border or cross-sector interoperability solutions and common frameworks (Article 3 (b) of the ISA2 Decision) | ·Increased reuse of existing components in the development of new interoperable solutions and common frameworks  ·Increased availability of services and common frameworks that meet the needs of sectors and Member States  ·Delivery of new services and common frameworks in time  ·Increased availability of tools that meet the needs of sectors and Member States  ·Delivery of new tools in time | ·Number of new common services and generic tools by European public administrations  ·Adoption of new common frameworks by European public administrations |
| Operational objective (3):  To support and promote the assessment of the ICT implications of proposed or adopted Union law (Article 3 (c) of the ISA2 Decision) | ·Smooth implementation of EU legislation  ·Identification of the needs for services and tools in time so that requirements are fulfilled when the legislation comes into force  ·Understanding of ICT dimensions of EU policies | ·Number of pieces of legislation reviewed  ·Number of assessments of ICT implications of new EU legislation conducted |
| Operational objective (4):  To support and promote the identification of legislation gaps, at Union and national level, that hamper cross-border or cross-sector interoperability between European public administrations (Article 3 (d) of the ISA2 Decision) | ·A defined role and process for the ISA² programme to support the DGs and services undertaking assessments of ICT implications of new EU legislation as part of the Commission’s Impact Assessment process (Better Regulation Guidelines) [114](#footnote115)    ·Identification and resolution of legislation gaps that hamper cross-border or cross-sector interoperability between European public administrations | ·Number of instances where ISA² is involved (e.g. participation of ISA² in evaluations, impact assessments, and REFIT of EU legislation)  ·Number of pieces of gaps identified |
| Operational objective (5):  To support and promote the development of mechanisms that measure and quantify the benefits of interoperability solutions including methodologies for assessing cost-savings (Article 3 (e) of the ISA2 Decision) | ·Functioning mechanisms to promote the use and monitor the adoption of interoperability solutions as well as their numbers of users  ·Cost/Benefit model/methodology available for assessing cost-savings based on the study related to the cost-benefit of interoperability. | ·Usage of the Cost/Benefit model to produce figures to show the value of interoperability. |
| Operational objective (6):  To support and promote the mapping and analysis of the overall interoperability landscape in the Union through the establishment, maintenance and improvement of the EIRA and the EIC as instruments to facilitate the re-use of existing interoperability solutions and to identify the areas where such solutions are still lacking (Article 3 (f) of the ISA2 Decision) | ·Existing interoperability solutions documented within the European Interoperability Cartography (EIC/EICart) applying the principles and guidelines from the European Interoperability Reference Architecture (EIRA)  ·EIRA used by European public administrations  ·EIC/EICart facilitating the deployment of interoperable solutions within and between European public administrations | ·Usage of EIRA by European public administrations  ·Usage of EIC/EICart by European public administrations  ·Number of references in implementations of interoperable solutions  ·Number of interoperable solutions documented in the EIC/EICart |
| Operational objective (7):  To support and promote the maintenance, updating, promotion and monitoring of the implementation of the EIS, the EIF and the EIRA (Article 3 (g) of the ISA2 Decision) | ·Interoperable solutions developed in the frame of ISA² that take into account EIF and EIRA  ·Interoperable solutions developed by other Commission DGs that take into account EIF and EIRA  ·ISA² actions implementing the priorities/actions set by the EIS | ·Usage of EIRA by European public administrations  ·Number of EIS priorities implemented by European public administrations  ·Level of implementation of the EIF at national and Commission level |
| Operational objective (8):  To support and promote the assessment, updating and promotion of existing common specifications and standards and the development, establishment and promotion of new common specifications and open specifications and standards through the Union's standardisation platforms and in cooperation with European or international standardisation organisations as appropriate (Article 3 (h) of the ISA2 Decision) | ·Adoption of ‘standardised’ specific outputs from previous programme (e.g. Core Vocabularies, ADMS) by European public administrations.  ·Standards, open specifications and assessment procedures aligned with Member States  ·International/European recognition of standards and open specifications  ·Established new standards and open specifications  ·Alignment of standards with EU standardisation platforms | ·Number of new or revised standards used by European public administrations  ·Number of new or revised common specifications and open specifications used by European public administrations  ·Number of assessments (e.g. through multi-stakeholder platform or CAMMS) |
| Operational objective (9):  To support and promote the maintenance and publication of a platform allowing access to, and collaboration with regard to, best practices, functioning as a means of raising awareness and disseminating available solutions, including security and safety frameworks, and helping to avoid duplication of efforts while encouraging the re-usability of solutions and standards (Article 3 (i) of the ISA2 Decision) | ·All existing ready-for-use interoperable and best practice solutions are documented in and accessible via Joinup [115](#footnote116)  to raise awareness and avoid duplication of efforts among stakeholders | ·Number of ready-for-use interoperable and best practice solutions housed in Joinup |
| Operational objective (10):  To support and promote the bringing of new interoperability services and tools to maturity, and maintaining and operating existing interoperability services and tools on an interim basis (Article 3 (j) of the ISA2 Decision) | ·Continued development and interim operation of ISA solutions, e.g. sTESTA, with a view to achieving longer-term sustainable financing and operational governance structures | ·Number of new interoperability services and tools developed, maintained and operated by ISA²  ·Number of new interoperability services brought to maturity |
| Operational objective (11):  To support and promote the identification and promotion of best practices, to develop guidelines to coordinate interoperability initiatives and to animate and support communities working on issues relevant to the area of electronic cross-border or cross-sector interaction between end-users (Article 3 (k) of the ISA2 Decision) | ·Continued use of Joinup to house ready-for-use interoperable and best practice solutions to promote interoperability  ·Enhanced coordination, notably, through the Inter-Service Group on Public Administration Quality and Innovation and also in publications, such as the NIFO and eGovernment factsheets | ·Number of ready-for-use interoperable and best practice solutions housed in Joinup  ·Number of joint-initiatives (studies etc.) in interoperability by EU-wide groups, such as the Commission’s Inter-Service Group on Public Administration Quality and Innovation.  ·Number of joint-publications |
| Operational objective (12):  To develop a communication strategy […], aiming to enhance information and increase awareness with regard to the ISA² programme and its benefits, targeting businesses, including SMEs, and citizens, and employing user-friendly means on the ISA² programme's website (Article 3 of the ISA2 Decision) | ·An updated Communication Strategy with clear stakeholder, monitoring and measurement targets. | ·Monitoring and measurement of the Communication Strategy (events, satisfaction surveys, publications).  ·Numbers of stakeholders engaged in different events, actions, initiatives etc. |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

2.The intervention: ‘what actions are supported by the ISA² programme?’

After identifying the needs, problems and objectives underlying the ISA² programme, the intervention logic approach requires the assessment of the main features (activities/inputs) of the programme itself. Within the intervention logic, activities and inputs represent the means to address the needs and problems, and to meet the identified objectives. In this context, these means largely correspond to the actions of the ISA² programme, as defined in the ISA2 Rolling Work Programme
[116](#footnote117)
. The ISA2 actions are grouped into the following nine packages.

Key and generic interoperability enablers are actions that develop interoperability solutions to support public administrations in providing services, ranging from the secure exchange of files to improving cross-border access to government data and information. Examples include TESTA
[117](#footnote118)
 and Open e-TrustEx
[118](#footnote119)
.

The semantic interoperability package consists of initiatives to establish and increase semantic interoperability among public administrations through information standards and specifications. Examples include Core Vocabularies
[119](#footnote120)
 and DCAT Application Profile for data portals in Europe
[120](#footnote121)
.

Under the access to data / data sharing / open data package, the ISA² programme funds actions that help facilitate the reuse of national data repositories, of data across borders and sectors and widen access to data created by the public sector. Example include ‘Providing big data opportunities for public administrations’
[121](#footnote122)
 and ‘FISMA: Financial Data Standardisation’
[122](#footnote123)
.

The geospatial solutions package consists of legal/policy, organisational, semantic and technical interoperability solutions that aims to facilitate efficient and effective electronic cross-border and cross-sector interaction between European public administrations, and between them and citizens and companies, in the field of location information and services. This package led to the Re3gistry
[123](#footnote124)
 solution.

The eProcurement/eInvoicing package helps update and improve existing EU tools for e-procurement. It aims to develop a common public procurement knowledge base to facilitate the creation, exchange, dissemination and reuse of procurement data. Examples include Open e-Prior
[124](#footnote125)
, ESPD
[125](#footnote126)
 and eCertis
[126](#footnote127)
.

The decision-making and legislation package covers actions that support the decision-making process, in particular by improving interoperability between EU institutions and between them and the Member States. These actions address several processes – ranging from collecting feedback from various stakeholders to drafting new legislation and monitoring that this legislation is implemented. The package led to the LEOS
[127](#footnote128)
 solution.

The EU policies — supporting instruments package covers actions that support the implementation of EU policies for which interoperability aspects need to be considered. The package has led to the following solution: Online Collection Software to support European Citizens' Initiatives.
[128](#footnote129)

The supporting instruments for public administrations package maps the EU’s interoperability landscape and provides solutions to support greater interoperability, ranging from sharing best practices to supporting the reuse of solutions.  Examples include EUSurvey
[129](#footnote130)
, Joinup – European Collaborative Platform and Catalogue
[130](#footnote131)
 and the National Interoperability Framework Observatory (NIFO)
[131](#footnote132)
.

·The accompanying measures package consists of activities to raise awareness of interoperability and monitoring and evaluating the programme’s implementation. Examples include the annual SEMIC Conference
[132](#footnote133)
, the ISA2 Mid-Term Conference
[133](#footnote134)
 and the ISA2 Dashboard
[134](#footnote135)
.

3.The expected results of the intervention: ‘what are the expected effects of the actions supported by the ISA² programme?’

When it began, the ISA² programme was expected to have certain effects (i.e. to cause changes) on certain categories of stakeholders. These effects fall under three categories (outputs, outcomes and impacts) depending on when they occur and the groups they address. Note that external factors and other EU policies may influence the performance of the ISA² programme.

Outputs (expected)

The outputs of the ISA² programme are its most immediate effects, i.e. the deliverables/products of the funded actions. As the intervention logic looks at a certain intervention by simulating an ex ante perspective, one must identify the outputs that were expected to stem from the programme when it was initially adopted. Note that expected outputs usually reflect the operational objectives identified in previous analytical steps. In this respect, the expected outputs and related indicators are listed in the ex-ante evaluation of ISA². 
[Table 6](#_Ref9329003)
 above summarises the expected outcomes and a series of key performance indicators for each operational objective of the ISA² programme.

Outcomes (expected)

The outcomes are the short-term and medium-term changes for the groups the programme directly addresses. As outcomes are usually connected to the specific objectives, they can be summarised as follows:

a coherent interoperability landscape in the Union based on a holistic approach to interoperability;

efficient and effective electronic cross-border or cross-sectoral interactions between European Public Administrations as well as between European public administrations and businesses/citizens;

more effective, simplified and user-friendly e-administrations in European public administrations;

advancement of Union policies and activities by supporting their implementation; and

·take-up and re-use of the ISA2 programme’s results by European public administrations.

Impacts (expected)

As mentioned above, the intervention logic also includes the impacts, i.e. the changes caused by an EU intervention over a longer period and also affecting the society as a whole rather than only the direct addressees of the intervention itself. These changes are related to the general objectives of a certain intervention. By analysing the ISA2 Decision and the ISA2 proposal, the following impacts are expected:

increasing the speed, efficiency and quality in the creation and delivery of cross-border and cross-sector electronic public services to meet the needs of businesses and citizens;

improving the efficiency and productivity of the European public administrations, which can be a strong driver of economic growth through its support for, and governance of, the private sector;

reducing the cost and administrative burden of cross-border interaction, removing administrative e-barriers and contributing to the swift implementation of ICT systems supporting EU legislation; and

·contributing to the successful achievement of the DSM.

Annex 4: Evaluation Framework

This annex presents the evaluation framework
[135](#footnote136)
 on which the entire evaluation is based.

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| Evaluation questions | Success/judgment criteria | Indicators | Data sources | Data collection / analysis methods |
| Evaluation criterion #1: Relevance | | | | |
| 1.To what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels? | ·Degree of alignment between stakeholders’ perception of needs and problems at national and sub-national levels and the objectives of the programme.  ·Degree of alignment between stakeholders’ perception of needs and problems at EU level and the objectives of the programme.  ·Degree of alignment between needs and problems originally addressed by the programme and stakeholders’ perception of needs and problems | ·Share of stakeholders confirming the alignment between needs and problems addressed by the programme and current needs and problems.  ·Share of stakeholders confirming the alignment between the objectives of the programme and current needs and problems at national and sub-national levels.  ·Share of stakeholders confirming the alignment between the objectives of the programme and current needs and problems at EU level.  ·Qualitative assessment of the alignment between the objectives of the programme and current needs and problems.  ·Qualitative assessment of the alignment between needs and problems addressed by the programme and current needs and problems. | ·Primary information on needs and problems from the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked EU policies/initiatives.  oExperts.  oIndirect beneficiaries and wider public.  ·Secondary information on needs and problems from operational documents, other official documents and relevant literature, such as:  oISA² Decision and accompanying documents.  oISA² Rolling Work Programme.  oeGovernment factsheets.  oStudy on the role of eGovernment and interoperability in the European Semester.  oState of Play of Interoperability in Europe.  oData on the digital economy from Eurostat.  oInterim review of the DSM Strategy. | ·Desk research.  ·Interviews with the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked EU policies/initiatives.  oStandardisation organisations.  ·Online surveys targeted to the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked policies/initiatives.  ·Short questionnaire to be administered during the ISA2 Mid-Term Conference.  ·Public consultation.  ·Quantitative assessment of responses to interviews and surveys (Likert scale).  ·Qualitative assessment of responses to interviews and surveys and data and information collected via desk research. |
| Evaluation criterion #2: Effectiveness | | | | |
| 2.How far are the ISA² programme's results in the process of achieving the programme's objectives?  3.Are there aspects that are more or less effective than others, and if so, what lessons can be drawn from this? | ·Degree of alignment between actual and expected results and objectives of the programme.  ·Impact of external factors on the performance of the programme.  ·Measurement of the indicators summarising the outputs of the programme.  ·Degree of alignment with principles spelled out in Article 4 of the ISA2 Decision.  ·Awareness of the programme. | ·Share of stakeholders confirming the alignment between actual and expected results of the programme.  ·Share of stakeholders confirming the alignment between the objectives and actual results of the programme.  ·Share of stakeholders who are aware of the programme.  ·Share of stakeholders identifying external factors contributing to/jeopardising the performance of the programme.  ·Share of stakeholders who are aware of specific ISA2 packages/ actions/ solutions.  ·Qualitative assessment of the alignment between objectives, expected and actual results of the programme.  ·Quantitative assessment of performance indicators of outputs.  ·Qualitative assessment of the alignment with principles spelled out in Article 4 of the ISA2 Decision. | ·Primary information actual results and contribution to the programme’s objectives from the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oExperts.  oIndirect beneficiaries and wider public.  ·Secondary information from operational documents and other official documents, such as:  oISA² Rolling Work Programme  oISA² Annual Monitoring and Evaluation Reports.  oISA² Quarterly Monitoring Reports.  oISA² Dashboard.  oDocumentary evidence on funded actions (e.g. deliverables, final reports). | ·Desk research.  ·Interviews with the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStandardisation organisations.  ·Online surveys targeted to the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  ·Short questionnaire to be administered during the ISA2 Mid-Term Conference.  ·Public consultation.  ·Quantitative assessment of responses to interviews and surveys (Likert scale).  ·Qualitative assessment of responses to interviews and surveys and data and information collected via desk research.  ·Multi-criteria analysis.  ·Success stories/lessons learnt. |
| Evaluation criterion #3: Efficiency | | | | |
| 4.To what extent has the programme been cost-effective?  5.Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised?  5.1    How is the programme performing relative to the planned work and budget? | ·Cost-effectiveness analysis to assess the ratio between allocated funds and actual results of the programme.  ·Earned value management analysis.  ·Efficiency of the selection process of the actions to be included in the Rolling Work Programme. | ·Comparison between the costs of ISA2 packages and the results measured via performance indicators.  ·Earned Value, Actual Costs, Planned Value, and Schedule Performance Index of ISA2 packages. [136](#footnote137)  ·Share of stakeholders confirming that the selection process of the actions is fit-for-purpose.  ·Assessment of regulatory costs linked to the selection process of the actions included in the Rolling Work Programme. | ·Primary information on costs from the following categories of stakeholders:  oProgramme governance.  oAction owners.  ·Secondary information from operational documents and other official documents, such as:  oISA² Rolling Work Programme  oISA² Annual Monitoring and Evaluation Reports.  oISA² Quarterly Monitoring Reports.  oISA² Dashboard.  oDocumentary evidence on funded actions (e.g. deliverables, final reports). | ·Desk research.  ·Interviews with the following categories of stakeholders:  oProgramme governance.  oAction owners.  ·Online surveys targeted to the following categories of stakeholders:  oProgramme governance.  oAction owners.  ·Quantitative assessment of responses to surveys (Likert Scale).  ·Qualitative assessment of responses to interviews and surveys and data and information collected via desk research.  ·Cost-effectiveness analysis.  ·Standard cost model.  ·Earned value management.  ·Success stories/lessons learnt. |
| Evaluation criterion #4: Coherence | | | | |
| 6.To what extent do the ISA² actions form part of a ‘holistic’ approach within the framework of the programme? (internal coherence)  7.To what extent is the ISA² programme coherent with other EU interventions that have similar objectives and with global initiatives in the same field? (external coherence) | ·Degree of coherence among actions funded by the ISA² programme (internal coherence).  ·Degree of coherence between the programme and other EU supported programmes (external coherence).  oFocus on CEF, SRSP, Horizon 2020  ·Degree of coherence between the programme and other EU policies (external coherence).  oFocus on DSM, Digital Strategy (forthcoming), ICT standardisation, Single digital gateway, eGovernment Action Plan, Tallinn Declaration on eGovernment.  ·Degree of coherence between the programme and global initiatives in the field (external coherence).  oFocus on OECD Digital Government and the UNPAN.  ·Level of reuse of results of a funded action by another action within the ISA² programme (internal coherence).  ·Level of reuse of results delivered by ISA² actions by other EU programmes (external coherence). | ·Share of stakeholders identifying synergies/overlaps between funded actions.  ·Share of stakeholders identifying synergies/overlaps between the programme and other relevant EU programmes/policies.  ·Qualitative assessment of synergies/overlaps and links between funded actions.  ·Qualitative assessment of synergies/overlaps between objectives of the programme and other relevant EU programmes/policies.  ·Qualitative assessment of synergies/overlaps between funded actions and those of other relevant EU programmes.  ·Qualitative assessment of synergies/overlaps between the programme and global initiatives in the same field.  ·Instances of reuse of results delivered by funded actions by other actions within the programme.  ·Instances of reuse of results delivered by funded actions by other EU programmes. | ·Primary information on internal coherence from the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  ·Primary information on external coherence from the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked EU policies/initiatives.  oExperts.  oIndirect beneficiaries and wider public.  ·Secondary information from operational documents and other official documents, such as:  oISA² Decision and accompanying documents.  oISA² Rolling Work programme.  oISA² Annual Monitoring and Evaluation Reports.  oISA² Quarterly Monitoring Reports.  oDocumentary evidence on funded actions (e.g. official deliverables, final reports).  oLegal texts establishing other relevant programmes accompanying documents.  oLegal texts devising other relevant EU policies and accompanying documents.  oDocuments describing global initiatives in the same field. | ·Desk research.  ·Interviews with the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked EU policies/initiatives.  oStandardisation organisations.  ·Online surveys targeted to the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked policies/initiatives.  ·Short questionnaire to be administered during the ISA2 Mid-Term Conference.  ·Public consultation.  ·Quantitative assessment of responses to interviews and surveys (Likert scale).  ·Qualitative assessment of responses to interviews and surveys and data and information collected via desk research.  ·Success stories/lessons learnt. |
| Evaluation criterion #5: EU added value | | | | |
| 8.What is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels? | ·Achievement of objectives that could not be otherwise attained with national or sub-national interventions.  ·Achievement of objectives at a cost lower than what could be attained via national or sub-national interventions.  ·Achievement in terms of cross-border interoperability.  ·Contribution to the advancement of common EU policies. | ·Share of stakeholders confirming the need for an EU intervention to achieve the objectives of the programme.  ·Share of stakeholders confirming that an EU intervention is able to achieve the objectives of the programme at cost lower than costs of national or sub-national interventions.  ·Share of stakeholders’ providing positive feedback on achievements in terms of cross-border interoperability.  ·Qualitative assessment of the contribution to the advancement of common EU policies.  ·Quantitative assessment of indicators summarising cross-border outputs of the programme. | ·Primary information on cross-border interoperability and EU added value from the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oExperts.  oIndirect beneficiaries and wider public.  ·Secondary information from operational documents and other official documents, such as:  oISA² Decision and accompanying documents.  oISA² Rolling Work programme.  oISA² Annual Monitoring and Evaluation Reports.  oISA² Quarterly Monitoring Reports.  oDocumentary evidence on funded actions (e.g. official deliverables, final reports). | ·Desk research.  ·Interviews with the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStandardisation organisations.  ·Online surveys targeted to the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  ·Short questionnaire to be administered during the ISA2 Mid-Term Conference.  ·Public consultation.  ·Quantitative assessment of responses to interviews and surveys (Likert scale).  ·Qualitative assessment of responses to interviews and surveys and of data and information collected via desk research.  ·Success stories/lessons learnt. |
| Evaluation criterion #6: Utility [137](#footnote138) | | | | |
| 9.How do the ISA² programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address? [138](#footnote139) | ·Degree of alignment between stakeholders’ perception of needs and problems at national and sub-national levels and the results of the programme.  ·Degree of alignment between stakeholders’ perception of needs and problems at EU level and the results of the programme.  ·User satisfaction, with a breakdown by stakeholder group. | ·Share of stakeholders confirming the alignment between the results of the programme and current needs and problems at national and sub-national levels.  ·Share of stakeholders confirming the alignment between the results of the programme and current needs and problems at EU level.  ·Qualitative assessment of the alignment between the results of the programme and current needs and problems.  ·Quantitative assessment of users’ satisfaction (Likert Scale).  ·Quantitative assessment of the take-up by EU, national and sub-national administrations of ISA2 outputs. | ·Primary information on user satisfaction and utility from the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked EU policies/initiatives.  oExperts.  oIndirect beneficiaries and wider public.  ·Secondary information on utility from operational documents, other official documents and relevant literature, such as:  oISA² Rolling Work Programme.  oISA² Annual Monitoring and Evaluation Reports.  oISA² Quarterly Monitoring Reports.  oDocumentary evidence on funded actions (e.g. official deliverables, final reports). | ·Desk research.  ·Interviews with the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked EU policies/initiatives.  oStandardisation organisations.  ·Online surveys targeted to the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStakeholders responsible for linked policies/initiatives.  ·Short questionnaire to be administered during the ISA2 Mid-Term Conference  ·Public consultation  ·Quantitative assessment of responses to interviews and surveys (Likert scale)  ·Qualitative assessment of responses to interviews and surveys and of data and information collected via desk research.  ·Success stories/lessons learnt. |
| Evaluation criterion #7: Sustainability | | | | |
| 10.To what extent is the financial, technical and operational sustainability of the developed solutions – maintained and operated through the ISA² programme –ensured? | ·Extent to which the results achieved by the ISA2 programme are expected to last if funding for actions covered by the programme would not be available in the future.  ·Extent to which ‘cost recovery’ solutions could be introduced. | ·Share of stakeholders expecting that results achieved so far would last if funding for actions covered by the programme would not be available in the future.  ·Share of actions requiring operation and maintenance costs to deliver their results.  ·Share of actions requiring technical and operational support to deliver their results.  ·Share of stakeholders who would pay to keep on using specific ISA2 solutions. | ·Primary information on sustainability from the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oExperts.  oIndirect beneficiaries and wider public.  ·Secondary information on sustainability from operational documents, other official documents and relevant literature, such as:  oISA² Rolling Work Programme.  oISA² Annual Monitoring and Evaluation Reports.  oISA² Quarterly Monitoring Reports.  oDocumentary evidence on funded actions (e.g. official deliverables, final reports). | ·Desk research.  ·Interviews with the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  oStandardisation organisations.  ·Online surveys targeted to the following categories of stakeholders:  oProgramme governance.  oAction owners.  oSolution Users – European Commission.  oSolution Users – Member States.  ·Quantitative assessment of responses to interviews and surveys (Likert scale).  ·Qualitative assessment of responses to interviews and surveys and of data and information collected via desk research.  ·Quantitative assessment of operation and maintenance costs.  ·Success stories/lessons learnt. |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Annex 5: Sample of actions

The purpose of the interim evaluation is to provide an overall assessment of the performance of ISA2. A sample of actions has therefore been selected to better guide the data collection activities.
[139](#footnote140)
 More specifically, the evaluation is based on a sample of 20 of the 53 actions included in the 2018 Rolling Work Programme. The following four criteria were used when selecting the sample:

1.Action packages: the selected actions should be largely representative of the nine ISA2 packages of actions, as they are defined in the Rolling Work Programme.

2.New actions and actions continued from ISA: within each action package, the selected actions should be representative of two clusters, namely those actions that have been continued from ISA and those actions that have been started under ISA2.

3.Action start date: to provide the evaluation with enough data, no action started in 2018 should be selected.

4.Action end date: to the extent possible, those actions that have already ended should be included in the sample.

To build the sample, a number of steps were taken. First, the 53 actions funded by ISA2 were grouped according to the packages to which they belong. Second, within each package, the evaluation team identified those actions that were continued from ISA and those that were created under ISA2. Third, a target sample size of 20 actions (i.e. more than one-third of the total) was set to ensure the feasibility of the evaluation while allowing for a comprehensive sample of typical actions. Then the evaluation team identified the number of actions to be selected in order to reflect: i) the proportions of actions from each package relative to the total action population; and ii) the proportion of old and new actions compared to the total number of actions per package. The relative proportions were rounded off, taking into consideration the third and fourth sampling criteria, namely the exclusion of actions started in 2018 and, to the extent possible, the inclusion of actions that have already ended (
[Table 7](#_Ref9347455)
). The final sample is presented in 
[Table 8](#_Ref9347844)
, which also provides an overview of the solutions stemming from selected actions. Sampled actions have been randomly selected to avoid any selection bias.

Table 7: Number of sampled actions by package and programme

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Package | Number of actions | Number of ISA actions | Number of ISA2 actions | Package actions / Total actions proportion | Number of sampled actions (total) | Number of sampled actions (ISA) | Number of sampled actions (ISA2) |
| 1. Key and generic interoperability enablers | 7 | 5 | 2 | 13% | 2 | 2 | 0 |
| 2. Semantic interoperability | 4 | 1 | 3 | 8% | 2 | 1 | 1 |
| 3. Access to data/data sharing/open data | 7 | 2 | 5 | 13% | 3 | 1 | 2 |
| 4. Geospatial solutions | 1 | 1 | 0 | 2% | 1 | 1 | 0 |
| 5. eProcurement/ eInvoicing - Supporting instruments | 1 | 1 | 0 | 2% | 1 | 1 | 0 |
| 6. Decision making and legislation - Supporting instruments | 10 | 5 | 5 | 19% | 3 | 1 | 2 |
| 7. EU Policies - Supporting instruments | 5 | 3 | 2 | 9% | 1 | 1 | 0 |
| 8. Supporting instruments for public administrations | 16 | 11 | 5 | 30% | 6 | 5 | 1 |
| 9. Accompanying measures | 2 | 0 | 2 | 4% | 1 | 0 | 1 |
| TOTAL | 53 | 29 | 24 | 100% | 20 | 13 | 7 |

Note: The column ‘Number of ISA actions’ shows the number of actions continued under ISA2 from the programme’s previous editions. The column ‘Number of ISA2 actions’ shows the number of new actions under ISA2 that are not a direct continuation of any actions of the previous editions of the programme.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Table 8: Sampled actions and relevant solutions

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Package | Action number | Action Name | ISA/ ISA2 | Solutions | | | |
| 1. Key and generic interoperability enablers | 2016.19 | Trusted Exchange Platform (e-TrustEx) | ISA | Common tool/service: Open e-TrustEx |  |  |  |
| 1. Key and generic interoperability enablers | 2016.29 | Catalogue of Services | ISA | Common framework: Core Public Service Vocabulary Application Profile (CPSV-AP) |  |  |  |
| 2. Semantic interoperability | 2016.07 | SEMIC: Promoting Semantic Interoperability Amongst the European Union Member States | ISA | Common framework (data models, data standards): Core vocabularies | Common specification/standard: Asset Description Metadata Schema (ADMS) | Common specification: DCAT Application Profile for data portals in Europe (DCAT-AP), GeoDCAT-AP, StatDCAT-AP | Common tool/service: VocBench3 |
| 2. Semantic interoperability | 2016.16 | Public Multilingual Knowledge Management Infrastructure for the Digital Single Market | ISA2 | Common framework: PMKI Core data model for multilingual taxonomies/terminologies | Common specification/standard: Semantic links - core dataset with additional semantic links between different language resources (in particular having EuroVoc as pivot) |  |  |
| 3. Access to data/data sharing/open data | 2016.03 | Big Data for Public Administrations | ISA | Study: Big data analytics for policy making | Common tool/service: DORIS - stakeholders' feedback analysis tool | Study: Big Data Test Infrastructure |  |
| 3. Access to data/data sharing/open data | 2016.06 | Sharing Statistical Production and Dissemination Services and Solutions in the European Statistical System | ISA2 | Common framework: ESS: Statistical Production Reference Architecture v1.0 |  |  |  |
| 3. Access to data/data sharing/open data | 2016.18 | Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification | ISA2 | Common tool/service: Catalogue of data visualisation tools (part of EU Open Data portal) |  |  |  |
| 4. Geospatial solutions | 2016.10 | European Location Interoperability Solutions for e-Government (ELISE) | ISA | Common tool/service: Re3gistry | Common framework: EULF Blueprint | Common tool/services: Common services, pilots and applications; | Common tool/service: INSPIRE Reference Validators and Interoperability Testing (test framework) |
| 5. eProcurement/  eInvoicing - Supporting instruments | 2016.05 | European Public Procurement Interoperability Initiative | ISA | Common tool/service: Open e-Prior | Common tool/service: eCertis | Common tool/service: European Single Procurement Document (ESPD) service as well as a data model on the ESPD |  |
| 6. Decision making and legislation - Supporting instruments | 2016.23 | Legal interoperability (former ICT Implications of EU Legislation) | ISA | Common tool: Tool #27 of the Better Regulation Toolbox: The digital economy and society & ICT issues | Common framework: ICT Impact Assessment Guidelines |  |  |
| 6. Decision making and legislation - Supporting instruments | 2017.03 | [REFIT Platform](https://ec.europa.eu/isa2/dashboard/node/92618) | ISA2 | Common tool/service: REFIT Platform IT Tool |  |  |  |
| 6. Decision making and legislation - Supporting instruments | 2017.04 | [Inter-Institutional Register of Delegated Acts (RegDel)](https://ec.europa.eu/isa2/dashboard/node/45312) | ISA2 | Common tool/service: Register of Delegated Acts (RegDel) |  |  |  |
| 7. EU Policies - Supporting instruments | 2016.14 | European Citizens’ Initiatives and European Parliament Elections | ISA | Common tool/service: Online Collection Software to support European Citizens’ Initiatives (OCS for ECIs) | Common tool/service: European Parliament Crypto Tool |  |  |
| 8. Supporting instruments for public administrations | 2016.20 | [Joinup – European Collaborative Platform and Catalogue](https://ec.europa.eu/isa2/dashboard/node/645) | ISA | Common tool/service: Joinup |  |  |  |
| 8. Supporting instruments for public administrations | 2016.21 | National Interoperability Framework Observatory | ISA | Common framework: NIFO factsheets | Common framework: State of play of interoperability in Europe |  |  |
| 8. Supporting instruments for public administrations | 2016.32 | European Interoperability Architecture (EIA) | ISA | Common framework: European Interoperability Reference Architecture (EIRA) | Common tool/service: CarTool |  |  |
| 8. Supporting instruments for public administrations | 2016.35 | EUSurvey | ISA | Common tool/service: EUSurvey | Common tool/service: DORIS |  |  |
| 8. Supporting instruments for public administrations | 2016.37 | Interoperability Maturity Assessment of a Public Service (IMAPS) | ISA | Common tool/service: Interoperability Maturity Assessment of a Public Service (IMAPS) |  |  |  |
| 8. Supporting instruments for public administrations | 2017.01 | Standard-Based Archival Data Management, Exchange and Publication | ISA2 | Study: Study on Standard-Based Archival Data Management, Exchange and Publication | Common tool: Assessment tool offering support for the selection of IT solutions for archives management. |  |  |
| 9. Accompanying measures | 2016.3 | Raising Interoperability Awareness – Communication Activities | ISA2 | Events organised by ISA2 | Events in which ISA2 participated |  |  |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Annex 6: Supporting evidence from desk research

This annex presents evidence collected from desk research of performance indicators, the rolling work programme, the information available online in the webpages dedicated to the individual ISA2 actions and solutions, and additional evidence received from the action owners of the 20 sampled actions. In the following section, evidence is presented by evaluation criterion and question.

Effectiveness

EQ2: How far are the ISA² programme's results in the process of achieving the programme's objectives?

EQ3: Are there aspects that are more or less effective than others, and if so, what lessons can be drawn from this?

Types of users and their geographical distribution

Table 9: EU Member States using solutions provided by sampled ISA2 actions

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | BE | BG | CZ | DK | DE | EE | IE | EL | ES | FR | HR | IT | CY | LV | LT | LU | HU | MT | NL | AT | PL | PT | RO | SL | SK | FI | SE | UK | TOT. |
| ADMS |  |  |  |  | x |  |  |  | x |  |  | x |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 3 |
| Core Vocabularies | x |  |  |  |  | x |  | x | x |  |  | x |  |  |  |  |  |  | x |  |  |  |  |  |  | x |  |  | 7 |
| CPSV-AP | x |  |  |  |  | x |  |  | x |  |  | x |  |  |  |  |  |  | x |  |  |  |  |  |  | x |  |  | 6 |
| DCAT-AP | x |  |  |  | x |  | x |  | x |  |  | x |  |  |  |  |  |  | x |  |  |  |  |  |  |  | x |  | 7 |
| eCertis | x | x | x | x | x |  | x | x | x | x | x | x | x |  | x |  | x |  | x | x | x | x |  | x | x | x | x | x | 23 |
| EIRA | x |  | x | x |  | x |  |  | x |  |  |  |  |  |  | x |  |  | x |  |  |  |  |  |  |  |  |  | 7 |
| e-PRIOR | x |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 1 |
| ESPD\* | x | x |  | x | x | x | x | x | x | x | x | x |  | x | x |  | x | x | x | x | x | x | x |  | x | x | x | x | 23 |
| Open e-TrustEx\*\* | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | 28 |
| European Parliament Crypto Tool | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | x | 28 |
| EUSurvey |  |  |  |  |  |  |  |  | x |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 1 |
| GeoDCAT-AP |  |  |  |  | x |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 1 |
| IMAPS |  |  | x |  |  |  |  | x |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | x |  | 2 |
| Joinup | x | x | x | x | x | x | x | x | x | x | x | x |  | x | x | x | x |  | x | x | x | x | x | x | x | x | x | x | 26 |
| NIFO | x | x | x | x | x | x | x | x | x | x | x |  | x | x | x | x | x | x | x |  | x | x | x | x | x | x | x | x | 26 |
| OCS for ECIs |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | x |  |  |  |  |  |  |  |  |  |  |  |  | 1 |
| Re3gistry |  |  |  |  |  |  |  |  | x | x |  | x |  |  |  |  |  |  |  | x |  |  |  |  | x | x |  |  | 6 |

\*Note: Information provided by action owner based on the following Commission document on the CEF eProcurement Digital Service Infrastructure (November 2018): 
<https://ec.europa.eu/inea/sites/inea/files/dsi_fiche_eprocurement.pdf>
 

\*\*Note: Information provided by action owner.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)
, which built on the data available on 
[ISA2 Solutions webpage](https://ec.europa.eu/isa2/solutions_en)
 as of January 2019.

Table 10: EU institutions using solutions provided by sampled ISA2 actions

|  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | European Commission | Council of the EU | European Parliament | Publications Office of the EU | European Council | European Central Bank | TOTAL |
| ADMS |  |  |  | x |  |  | 1 |
| Core Vocabularies | x |  |  | x |  |  | 2 |
| DCAT-AP | x |  |  | x |  |  | 2 |
| eCertis | x |  |  |  |  | x | 2 |
| EIRA and CarTool | x | x |  | x |  |  | 3 |
| e-TrustEx | x | x | x | x | x | x | 6 |
| EUSurvey | x |  | x | x | x |  | 4 |
| GeoDCAT-AP | x |  |  |  |  |  | 1 |
| Open e-Prior | x |  |  | x |  |  | 2 |
| Re3gistry | x |  |  |  |  |  | 1 |
| RegDel | x | x | x | x |  |  | 4 |
| VocBench3 | x |  |  | x |  |  | 2 |
| TOTAL | 11 | 3 | 3 | 9 | 2 | 2 | 30 |

\*Note: Information provided by action owner.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Table 11: Status of sampled actions and their cross-border feature

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Package | Action number | Action Name | Operational solutions | Out of which cross-border solution | Solutions under development | |
| 1. Key and generic interoperability enablers | 2016.19 | Trusted Exchange Platform (e-TrustEx) | 1 | 1 | - | |
| 1. Key and generic interoperability enablers | 2016.29 | Catalogue of Services | 1 | 1 | - | |
| 2. Semantic interoperability | 2016.07 | SEMIC: Promoting Semantic Interoperability Amongst the European Union Member States | 4 | 2 | - | |
| 2. Semantic interoperability | 2016.16 | Public Multilingual Knowledge Management Infrastructure for the Digital Single Market | - | - | 4 | |
| 3. Access to data/data sharing/open data | 2016.03 | Big Data for Public Administrations | 3 | 3 | 1 | |
| 3. Access to data/data sharing/open data | 2016.06 | Sharing Statistical Production and Dissemination Services and Solutions in the European Statistical System | 2 | 2 | 6 | |
| 3. Access to data/data sharing/open data | 2016.18 | Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification | 1 | 1 | 1 | |
| 4. Geospatial solutions | 2016.10 | European Location Interoperability Solutions for e-Government (ELISE) | 4 | 4 | 1 | |
| 5. e-Procurement/ e-invoicing - Supporting instruments | 2016.05 | European Public Procurement Interoperability Initiative | 3 | 3 | 5 | |
| 6. Decision making and legislation - Supporting instruments | 2016.23 | Legal interoperability (former ICT Implications of EU Legislation) | 2 | - | - | |
| 6. Decision making and legislation - Supporting instruments | 2017.03 | [REFIT Platform](https://ec.europa.eu/isa2/dashboard/node/92618) | - | - | 1 | |
| 6. Decision making and legislation - Supporting instruments | 2017.04 | [Inter-Institutional Register of Delegated Acts (RegDel)](https://ec.europa.eu/isa2/dashboard/node/45312) | 1 | - | - | |
| 7. EU Policies - Supporting instruments | 2016.14 | European Citizens’ Initiatives and European Parliament Elections | 2 | 2 | - | |
| 8. Supporting instruments for public administrations | 2016.20 | [Joinup – European Collaborative Platform and Catalogue](https://ec.europa.eu/isa2/dashboard/node/645) | 1 | 1 | - | |
| 8. Supporting instruments for public administrations | 2016.21 | National Interoperability Framework Observatory | 3 | 3 | - | |
| 8. Supporting instruments for public administrations | 2016.32 | European Interoperability Architecture (EIA) | 2 | 2 | 1 | |
| 8. Supporting instruments for public administrations | 2016.35 | EUSurvey | 2 | 1 | - | |
| 8. Supporting instruments for public administrations | 2016.37 | Interoperability Maturity Assessment of a Public Service (IMAPS) | 1 | 1 | - | |
| 8. Supporting instruments for public administrations | 2017.01 | Standard-Based Archival Data Management, Exchange and Publication | 2 | 2 | 1 | |
| 9. Accompanying measures | 2016.3 | Raising Interoperability Awareness – Communication Activities | The full overview of events organised is listed   in  [Annex 1](#_Ref9346654) :  [Table 13](#_Ref9595195) . | | | |
| Totals | | | 35 | 29 | | 21 |

Source: European Commission’s own elaboration

Performance indicators

Table 12: Overview of performance indicators for sampled ISA2 actions

|  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Package | ISA / ISA2 | Action number | | Solution | Value performance indicator 1 | Performance indicator 1 | Value performance indicator 2 | Performance indicator 2 |
| 1. Key and generic interoperability enablers | ISA | | 2016.19 | e-TrustEx | 200 | Public administrations in the 28 Member States using this solution | 8,800,000 | Number of documents exchanged between the EC, the Council, the Member States, and companies |
| 1. Key and generic interoperability enablers | ISA | | 2016.29 | CPSV-AP | 12 | Public administrations in Member States using this solution, including cross-border catalogues | 2,511 | Number of downloads on Joinup (up until March 2019) |
| 2. Semantic interoperability | ISA | | 2016.07 | ADMS | 9 | Public administrations (Member States, EC), company – using this solution | 4,700 | Number of interoperability solutions on Joinup described using ADMS |
| 2. Semantic interoperability | ISA | | 2016.07 | Core vocabularies | 11 | Public administrations (in the Member States and EC) using this solution |  |  |
| 2. Semantic interoperability | ISA | | 2016.07 | DCAT-AP | 29 | Public administrations (in the Member States and EC) and their data portals, associations, universities – using this solution |  |  |
| 2. Semantic interoperability | ISA | | 2016.07 | VocBench3 | 14 | Number of public administrations (in the Member States and EC), universities, institutes, international organisations using the previous versions of VocBench. |  |  |
| 2. Semantic interoperability | ISA2 | | 2016.16 | PMKI Core data model for multilingual taxonomies/ terminologies | 2 journal articles; 5 international conference proceedings | Instances of inclusion in academic journals and international conferences |  |  |
| 2. Semantic interoperability | ISA2 | | 2016.16 | Semantic links | 4 | Collaborations with European public administrations to establish semantic interoperability between national language resources and EuroVoc (3 Member States and the EP) |  |  |
| 3. Access to data/data sharing/open data | ISA2 | | 2016.06 | ESS Service Catalogue | 10 | Number of registered contributors | 13 | Number of statistical organisations listed |
| 4. Geospatial solutions | ISA | | 2016.10 | Re3gistry | 13 | Public administrations (Member States, EC) using this solution |  |  |
| 5. eProcurement/ eInvoicing - Supporting instruments | ISA | | 2016.05 | Open e-Prior | 70 | Public administrations (EC and other EU institutions) using this solution | 306 | Number of suppliers connected via the web portal |
| 5. eProcurement/ eInvoicing - Supporting instruments | ISA | | 2016.05 | eCertis | 44 | Public administrations (Member States, EU institutions), contracting authorities, companies – using this solution |  |  |
| 5. eProcurement/ eInvoicing - Supporting instruments | ISA | | 2016.05 | ESPD | 195,652 | Number of page visits in January 2019  (around 4 million visits until April 2019) | 23 | Number of Member States implementing the ESPD data model; |
| 6. Decision making and legislation - Supporting instruments | ISA | | 2016.23 | Digital screening mention in Tool #27 of the Better Regulation Toolbox: The digital economy and society & ICT issues | 280 | Number of Commission proposals screened for ICT impacts since 2015 |  |  |
| 6. Decision making and legislation - Supporting instruments | ISA | | 2016.23 | ICT Impact Assessment Guidelines | 13 | Instances of use since 2014 |  |  |
| 6. Decision making and legislation - Supporting instruments | ISA2 | | 2017.03 | REFIT Platform IT Tool | Not yet operational. |  |  |  |
| 6. Decision making and legislation - Supporting instruments | ISA2 | | 2017.04 | RegDel | 144,587 | Number of page views since the launch up until 1 February 2019 | 2,202 | Number of active subscriptions |
| 7. EU Policies - Supporting instruments | ISA | | 2016.14 | Online Collection Software to support European Citizens' Initiative | 38 | Number of ECIs launched using the Online Collection Software | 1 | Public administration in the Member States |
| 7. EU Policies - Supporting instruments | ISA | | 2016.14 | European Parliament Crypto Tool | 28 | Number of Member States using the solution |  |  |
| 8. Supporting instruments for public administrations | ISA | | 2016.20 | Joinup | 13,440 | Professionals working in the field of eGovernment registered on Joinup | 2,815 | Solutions created within the 109 Collections (thematic collaborative spaces) on Joinup. |
| 8. Supporting instruments for public administrations | ISA | | 2016.21 | NIFO | 193 | Number of users for Q3 2018, according to ISA2 Dashboard - Effectiveness Indicator (approximately equal to the number of members of the NIFO community on Joinup) | 34 | Number of countries covered (32 countries covered by NIFO factsheets and 34 by the Interoperability State of Play reports) |
| 8. Supporting instruments for public administrations | ISA | | 2016.32 | EIRA and CarTool | 8 | Number of public administrations in Member States deploying the solution | 1,364 | Number of downloads of EIRA on Joinup up to February 2019 |
| 8. Supporting instruments for public administrations | ISA | | 2016.35 | EUSurvey | 17,443 | Number of surveys created (sum for 2016, 2017, Q1 and Q3 of 2018), based on the ISA2 solutions webpage and the 2018 Rolling Work Programme. |  |  |
| 8. Supporting instruments for public administrations | ISA | | 2016.37 | IMAPS | 17 | Trans-European Systems assessed with IMM | 68 (in 2017); 51 (in 2018) | Public services benchmarked (Q3 2017) |
| 8. Supporting instruments for public administrations | ISA2 | | 2017.01 | Study on Standard-based Archival Data Management, Exchange, and Publication | 80 | Unique visitors/downloads since the publication (2018) |  |  |
| 8. Supporting instruments for public administrations | ISA2 | | 2017.01 | Assessment tool offering support for the selection of IT solutions for archives management | 58 | Unique visitors/downloads since the publication (2018) |  |  |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Communication activities

Table 13: Events funded by ISA2 between 2016 and 2018

|  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Year |  | ISA2 workshops/conferences | Location | Total participants onsite | Total remote participants | Number of international participants | Number of Member States represented | Number of EEA and acceding countries represented | Number of EC officials (non-DIGIT) |
| 2016 |  | From ISA to ISA2 | Brussels, Belgium | 300 | - | - | - | - | - |
| 2016 |  | SEMIC2016\* | Rome, Italy | 206 | - | - | 25 | 3 | - |
| 2017 |  | Workshop on the EIF | Thessaloniki | N/A | - | - | - | - | - |
| 2017 |  | Sharing and reuse Conference | Lisbon, Portugal | 220 | 810 | 204 | 21 | 0 |  |
| 2017 |  | SEMIC2017 | Valletta, Malta | 224 | 609 | 195 | 18 | 2 |  |
| 2018 |  | Open PM2 Conference (partially funded by ISA2) | Brussels, Belgium | 538 | 1,800 (connections to web streaming) | - | - | - | - |
| 2018 |  | SEMIC 2018 | Sofia, Bulgaria | 220 | 701 | 204 | 19 | 2 |  |
| 2018 |  | ISA2CONF18 | Brussels, Belgium | 325 | - | 249 | 26 | 7 | 120 |
| 2018 |  | Workshop organised as part of the European Week of Regions and Cities 2018 | Brussels, Belgium | 98 | - | - | - | - | - |
| 2018 |  | Workshop organised as part of the Inspire Conference | Antwerp, Belgium | 95 | - | - | - | - | - |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Table 14: Events in which ISA2 representatives played an active role

|  |  |  |  |
| --- | --- | --- | --- |
| Year | Events | Location | Total participants |
| 2016 | Ljubljana ICT Procurement workshop | Ljubljana, Slovenia | N/A |
| 2016 | Nordic Digital Day | Tallinn, Estonia | N/A |
| 2016 | Open Source Summit Paris | Paris, France | 180 |
| 2016 | Digital Stakeholders Forum | Brussels, Belgium | 150 |
| 2016 | Digitec16 | Brussels, Belgium | 500 |
| 2016 | Manage IT 2016 | Antwerp, Belgium | N/A |
| 2017 | Open Belgium | Antwerp, Belgium | 150 |
| 2017 | SG IT Day | Brussels, Belgium | 200 |
| 2017 | Digital Day Rome | Rome, Italy | 600 |
| 2017 | ECI Day 2017 | Brussels, Belgium | 150 |
| 2017 | ICT Spring Luxembourg 2017 | Luxembourg, Luxembourg | 5,000 |
| 2017 | Conference Krems | Krems, Austria | N/A |
| 2017 | Digital Assembly 2017 | Valletta, Malta | 5,000 |
| 2017 | Semantics Conference 2017 | Amsterdam, The Netherlands | 200 |
| 2017 | ICA Conference 2017 |  | N/A |
| 2017 | Inspire Conference 2017 | Strasbourg, France | 3,000 |
| 2017 | Data for Policy | London, UK | 200 |
| 2017 | DK Architecture Conference | Copenhagen, Denmark | N/A |
| 2017 | Jornada sobre Interoperabilidad y Archivo Electronico | Madrid, Spain | 180 |
| 2017 | Digitalisseringsmessen 17 | Odense, Denmark | 350 |
| 2017 | Egov Conference Tallinn, 2017 | Tallinn, Estonia | 200 |
| 2017 | 3rd Annual Public Sector Transformation Conference | Brussels, Belgium | N/A |
| 2017 | 3rd ELRC Conference | Brussels, Belgium | N/A |
| 2017 | ICT Proposers Day 2017 | Budapest, Hungary | 150 |
| 2017 | Metaforum 2017 | Brussels, Belgium | N/A |
| 2017 | Informatika v Javni Upravi | Brdo, Slovenia | 175 |
| 2017 | Paris Open Source Summit 2017 | Paris, France | 150 |
| 2017 | CEF Conference | Brussels, Belgium | 150 |
| 2017 | eDemocracy Conference | Athens, Greece | 100 |
| 2018 | Connected Smart Cities Conference | Brussels, Belgium | 500 |
| 2018 | Flosscon | Brussels, Belgium | 200 |
| 2018 | GDPR Conference | Berlin, Germany | 150 |
| 2018 | Digital Czech Republic | Prague, Czech Republic | 250 |
| 2018 | Interop Summit 2018 | Brussels, Belgium | 150 |
| 2018 | Open Belgium | Louvain La Neuve, Belgium | 250 |
| 2018 | RDA Berlin | Berlin, Germany | 200 |
| 2018 | IESA 2018 | Berlin, Germany | 250 |
| 2018 | CNIS2018 | Madrid, Spain | 300 |
| 2018 | Digital Day | Rome, Italy | 300 |
| 2018 | CEEE Gov Days 2018 | Budapest, Hungary | 150 |
| 2018 | German Federal Level Conference | Berlin, Germany | 120 |
| 2018 | Good Governance Conference 2018 | Brussels, Belgium | 650 |
| 2018 | Civil Society Days 2018 | Brussels, Belgium | 200 |
| 2018 | Language Technology Industry Summit | Brussels, Belgium | 200 |
| 2018 | Egov Conference Tallinn, 2018 | Tallinn, Estonia | 300 |
| 2018 | Conference Supervisory reporting for the Digital Age | Brussels, Belgium | 150 |
| 2018 | Digital Assembly 2018 | Helsinki, Finland | 125 |
| 2018 | ICA Conference 2018 | Sofia, Bulgaria | 5,000 |
| 2018 | ICT implications presentation in Vienna | Vienna, Austria | 120 |
| 2018 | Inspire Conference 2018 | Antwerp, Belgium | 900 |
| 2018 | TOOP Conference | Vienna, Austria | 150 |
| 2018 | eGov High Level Conference 2018 | Vienna, Austria | N/A |
| 2018 | Infofest Montenegro | Podgorica, Montenegro | 150 |
| 2018 | European Week of Regions and Cities 2018 | Brussels, Belgium | 6,000 |
| 2018 | GovTech Summit 2018 | Paris, France | 3,000 |
| 2018 | Digitec18 | Brussels, Belgium | 900 |
| 2018 | ICT Vienna | Vienna, Austria | 5,000 |
| 2018 | Paris Open Source Summit 2018 | Paris, France | 200 |
| 2018 | Symposium on Digital Transformation of the public sector 2018 | Belgium | 200 |
| 2018 | Webinar on Government Transformation: ‘How co-creation will shape the future of value creation in the public sector’ | Belgium | 200 |
| 2018 | 1st CEF eTranslation Conference | Brussels, Belgium | N/A |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Alignment with principles spelled out in Article 4(b) of the ISA2 Decision

Table 15: Contribution of ISA2 actions to the principles listed in Art. 4(b) of the ISA2 Decision

|  |  |
| --- | --- |
| Art. 4(b) Principles | Relevant evidence |
| Subsidiarity and proportionality | ISA2 aims to contribute to a holistic interoperability landscape by creating solutions at EU level. The adoption of solutions by European public administrations is voluntary. |
| User-centricity | As part of the process of proposing actions to be included in the rolling work programme, all proposed actions must take user-centricity into account and describe the ways in which user input is integrated into the action and the development of solutions. |
| Inclusion and accessibility | ISA2 uses a number of channels to disseminate information about the actions and the solutions developed. These include: i) the ISA2 website, which contains information on each action and solution, official documents, infographics, videos, presentations, training material, and publications; ii) the Joinup platform to share the solutions developed under ISA2; iii) events organised by ISA2 in various Member States where promotional material is distributed; and iv) events with ISA2 participation in various Member States where promotional material is distributed. In addition, solutions take into account potential barriers to accessibility and inclusion, detailing such barriers where relevant and providing solutions. An example is the ‘EU CAPTCHA’ [140](#footnote141)  action, which aims to address the issues of accessibility of CAPTCHA tests by exploring alternative solutions to improve the user experience. |
| Delivery of public services in such a way as to prevent digital divide | As part of the process of proposing actions to be included in the rolling work programme, all the proposed actions must specify their expected contribution to the EU’s higher political priorities, such as the Digital Single Market. Where applicable, the rolling work programme details how specific actions contribute to the third pillar of the Digital Single Market Strategy, which also includes support for an inclusive digital society. |
| Security, respect for privacy and data protection | Where appropriate, the descriptions of actions included in the rolling work programme must specify the measures taken to ensure security, respect for privacy and data protection. |
| Multilingualism | Examples of ISA2 actions that produce tools and frameworks that contribute to multilingualism are: ‘SEMIC’; ‘PMKI’; ‘Development of an open data service, support and training package in the area of linked open data, data visualisation and persistent identification’, and ‘EUSurvey’. |
| Administrative simplification and modernisation | Administrative simplification and modernisation are taken into consideration in the rolling work programme of ISA2 under the section ‘main impact list’. The detailed list of expected impacts for each action indicates the extent to which the proposed actions contribute to this aspect. |
| Transparency | Various channels ensure the transparency of ISA2 actions. These include: i) the rolling work programme that provides an overview of each action’s objectives, planned and developed solutions, expected impacts, and budget; ii) the ISA2 Dashboard that provides quarterly updates on the efficiency and effectiveness of actions, in terms of costs, earned value management, effectiveness indicators and targets; iii) a specific ISA2 webpage with information on the level of take-up of solutions; and iv) the Joinup platform where developed solutions can be accessed. |
| Preservation of information | The ‘Library’ section of the ISA2 website preserves information about ISA2 in various formats including presentations, videos, publications, infographics, posters, leaflets, training course materials, press releases, articles, ISA2 work programmes, speeches, and official documents. |
| Openness | The publication of ISA2 solutions on Joinup ensures that solutions are openly available for (potential) users. |
| Re-usability and avoidance of duplication | As part of the proposal process for actions to be included in the rolling work programme, the descriptions of proposed actions specify (i) the extent to which the action proposed reuses other readily available solutions and (ii) the reusability of the action outputs. |
| Technological neutrality, solutions which, insofar as possible, are future-proof, and adaptability | The rolling work programme’s section on ‘Contribution to the interoperability landscape’ describes, among others, the contributions made by actions to the European Interoperability Framework, aspects of which include technological neutrality, future-proof solutions and adaptability. As such, this principle is already taken into account in the proposal phase. |
| Effectiveness and efficiency | The ISA2 Dashboard provides quarterly data on the efficiency and effectiveness of actions. The historic data can be accessed together with the most recent data available. |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Efficiency

EQ4: To what extent has the programme been cost-effective?

Table 16: Labour costs per hour in euro, services in 2017

|  |  |
| --- | --- |
| Country | €/h |
| Belgium | 41.1 |
| Bulgaria | 5.2 |
| Czech Republic | 11.5 |
| Denmark | 43.8 |
| Germany | 31.5 |
| Estonia | 12.5 |
| Ireland | 28.9 |
| Greece | 14.3 |
| Spain | 20.0 |
| France | 36.4 |
| Croatia | 11.4 |
| Italy | 27.4 |
| Cyprus | 16.5 |
| Latvia | 8.8 |
| Lithuania | 8.5 |
| Luxembourg | 40.6 |
| Hungary | 9.3 |
| Malta | 13.5 |
| Netherlands | 32.9 |
| Austria | 33.5 |
| Poland | 9.3 |
| Portugal | 14.5 |
| Romania | 6.7 |
| Slovenia | 17.5 |
| Slovakia | 11.4 |
| Finland | 32.0 |
| Sweden | 40.9 |
| United Kingdom | 25.2 |

Note: The table details the hourly labour costs for the service sector at Member State level. Such costs include wage and non-wage costs net of subsidies. They do not include vocational training costs or other expenditures such as recruitment costs, spending on working clothes, etc.
[141](#footnote142)

Source: Eurostat.

  

Coherence

EQ6: To what extent do the ISA² actions form part of a ‘holistic’ approach within the framework of the programme? (Internal coherence)

Table 17: Number of links between sampled actions

|  |  |  |
| --- | --- | --- |
| Actions | Number of other ISA² solutions that are used by the action | Number of other ISA² actions that use the solutions of the action |
| 1. Key and generic interoperability enablers | | |
| Trusted Exchange Platform (e-TrustEx) | 4 | 5 |
| Catalogue of Services | 2 | 3 |
| 2. Semantic Interoperability | | |
| Public Multilingual Knowledge Management Infrastructure for the Digital Single Market (PMKI) | 1 | 0 |
| SEMIC: Promoting Semantic Interoperability Amongst the European Union Member States | 5 | 17 |
| 3. Access to data/data sharing/open data | | |
| Big Data for Public Administrations | 5 | 0 |
| Sharing Statistical Production and Dissemination Services and Solutions in the European Statistical System | 3 | 0 |
| Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification | 1 | 0 |
| 4. Geospatial Solutions | | |
| European Location Interoperability Solutions for e-Government (ELISE) | 6 | 1\* |
| 5. eProcurement/eInvoicing-Supporting instruments | | |
| European Public Procurement Interoperability Initiative | 6\*\* | 4 |
| 6. Decision making and legislation-Supporting instruments | | |
| ICT Implications of EU Legislation | 6 | 0 |
| REFIT Platform | 0 | 2 |
| Inter-Institutional Register of Delegated Acts | 1 | 1 |
| 7. EU Policies-Supporting instruments | | |
| European Citizens' Initiatives and European Parliament Elections | 0 | 0 |
| 8. Supporting instruments for public administrations | | |
| Joinup - European Collaborative Platform and Catalogue | 4 | 14 |
| National Interoperability Framework Observatory | 7 | 0 |
| European Interoperability Architecture (EIA) | 7 | 9 |
| EUSurvey | 0 | 0 |
| Interoperability Maturity Model | 9 | 2 |
| Standard-Based Archival Data Management, Exchange and Publication | 0 | 1 |
| 9. Accompanying measures | | |
| Raising Interoperability Awareness - Communication Activities | 1 | 0 |

\*Note: For the ‘ELISE’ action, further information was provided by the action owner regarding the number of other ISA² actions that use the solutions of ELISE.

\*\*Note: For the ‘European Public Procurement Interoperability Initiative’ action, further information was provided by the action owner regarding the number of other ISA² solutions that are used by the action.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

EQ7: To what extent is the ISA² programme coherent with other EU interventions that have similar objectives and with global initiatives in the same field? (External coherence)

Table 18: Number of links between the sample actions and other EU programmes / policies / initiatives

|  |  |  |
| --- | --- | --- |
| Actions | Number of other EU programmes / policies / initiatives that the action relies on | Number of other EU programmes / policies / initiatives that use the solution(s) provided by the action |
| 1. Key and generic interoperability enablers | | |
| Trusted Exchange Platform (e-TrustEx) | 5 | 7\* |
| Catalogue of Services | 4 | 2 |
| 2. Semantic Interoperability | | |
| Public Multilingual Knowledge Management Infrastructure for the Digital Single Market (PMKI) | 1 | 1 |
| SEMIC: Promoting Semantic Interoperability Amongst the European Union Member States | 6 | 2 |
| 3. Access to data/data sharing/open data | | |
| Big Data for Public Administrations | 0 | 2 |
| Sharing Statistical Production and Dissemination Services and Solutions in the European Statistical System | 12 | 2 |
| Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification | 3 | 1 |
| 4. Geospatial Solutions | | |
| European Location Interoperability Solutions for e-Government (ELISE) | 3 | 3\*\* |
| 5. eProcurement/e-invoicing-Supporting instruments | | |
| European Public Procurement Interoperability Initiative | 7 | 0 |
| 6. Decision making and legislation-Supporting instruments | | |
| ICT Implications of EU Legislation | 3 | 2 |
| REFIT Platform | 0 | 1 |
| Inter-Institutional Register of Delegated Acts | 0 | 2 |
| 7. EU Policies-Supporting instruments | | |
| European Citizens' Initiatives and European Parliament Elections | 2 | 0 |
| 8. Supporting instruments for public administrations | | |
| Joinup - European Collaborative Platform and Catalogue | 1 | 1 |
| National Interoperability Framework Observatory | 0 | 1 |
| European Interoperability Architecture (EIA) | 3 | 1 |
| EUSurvey | 1 | 0 |
| Interoperability Maturity Model | 0 | 1 |
| Standard-Based Archival Data Management, Exchange and Publication | 0 | 4 |
| 9. Accompanying measures | | |
| Raising Interoperability Awareness - Communication Activities | 1 | 1 |

\*Note: Additional information was retrieved from the 2019 rolling work programme.

\*\*Note: Additional information was provided by the actions owners of the ‘ELISE’ action.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Alignment with the DSM Strategy

Table 19: ISA2 contribution to the DSM Strategy

Contribution to Pillar I
[142](#footnote143)
 of the DSM Strategy

|  |  |  |
| --- | --- | --- |
| Action number | Action name | Contribution to Pillar I: Better access for consumers and businesses to online goods and services across Europe |
| 2016.29 | Catalogue of Services | Obliges Member States to create Points of Single Contact with combined information for business setup and other citizen formalities. |
| 2016.07 | SEMIC: Promoting Semantic Interoperability Amongst the European Union Member States | Semantic interoperability is a prerequisite for enacting most levels of systems’ interoperability, including the once-only principle and ensuring open data. |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Contribution to Pillar II
[143](#footnote144)
 of the DSM Strategy

|  |  |  |
| --- | --- | --- |
| Action number | Action name | Contribution to Pillar II: Creating the right conditions for digital networks and services to flourish |
| 2016.06 | Sharing Statistical Production and Dissemination Services and Solutions in the European Statistical System | A reusable solution for the dissemination of statistics for use by any administration to reduce the cost of dissemination and improve the delivery of data to public consumers |
| 2016.05 | European Public Procurement Interoperability Initiative | The activities supported by this proposal will help to further create or improve standards regarding public procurement and encourage Member States to shift towards full eProcurement |
| 2016.20 | Joinup-European Collaborative Platform and Catalogue | Joinup creates a central platform for observatory functionalities, collaborative features, and interoperability solutions |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Contribution to Pillar III
[144](#footnote145)
 of the DSM Strategy

|  |  |  |
| --- | --- | --- |
| Action number | Action name | Contribution to Pillar III: Maximising the growth potential of our European Digital Economy |
| 2016.19 | e-TrustEx | Acts as a broker in the exchange of data and documents to foster greater interaction across heterogeneous systems |
| 2016.16 | Public Multilingual Knowledge Management Infrastructure for the Digital Single Market | The creation of a Public Multilingual Knowledge Infrastructure will support EU public administrations in creating services that can be accessible and shareable independently from the language actually used, as well as the SMEs to sell goods and service cross-border in a digital single market. |
| 2016.15 | FISMA: Financial Data Standardization | Looks to work on ICT standardisation to reduce administrative costs of legacy and data systems in the financial sector. |
| 2016.18 | Development of an Open Data Service, Support, and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification | Open government data is a core asset for the knowledge-based economy, since its reuse is a basis for innovative information products and services as it is a key enabler for transparency, evidence-based decision-making and a broader participation in the political discourse. This package will enable administrations to enhance their data visualisation capacity, to further open up their data as well as to increase data quality and interoperability in view of better data reusability. |
| 2016.03 | Big Data for Public Administrations | This action addresses the need to provide the right framework conditions for a single market for big data and cloud computing as a means for helping to accelerate the transition towards a data-driven economy. |
| 2016.10 | European Location Interoperability Solutions for e-Government (ELISE) | This action contributes to the interoperability landscape by ensuring that the ‘location’ dimension has an impact, adds value and is appropriately addressed within solutions across borders and sectors. |
| 2016.23 | Legal Interoperability | The action has a horizontal value as it can be used for the law-making/evaluation of every EU policy. Special contribution is indirectly made to the DSM, as the more the action is assessing EU legislations the more digital and interoperable they become. |
| 2016.32 | European Interoperability Architecture (EIA) | The EIA looks to maintain the EIRA, an interoperability reference that is key to the once-only principle |
| 2016.21 | National Interoperability Framework Observatory | The EIF and the EIF Implementation Strategy foster interoperability and contribute to the DSM. By contributing to the implementation of the EIF, this action is relevant for the DSM. |
| 2016.35 | EUSurvey | By offering an easy means of collecting opinions and information between heterogeneous parties, the EUSurvey considerably facilitates the organisation and consolidation of any types of 'feedback-based' decision |
| 2017.01 | Standard-Based Archival Data Management, Exchange and Publication | By clarifying and supporting technical standards for archival management, it will ensure better access to digital goods and services and by providing access to Commission archives in Open Data format, it will generate value, allowing the reuse of this information producing new products and services. |
| 2016.30 | Raising Interoperability Awareness-Communication Activities | Communicating the ISA² programme and its results reinforces the programmes contributions to the DSM, the eGovernment action plan, the EIS and the new EIF, but also the Open Data Initiative of the European Union and INSPIRE, among other. |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Alignment with the Tallinn Declaration on eGovernment

Box 10: Alignment between ISA2 and the Tallinn Declaration on eGovernment

|  |
| --- |
| The Tallinn Declaration is a statement of intents in which the 32 signatory countries made a commitment to move towards a digital government following five principles:  1.Digital-by-default, inclusiveness and accessibility  2.Once-only  3.Trustworthiness and security  4.Openness and transparency  5.Interoperability by default  These principles inform the policy action lines, together with the additional line ‘Horizontal enabling policy steps’. Out of these five principles, principles #2, #4 and #5 are specifically reflected in the ISA2 programme. In particular:  ·Principle #2, ‘once-only’, is reflected in actions such as ‘SEMIC’, ‘European Interoperability Architecture’, and ‘European public procurement interoperability initiative’ that support the implementation of the principle.  ·Principle #4, openness and transparency, is reflected in the attention paid to Linked Open Data in actions like ‘SEMIC’, ‘PMKI’, ‘ELISE’, ‘Standard-Based Archival Data Management, Exchange and Publication’ of ISA2, even though in the Declaration the scope is wider and is specifically addressed to the openness of public institutions; and  ·Principle #5, interoperability by default, is directly linked to the objectives of ISA2 programme.  Furthermore, the policy action line ‘Horizontal enabling policy steps’ involves some aspects that are characteristic of the ISA2 programme, such as the digital transformation of public administrations and integration of digital considerations in policy initiatives. Among the many proposals of the Declaration, a few are directly coherent with ISA2 objectives:  ·to implement the European Interoperability Framework and the Interoperability Action Plan (policy line 5);  ·to consider strengthening the requirements for use of open source solutions and standards when (re)building of ICT systems and solutions takes place with EU funding (policy line 5);  ·to fully integrate digital considerations into existing and future policy and regulatory initiatives (policy line 6);  ·to prepare proposals on the future (post 2020) and sustainability of existing EU-level cross-border digital service infrastructures and building blocks (policy line 6); and  ·to convene and support the work of groups of interested countries and other parties to exchange practices and develop reference guidelines and standards for taking emerging ICT into use in the public administration (policy line 6).  From this standpoint, there is no contradiction between the Tallinn Declaration and the ISA2 programme. Although their scope and aims differ, they proceed in the same direction and share common intents. There may be synergies in the above-mentioned points, but they can also turn into overlaps and duplications, or even worse, misalignments, if there is no strong coordination.  While the ISA2 programme is multi-national, the Declaration also affects each country individually. Therefore, initiatives and projects may be promoted separately by individual countries to fulfil the commitments agreed upon by the Member States. The Declaration also includes specific calls to EU institutions to enhance interoperability within the EU framework. Overall, this document stresses the need for more effort, both at EU institutions-level and at Member State-level, to ensure citizen-centric eGovernment and interoperability.  Finally, as the annex of the Tallinn Declaration highlights the importance of principles like the ‘Protection of personal data and privacy’ and ‘Incentives for digital service use’, better coherence would be ensured if the ISA2 programme began to include some of these principles in its actions. |

EU added value

EQ8: What is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels?

Table 20: Number of participants in the ISA2 Coordination Group meetings (per Member State)

|  |  |  |  |
| --- | --- | --- | --- |
| Member State | Coordination Group Meeting 13/10/2017 Presence | Coordination Group Meeting 19/10/2016 Presence | Coordination Group Meeting 24/10/2018 Presence |
| Belgium | 1 | 1 | 1 |
| Bulgaria | 1 | 2 | 0 |
| Czech Republic | 2 | 2 | 1 |
| Denmark | 1 | 1 | 2 |
| Germany | 2 | 2 | 3 |
| Estonia | 2 | 2 | 1 |
| Ireland | 2 | 1 | 1 |
| Spain | 2 | 3 | 0 |
| France | 1 | 1 | 0 |
| Croatia | 2 | 2 | 1 |
| Italy | 1 | 1 | 1 |
| Cyprus | 2 | 2 | 1 |
| Latvia | 2 | 2 | 1 |
| Lithuania | 1 | 1 | 1 |
| Luxembourg | 2 | 2 | 1 |
| Hungary | 2 | 2 | 1 |
| Malta | 3 | 2 | 1 |
| Netherlands | 3 | 2 | 1 |
| Austria | 2 | 2 | 1 |
| Poland | 3 | 2 | 0 |
| Portugal | 2 | 2 | 0 |
| Romania | 2 | 2 | 0 |
| Slovenia | 1 | 1 | 0 |
| Slovakia | 2 | 2 | 1 |
| Finland | 2 | 2 | 1 |
| Sweden | 2 | 2 | 2 |
| United Kingdom | 1 | 2 | 1 |
| Norway | 1 | 1 | 1 |
| Iceland | 1 | 0 | 0 |
| Montenegro | 0 | 0 | 1 |
| Greece | 0 | 0 | 0 |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Table 21: Number of participants in the ISA2 Committee meetings (per Member State)

|  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Member State | Committee Meeting 02/03/2016 Presence | Committee Meeting 09/06/2016 Presence | Committee Meeting 19/01/17 Presence | Committee Meeting 20/06/2018 Presence | Committee Meeting 24/01/2018 Presence |
| Belgium | 2 | 1 | 2 | 1 | 1 |
| Bulgaria | 3 | 2 | 2 | 1 | 2 |
| Czech Republic | 3 | 3 | 2 | 1 | 3 |
| Denmark | 1 | 2 | 3 | 1 | 2 |
| Germany | 3 | 4 | 3 | 1 | 3 |
| Estonia | 5 | 2 | 2 | 1 | 1 |
| Ireland | 1 | 2 | 3 | 1 | 2 |
| Spain | 3 | 3 | 3 | 1 | 3 |
| France | 1 | 1 | 1 | 1 | 1 |
| Croatia | 2 | 2 | 2 | 1 | 2 |
| Italy | 2 | 2 | 2 | 1 | 2 |
| Cyprus | 2 | 2 | 2 | 1 | 2 |
| Latvia | 2 | 3 | 2 | 1 | 2 |
| Lithuania | 3 | 2 | 2 | 1 | 2 |
| Luxembourg | 3 | 3 | 3 | 1 | 3 |
| Hungary | 2 | 2 | 2 | 1 | 2 |
| Malta | 2 | 2 | 2 | 1 | 2 |
| Netherlands | 3 | 2 | 2 | 1 | 2 |
| Austria | 3 | 2 | 2 | 1 | 3 |
| Poland | 2 | 2 | 2 | 1 | 2 |
| Portugal | 2 | 1 | 2 | 1 | 2 |
| Romania | 2 | 2 | 2 | 1 | 2 |
| Slovenia | 2 | 2 | 2 | 1 | 2 |
| Slovakia | 2 | 2 | 2 | 1 | 4 |
| Finland | 3 | 2 | 3 | 1 | 2 |
| Sweden | 1 | 1 | 1 | 1 | 1 |
| United Kingdom | 3 | 2 | 2 | 1 | 1 |
| Norway | 2 | 0 | 3 | 1 | 3 |
| Iceland | 0 | 0 | 1 | 1 | 1 |
| Montenegro | 0 | 0 | 1 | 2 | 1 |
| Greece | 0 | 0 | 0 | 1 | 3 |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

  

Table 22: Contribution of sampled actions to cross-border interoperability

|  |  |  |  |
| --- | --- | --- | --- |
| Package | Action number | Action Name | Contribution to cross-border interoperability |
| 1. Key and generic interoperability enablers | 2016.19 | Trusted Exchange Platform (e-TrustEx) | The e-TrustEx platform is currently used as a key element of pan-European messaging infrastructures for projects such as: e-PRIOR (DIGIT), DECIDE (SG), EDMA (COMP), eJustice Portal (JUST), OPOCE (OP), EU-CEG (DG SANTE), ESDEN (ESTAT). Around 200 public institutions across the 28 Member States are in scope of these projects (such as national parliaments and permanent representations). |
| 1. Key and generic interoperability enablers | 2016.29 | Catalogue of Services | The CPSV-AP is already being used by public administrations in Belgium, Italy, Finland, and Estonia to create a cross-border federated catalogue of public services. |
| 2. Semantic interoperability | 2016.07 | SEMIC: Promoting Semantic Interoperability Amongst the European Union Member States | The proposal will facilitate the cross-border interoperability thanks to its inherent support for multilingualism. Further cross-border interoperability improvements can be expected through the alignment of the generic EuroVoc thesaurus that covers the EU policy domains with specialised EU and national controlled vocabularies through VocBench. VocBench is already used by public administrations in France, Italy, the Netherlands and Scotland and interest has been expressed from public administrations in Belgium and Slovenia. |
| 2. Semantic interoperability | 2016.16 | Public Multilingual Knowledge Management Infrastructure for the Digital Single Market | The objective of this action is to support enterprises and particularly the language technology industry with the implementation of the necessary multilingual tools and features in order to improve cross-border accessibility of e-Commerce solutions. The outputs in the form of semantic links developed so far contribute to the cross-border service interoperability from a semantic perspective. |
| 3. Access to data/data sharing/open data | 2016.03 | Big Data for Public Administrations | In 2017, a long list of requirements has been collected from different Member States to understand their needs in the area of (Big) data analytics for policymaking, especially with regard to analytics use cases and infrastructure needs. The needs have been collected through a consultation of the ISA network, through the creation of a working group. Member States on board so far are the Czech Republic, Estonia, Hungary, Malta, Norway, Slovenia, Spain, and Portugal. The working group has shown significant interest in the action and its outputs. Additionally, the action has been presented to the ESS Big Data Task Force: the representatives have highlighted their interest in the action and shown availability to (re-)use its outcomes. |
| 3. Access to data/data sharing/open data | 2016.06 | Sharing Statistical Production and Dissemination Services and Solutions in the European Statistical System | The development of statistical services includes a broad international community. In the ESS, 14 Member States are actively involved in a Task Force and a consortium of six Member States (FR, PT, UK, LT, DE, SI) has been set up to provide input and take part in the development of the guidelines for sharing of statistical services and to implement the reuse of developed solutions and services with the European Commission.  In the architecture domain, the ESS reference architecture in its current state has been adopted by the 28 NSIs CIOs and Heads of Methodology. Its upgrading towards greater interoperability through more standards and deeper architectural guidance is done in collaboration with an ESS EA Board involving five Member States. |
| 3. Access to data/data sharing/open data | 2016.18 | Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification | Member States can consult and reuse the project outputs (knowledge base, trainings, the description of the tools and projects included in the catalogue of data visualisation tools). |
| 4. Geospatial solutions | 2016.10 | European Location Interoperability Solutions for e-Government (ELISE) | Road safety data-exchange solutions piloted and implemented in Norway and Sweden are being rolled-out to five other Member States using CEF funds, with more rollouts planned. INSPIRE Registry services have 450k accesses per quarter.\* |
| 5. eProcurement/ eInvoicing - Supporting instruments | 2016.05 | European Public Procurement Interoperability Initiative | ePrior is used by several EU bodies and some components are used by the Belgian administration. In particular, in the EU Bodies context, economic operators using the ePrior system are from various EU countries. For eCertis: Roughly 25 services in the EU are retrieving data from eCertis, using the CEF eProc DSI. Other services will follow. For ESPD services: Private and public entities from roughly 17 Member States have participated in the CEF eProc DSI Others are using Structural Funds to implement an ESPD service. They all have implemented or are currently implementing an ESPD service (incl. NO) using the ESPD data model or the open source code developed under the ISA2 project. In more than 20 Member States, there is already at least one ESPD service available. |
| 6. Decision making and legislation - Supporting instruments | 2017.03 | [REFIT Platform](https://ec.europa.eu/isa2/dashboard/node/92618) | The REFIT Platform consists of two Commission expert groups: a Government group in which all Member States are represented and a Stakeholder group with representatives of businesses, social partners, civil society organisations in various Member States, the Economic and Social Committee and the European Committee of the Regions. |
| 7. EU Policies - Supporting instruments | 2016.14 | European Citizens’ Initiatives and European Parliament Elections | ECI-OCS serves the citizens and public administrations in all Member States as it facilitates the verification of the statements of support for legislation. The European Parliament Crypto tool is useful to the public administrations of all the Member States |
| 8. Supporting instruments for public administrations | 2016.20 | [Joinup – European Collaborative Platform and Catalogue](https://ec.europa.eu/isa2/dashboard/node/645) | The end-users of Joinup are from different EU Member States and countries outside the EU (USA, Canada, New Zealand). In addition, several national repositories (NL, ES, EL, SL, BE) are stored on Joinup, making their national solutions available for re-use. |
| 8. Supporting instruments for public administrations | 2016.21 | National Interoperability Framework Observatory | The outputs of NIFO have already been reused by various Member States. The eGovernment factsheets are considered as a reference. The state of play reports on interoperability in Europe and the NIFO factsheets served as a source of input in the revision of the EIF and the IAP. |
| 8. Supporting instruments for public administrations | 2016.32 | European Interoperability Architecture (EIA) | EIRA has been deployed in EE, NL, DK, ES, CZ and PL. |
| 8. Supporting instruments for public administrations | 2016.35 | EUSurvey | EUSurvey is available in 23 EU languages, facilitating cross-border interoperability. In 2017, more than 7,800 surveys have been created with the tool, resulting in more than 2.7 million contributions. |
| 8. Supporting instruments for public administrations | 2017.01 | Standard-Based Archival Data Management, Exchange and Publication | The action addresses the semantic interoperability issue of how to describe electronic archives by means of the identification of existing standards for digital archives, facilitating the cross-border interoperability of electronic archives. |
| 9. Accompanying measures | 2016.3 | Raising Interoperability Awareness – Communication Activities | This action contributes to the promotion of interoperability across the EU. |

\*Note: Additional information was provided by the action owners of the ‘ELISE’ action.

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

Alignment with the EIF

Table 23: ISA2 contribution to the EIF, based on the sample of 20 actions

|  |  |  |  |
| --- | --- | --- | --- |
| Package | Action number | Action Name | Contribution to the EIF |
| 1. Key and generic interoperability enablers | 2016.19 | Trusted Exchange Platform (e-TrustEx) | e-TrustEx is a platform offered to public administrations at European, national and regional levels to undertake secure exchange of natively digital documents or scanned documents from system to system via standardised interfaces. The action thus contributes particularly to Recommendation 15 of the revised EIF, through enabling the secure exchange of documents. |
| 1. Key and generic interoperability enablers | 2016.29 | Catalogue of Services | The action addresses Recommendation 44 of the revised version of the EIF on the catalogue of public services. Catalogue of Public services is one of the interoperability enablers for integrated public services according to the conceptual model defined by the revised EIF.  To that end, the action is defining a technical specification (data model) and implementing a set of tools to facilitate the creation of catalogue of public services. |
| 2. Semantic interoperability | 2016.07 | SEMIC: Promoting Semantic Interoperability Amongst the European Union Member States | The Action supports the implementation of the EIF and the EIS by promoting semantic interoperability, through the definition and use of common specifications, thus contributing primarily to Recommendation 16 of the revised EIF. In addition, the action covers the following underlying principles of the EIF: Reusability, Multilingualism, Openness, Semantic interoperability, Technical interoperability and Standardisation. |
| 2. Semantic interoperability | 2016.16 | Public Multilingual Knowledge Management Infrastructure for the Digital Single Market | The action meets the recommendations and principles of the new EIF, in particular regarding multilingualism, accessibility, administrative simplification, transparency, and reusability of the solutions. The creation of a Public Multilingual Knowledge Infrastructure will support EU public administrations in creating services that can be accessible and shareable independently from the language actually used, as well as allowing SMEs to sell goods and service cross-border in a digital single market. |
| 3. Access to data/data sharing/open data | 2016.03 | Big Data for Public Administrations | This action will facilitate the sharing of open data between public administrations through the support to the execution of analytics projects on Big Data; increase the transparency of decision-making in public administrations by supporting knowledge sharing on evidence-based policy-making practices; support the re-use of open-source data analytics tools developed by Member States of EU Institutions; and provide public administrations with the opportunity to test (open source) technologies in this domain before making a decision on the technical way forward. |
| 3. Access to data/data sharing/open data | 2016.06 | Sharing Statistical Production and Dissemination Services and Solutions in the European Statistical System | The project contributes to the new EIF and the Interoperability Action Plan by  1) developing, maintaining and promoting interoperable solutions for the production and dissemination of statistics by EU public administrations (including the EC) EIF (Focus Area 4); 2) developing, maintaining and promoting a) a specification of the EIRA to support better interoperability and cooperation for the production and dissemination of Official Statistics in the European Statistical System; b) a common infrastructure for the exposure and consumption of shared statistical services. (Focus Area 5)  In addition, the proposal contributes significantly to the realisation of the ESS Vision 2020 objectives in the domain of sharing tools and improving statistical dissemination. |
| 3. Access to data/data sharing/open data | 2016.18 | Development of an Open Data Service, Support and Training Package in the Area of Linked Open Data, Data Visualisation and Persistent Identification | The action contributes to the new EIF, namely the interoperability principles: openness, transparency, reusability, user–centricity and multilingualism. The action contributes to priorities: organisational interoperability (Focus Area 2); sharing of good practices (Focus Area 3); governance structure (Focus Area 1) and key enablers: EU open data initiative (Focus Area 4). |
| 4. Geospatial solutions | 2016.10 | European Location Interoperability Solutions for e-Government (ELISE) | ELISE will deepen the understanding of location interoperability enablers and barriers related to the transition towards digital government. As such, it is intended to provide technical assessments and recommendations for the new EIF and contribute to the implementation of the Interoperability Action Plan, particularly when it comes to Actions 4, 6, 17 & 19. |
| 6. Decision making and legislation - Supporting instruments | 2016.23 | Legal interoperability (former ICT Implications of EU Legislation) | The action contributes to all EU policies, as it is about ensuring that EU legislation, no matter what the policy area, takes into account interoperability, ICT aspects and related impacts.  The action implements Recommendation 27 on legal interoperability of the new EIF and the interoperability Action Plan action 3 of Focus Area 1 and actions 19 and 20 of Focus Area 5. |
| 6. Decision making and legislation - Supporting instruments | 2017.04 | [Inter-Institutional Register of Delegated Acts (RegDel)](https://ec.europa.eu/isa2/dashboard/node/45312) | This action contributes primarily to the transparency principle of the EIF and to Recommendation 5 of the new EIF by providing a transparent overview of delegated acts. |
| 8. Supporting instruments for public administrations | 2016.20 | [Joinup – European Collaborative Platform and Catalogue](https://ec.europa.eu/isa2/dashboard/node/645) | The action facilitates the sharing and reuse of solutions for public administrations and provides the stakeholders with the means to collaborate via a collaborative platform, thus contributing to the reusability principle of the EIF. |
| 8. Supporting instruments for public administrations | 2016.21 | National Interoperability Framework Observatory | This action will fulfil Objectives 4 and 5 of the Action Plan for interoperability. The observatory will monitor the implementation of the EIF Recommendations by Member States and the achievement of the roadmap of actions proposed in the EIF-IS and AP. It will also provide ad hoc and support training to Member States' public administrations to ensure EIF implementation across all levels of their national administrations. |
| 8. Supporting instruments for public administrations | 2016.35 | EUSurvey | EUSurvey contributes primarily to the multilingualism principle of the EIF. |
| 8. Supporting instruments for public administrations | 2017.01 | Standard-Based Archival Data Management, Exchange and Publication | The action follows the line of the second pillar of EIF (Core interoperability principles: Openness, Transparency, Reusability, Technological neutrality and data portability) and of the fourth pillar of EIF (Foundation principles for cooperation amongst public administrations: Preservation of information, Effectiveness and Efficiency) by proposing the creation of a standard-based approach for facilitating the preservation and the exchange of archival information of the public administrations in an open, transparent and reusable way. It also aims at creating a set of recommendations for the publication of archival information as open data mainly for the usage of citizens, focusing thus on the principles of the third pillar of EIF (Principles related to generic user needs and expectation: User-centricity, Inclusion and accessibility, Multilingualism). |
| 9. Accompanying measures | 2016.30 | Raising Interoperability Awareness – Communication Activities | This action contributes to the principles of inclusion, accessibility and transparency by disseminating information about interoperability and the work of ISA2. |

Source: 
[CEPS final study](https://data.europa.eu/doi/10.2799/13397)

:   [(1)](#footnoteref2)
       Decision (EU) 2015/2240 of the European Parliament and of the Council of 25 November 2015 establishing a programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens (ISA2 programme) as a means for modernising the public sector, Brussels 4.12.2015.
:   [(2)](#footnoteref3)
       Recital (21) of the ISA2 Decision (see footnote 1).
:   [(3)](#footnoteref4)
       Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, European Interoperability Framework — Implementation Strategy, Brussels, 23.3.2017, COM(2017) 134 final.
:   [(4)](#footnoteref5)
       Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, A Digital Single Market Strategy for Europe, Brussels, 6.5.2015, COM(2015) 192 final.
:   [(5)](#footnoteref6)
       Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions EU, eGovernment Action Plan 2016-2020. Accelerating the digital transformation of government, COM/2016/0179 final, p. 3.
:   [(6)](#footnoteref7)
       Tallinn Declaration on eGovernment, made at a ministerial meeting during the Estonian Presidency of the Council of the EU on 6 October 2017.
:   [(7)](#footnoteref8)
       Regulation (EU) No 1316/2013 of the European Parliament and of the Council of 11 December 2013 establishing the Connecting Europe Facility, amending Regulation (EU) No 913/2010 and repealing Regulations (EC) No 680/2007 and (EC) No 67/2010.
:   [(8)](#footnoteref9)
       Regulation (EU) 2018/1724 of the European Parliament and of the Council of 2 October 2018 establishing a single digital gateway to provide access to information, to procedures and to assistance and problem-solving services and amending Regulation (EU) No 1024/2012, OJ L 295, 21.11.2018, p. 1-38.
:   [(9)](#footnoteref10)
       Regulation (EU) 2017/825 of the European Parliament and of the Council of 17 May 2017 on the establishment of the Structural Reform Support Programme for the period 2017 to 2020 and amending Regulations (EU) No 1303/2013 and (EU) No 1305/2013, OJ L 129, 19.5.2017, p. 1-16.
:   [(10)](#footnoteref11)
       More information about Horizon 2020 and its support for public sector innovation can be found here: ‘ICT-enabled public sector innovation in Horizon 2020’, 
    <https://ec.europa.eu/digital-single-market/en/ict-enabled-public-sector-innovation-horizon-2020>
    <.>
:   [(11)](#footnoteref12)
       More information about activities related to the modernisation of public institutions can be found here: ‘European Social Fund — More Effective Institutions’, 
    <http://ec.europa.eu/esf/main.jsp?catId=526&langId=en>
    <.>
:   [(12)](#footnoteref13)
       More information about the ERDF and its support for the ICT modernisation of public administrations can be found here: ‘Regional Policy — Information and communication technologies’, 
    <https://ec.europa.eu/regional_policy/en/policy/themes/ict/>
    <.>
:   [(13)](#footnoteref14)
       1) Community contribution for telematics interchange of Data between Administrations (
    [IDA](http://data.europa.eu/eli/dec/1995/468/oj)
    ), 1995-1997; 2) Second phase of the IDA programme (
    [IDA II](http://data.europa.eu/eli/dec/1999/1719/oj)
    ), 1999-2004; 3) Interoperable Delivery of Pan-European eGovernment Services to public Administrations, Business and Citizens (
    [IDABC](http://data.europa.eu/eli/dec/2004/387/corrigendum/2004-05-18/oj)
    ), 2005-2009; 4) Interoperability Solutions for European Public Administrations (
    [ISA](http://data.europa.eu/eli/dec/2009/922/oj)
    ), 2010-2015; 5) Interoperability solutions and common frameworks for European public administrations, businesses and citizens (
    [ISA2](http://data.europa.eu/eli/dec/2015/2240/oj)
    ), 2016-2020.
:   [(14)](#footnoteref15)
       Further details can be found at 
    <http://ec.europa.eu/archives/isa/>
    .
:   [(15)](#footnoteref16)
       Report from the Commission to the European Parliament and the Council, Results of the final evaluation of the ISA programme, Brussels, 1.9.2016, COM(2016) 550 final.
:   [(16)](#footnoteref17)
       In the 2016 and 2017 rolling work programmes, the semantic interoperability package is referred to as the ‘information interoperability’ package.
:   [(17)](#footnoteref18)
       In addition, working groups on specific topics are organised as part of specific actions. For instance, in the field of geospatial interoperability solutions, the ‘European Location Interoperability Solutions for e-Government’ (ELISE) action set up the ISA2 Working Group on Geospatial Solutions, bringing together representatives from Member States active in the field in order to set priorities and disseminate results. See: 
    <https://ec.europa.eu/isa2/actions/elise_en>
    .
:   [(18)](#footnoteref19)
       For more details, please see: 
    <https://ec.europa.eu/isa2/news/european-commission-reinforces-cooperation-uruguay-interoperability_en>
    .
:   [(19)](#footnoteref20)
       The dashboard is available at 
    <https://ec.europa.eu/isa2/dashboard/>
    .
:   [(20)](#footnoteref21)
       CEPS final study: Evaluation study supporting the interim evaluation of the programme on interoperability solutions for European public administrations, businesses and citizens (ISA²). 
    <https://data.europa.eu/doi/10.2799/13397>
    <.>
:   [(21)](#footnoteref22)
       For more details, please see 
    [Annex 1: Procedural information](#_Ref9349539)
     and 
    [Annex 2: Synopsis report of the consultation activities](#_Ref9349562)
    .
:   [(22)](#footnoteref23)
       European Commission (2017), Better Regulation Toolbox, 
    <https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_2.pdf>
    , p. 20.
:   [(23)](#footnoteref24)
       European Commission (2017), ‘Tool #60. The standard cost model for estimating administrative costs’, in the Better Regulation ‘Toolbox’ (see footnote 22).
:   [(24)](#footnoteref25)
       European Commission (2017), ‘Tool #57. Analytical methods to compare options or assess performance’, in the Better Regulation Toolbox (see footnote 22).
:   [(25)](#footnoteref26)
       European Commission (2018), PM² project management methodology, available at 
    <https://data.europa.eu/doi/10.2799/957700>
    .
:   [(26)](#footnoteref27)
       More specifically, the tailored version is based on the Earned Value Management Tutorial, Module 1: Introduction to Earned Value Management, prepared by Booz, Allen and Hamilton, Department of Energy, the United States of America, and on the guidelines on Earned Schedule in Action, developed by Kim Henderson, from the Project Management Institute (PMI) Oklahoma, 13.7.2007.
:   [(27)](#footnoteref28)
       Please see: 
    <https://ec.europa.eu/isa2/sites/isa/files/consultation_strategy_isa2_interim_evaluation.pdf>
    <.>
:   [(28)](#footnoteref29)
       The analysis presented in section 5.2 shows that more mature solutions perform relatively better. For further details regarding the performance of indicators for older and newer actions implemented under ISA2, please see 
    [Box 3](#_Ref9596697)
    .
:   [(29)](#footnoteref30)
       Needs are defined here as prerequisites for the efficient delivery of European public services; problems consist of specific bottlenecks that make it impossible to meet these needs.
:   [(30)](#footnoteref31)
       This need was indicated by respondents from the following stakeholder groups: programme governance (at the Member State level), solution users and action owners.
:   [(31)](#footnoteref32)
       See Annex 2 to the Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, European Interoperability Framework — Implementation Strategy, Brussels, 23.3.2017, COM(2017) 134 final.
:   [(32)](#footnoteref33)
       A ‘self-sovereign identity’ is an emerging concept referring to the creation and management of digital identities by individuals. For more information, please see: Der, Uwe, Stefan Jähnichen, Jan Sürmeli (2017), Self-sovereign Identity − Opportunities and Challenges for the Digital Revolution, available online at 
    <https://arxiv.org/abs/1712.01767>
    .
:   [(33)](#footnoteref34)
       The way information trickles down from the national level to the regional and local levels is important.
:   [(34)](#footnoteref35)
       As outlined in the intervention logic, the programme's general objective is as follows: ‘To promote the ICT-based modernisation of the public sector in Europe and to facilitate addressing the needs of businesses and citizens via improved interoperability of European public administrations, thus contributing to the completion of the Digital Single Market and, ultimately, to economic growth and the global competitiveness of the European Union’.
:   [(35)](#footnoteref36)
       See: 
    [https://webgate.ec.europa.eu/regdel/#/home](https://webgate.ec.europa.eu/regdel/)
    .
:   [(36)](#footnoteref37)
       See: 
    <https://ec.europa.eu/isa2/actions/improving-transparency-delegated-acts_en>
    .
:   [(37)](#footnoteref38)
       The statistics are based on input received from the action owner.
:   [(38)](#footnoteref39)
       See: 
    <https://ec.europa.eu/isa2/solutions/eira_en>
    .
:   [(39)](#footnoteref40)
       See: 
    <https://ec.europa.eu/isa2/solutions/joinup_en>
    .
:   [(40)](#footnoteref41)
       See for instance: 
    <https://joinup.ec.europa.eu/event/sharing-reuse-conference-2017>
    .
:   [(41)](#footnoteref42)
       Report from the Commission to the European Parliament and the Council, Results of the final evaluation of the ISA programme, Brussels, 1.9.2016, COM(2016) 550 final.
:   [(42)](#footnoteref43)
       European Commission, DG DIGIT (2017), ISA2 programme. Communication strategy and stakeholders engagement plan: 
    <https://ec.europa.eu/isa2/sites/isa/files/comms_strategy.pdf>
    .
:   [(43)](#footnoteref44)
       For further details see: 
    <https://ec.europa.eu/isa2/events_en>
    .
:   [(44)](#footnoteref45)
       EuroVoc is the EU’s multilingual and multidisciplinary thesaurus, containing terms in 23 EU languages. For further details, see: 
    <https://publications.europa.eu/en/web/eu-vocabularies/th-dataset/-/resource/dataset/eurovoc>
    <.>
:   [(45)](#footnoteref46)
       See: 
    <https://ec.europa.eu/isa2/solutions/vocbench3_en>
    .
:   [(46)](#footnoteref47)
       SCM Network (2005), ‘The International SCM Manual; Measuring and Reducing Administrative Burdens for Businesses’, available at 
    <http://www.administrativeburdens.com/filesystem/2005/11/international_scm_manual_final_178.doc>
    .
:   [(47)](#footnoteref48)
       European Commission, Better Regulation ‘Toolbox’, 7 July 2017, Tool #60 available at 
    <https://ec.europa.eu/info/sites/info/files/better-regulation-toolbox_2.pdf>
    .
:   [(48)](#footnoteref49)
       Eurofound (2016), ‘Working time developments in the 21st century: Work duration and its regulation in the EU’.
:   [(49)](#footnoteref50)
       The country of activity was recorded for each respondent who provided information on the number of days spent preparing the proposal. The hourly labour costs were identified for each country. For the Commission, the hourly labour costs in either Belgium or Luxembourg were used, depending on the Commission service involved. The regulatory costs were computed for each case recorded during the consultation activities, taking into consideration the country of activity. At the end, the average regulatory cost was calculated. Note that labour costs include wage and non-wage costs less subsidies. They do not include vocational training costs or other expenditure such as recruitment costs, spending on working clothes, etc. For further details see: 
    [https://ec.europa.eu/eurostat/statistics-explained/index.php/Hourly\_labour\_costs#Source\_data\_for\_tables\_and\_figures\_on\_this\_page\_.28MS](https://ec.europa.eu/eurostat/statistics-explained/index.php/Hourly_labour_costs)
    .
:   [(50)](#footnoteref51)
       The average was calculated based on the data available on the ISA2 dashboard for all ISA2 actions.
:   [(51)](#footnoteref52)
       According to Tool #57 of the Better Regulation Toolbox, cost-effectiveness analysis (CEA) entails the quantification (but not monetisation) of the benefits that one euro of costs imposed on society would generate. See page 452 at 
    <https://ec.europa.eu/info/sites/info/files/file_import/better-regulation-toolbox-57_en_0.pdf>
    .
:   [(52)](#footnoteref53)
       Actual costs refer to the executed budget. For more information, see: 
    <https://ec.europa.eu/isa2/dashboard/faq-page>
    .
:   [(53)](#footnoteref54)
       See: 
    <https://ec.europa.eu/tools/espd>
    <.>
:   [(54)](#footnoteref55)
       The earned value represents a quantification of the work accomplished to date. The planned value is a quantification of how far along the work on the programme is supposed to be in terms of the schedule and cost estimate at a given point in the programme. The actual costs represent the executed budget.
:   [(55)](#footnoteref56)
       The PV and AC are equal in all cases. As explained on the ISA2 Dashboard, most of the specific contracts under the ISA2 actions are on fixed price, therefore it is not relevant to keep track of the AC before the end of a Work Programme year. Taking this into account, the AC is then considered equal to the PV. For more details please see: ISA2 Dashboard — Frequently Asked Questions, 
    <https://ec.europa.eu/isa2/dashboard/faq-page>
    .
:   [(56)](#footnoteref57)
       Situation as of 2018.
:   [(57)](#footnoteref58)
       These data are available for each monitored action under the ‘coherence’ tab of the ISA2 dashboard page dedicated to the specific action.
:   [(58)](#footnoteref59)
       ‘EUSurvey’ maintains a tool — with the same name — designed to facilitate the gathering of feedback and opinions quickly and reliably via online surveys.
:   [(59)](#footnoteref60)
       This action developed two solutions: the online collection software supporting the European Citizens’ Initiative and the European Parliament crypto tool. The first tool enables citizens to bring initiatives to the attention of the Commission; the second puts in place mechanisms to prevent double voting.
:   [(60)](#footnoteref61)
       See footnote 7 and 
    <https://ec.europa.eu/inea/en/connecting-europe-facility>
    .
:   [(61)](#footnoteref62)
       See footnote 11.
:   [(62)](#footnoteref63)
       See footnote 10.
:   [(63)](#footnoteref64)
       See footnote 9.
:   [(64)](#footnoteref65)
       See footnote 12.
:   [(65)](#footnoteref66)
       For further details, please see: 
    <https://ec.europa.eu/info/departments/informatics/it-governance_en>
    .
:   [(66)](#footnoteref67)
       See footnote 8.
:   [(67)](#footnoteref68)
       The ISA2 action mentioned is ‘2017.05 Interoperability requirements for the Single Digital Gateway implementation’. For more information see: 
    <https://ec.europa.eu/isa2/actions/common-architecture-single-digital-gateway_en>
    .
:   [(68)](#footnoteref69)
       Please see the description of the ‘ABCDE — Administrations, business and citizens’ data exchanges in the domain of case management’ solution in the 2018 ISA2 rolling work programme: 
    [https://ec.europa.eu/isa2/library/isa%C2 %B2-work-programme\_en](https://ec.europa.eu/isa2/library/isa%C2%B2-work-programme_en)
    .
:   [(69)](#footnoteref70)
       Please see: 
    <https://ec.europa.eu/isa2/solutions/e-documents-reference-architecture_en>
    .
:   [(70)](#footnoteref71)
       See footnote 4.
:   [(71)](#footnoteref72)
       See footnote 5, p. 1.
:   [(72)](#footnoteref73)
       The ‘once-only’ principle project, 
    <http://www.toop.eu/>
    .
:   [(73)](#footnoteref74)
       Please see: 
    <https://ec.europa.eu/digital-single-market/en/rolling-plan-ict-standardisation>
    .
:   [(74)](#footnoteref75)
       Please see: 
    <https://ec.europa.eu/isa2/solutions/asset-description-metadata-schema-adms_en>
    .
:   [(75)](#footnoteref76)
       See details about the W3C note process: 
    <https://www.w3.org/Consortium/Process/NOTE.html>
    .
:   [(76)](#footnoteref77)
       Please see: 
    <https://ec.europa.eu/isa2/solutions/dcat-application-profile-data-portals-europe_en>
    .
:   [(77)](#footnoteref78)
       On a more general note, it is not only the various European standards that may need to be better aligned; the links between European, international and global standards could also be better clarified. The final ISA2 evaluation may investigate what the role of a European interoperability programme should be in this regard.
:   [(78)](#footnoteref79)
       See footnote 6.
:   [(79)](#footnoteref80)
       See: 
    <https://ec.europa.eu/isa2/solutions/imaps_en>
    .
:   [(80)](#footnoteref81)
       See: 
    <https://ec.europa.eu/isa2/news/interested-semantic-interoperability-read-about-semic-highlights-second-half-2018_en>
    .
:   [(81)](#footnoteref82)
       See footnote 3.
:   [(82)](#footnoteref83)
       See 
    [Figure 1](#_Ref1398290)
     for the original needs and problems.
:   [(83)](#footnoteref84)
     
       See: 
    <https://joinup.ec.europa.eu/collection/european-interoperability-reference-architecture-eira/catalogue-solutions>
:   [(84)](#footnoteref85)
     
       See: 
    <https://joinup.ec.europa.eu/collection/european-interoperability-reference-architecture-eira/good-practices>
:   [(85)](#footnoteref86)
     
       See: 
    <https://ec.europa.eu/isa2/solutions/circabc_en>
:   [(86)](#footnoteref87)
       See: 
    <http://ec.europa.eu/isa2/library/eprior/>
:   [(87)](#footnoteref88)
       See: 
    <https://e-justice.europa.eu/home.do?action=home>
:   [(88)](#footnoteref89)
       See: 
    [https://webgate.ec.europa.eu/regdel/#/home](https://webgate.ec.europa.eu/regdel/)
:   [(89)](#footnoteref90)
     
       See: 
    <http://www.europarl.europa.eu/doceo/document/TA-8-2016-0081_EN.html?redirect>
:   [(90)](#footnoteref91)
     
       See: 
    <https://data.europa.eu/euodp/en/visualisation-home>
:   [(91)](#footnoteref92)
       See for example: 
    [Box 2](#_Ref11681623)
     and 
    [Box 7](#_Ref11683292)
    .
:   [(92)](#footnoteref93)
       Operations and maintenance costs consist of the monetary costs incurred to run, monitor, and ensure the proper functioning of a solution.
:   [(93)](#footnoteref94)
       Technical and operational support refers to the human resources mobilised in activities such as helpdesks or providing IT support to users.
:   [(94)](#footnoteref95)
       The technical experts involved in the evaluation noted that for most solutions the continued updating and maintenance might be more expensive than the initial development.
:   [(95)](#footnoteref96)
       See: 
    <https://eupl.eu/>
    .
:   [(96)](#footnoteref97)
       See footnote 20.
:   [(97)](#footnoteref98)
       Please see: 
    <https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-2768206_en>
    <.>
:   [(98)](#footnoteref99)
       Please see: 
    <https://ec.europa.eu/isa2/sites/isa/files/consultation_strategy_isa2_interim_evaluation.pdf>
    <.>
:   [(99)](#footnoteref100)
       Please see: 
    <https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-2768206/feedback_en?p_id=239250>
    <.>
:   [(100)](#footnoteref101)
       Please see: 
    <https://ec.europa.eu/info/law/better-regulation/initiatives/ares-2018-2768206/public-consultation_en>
    <.>
:   [(101)](#footnoteref102)
       Please see: 
    <https://ec.europa.eu/isa2/isa2conf18_en>
    <.>
:   [(102)](#footnoteref103)
       Please see: 
    <https://ec.europa.eu/isa2/events/isa%C2%B2-interim-evaluation-kick-workshop_en>
    <.>
:   [(103)](#footnoteref104)
     
       Stakeholders were considered to be familiar with a package if they indicated that they had knowledge of the package to some extent, to a high extent or to the fullest extent.
:   [(104)](#footnoteref105)
       The PV and AC are equal in all cases. As explained on the ISA2 Dashboard, most of the specific contracts under the ISA2 actions are on fixed price, therefore it is not relevant to keep track of the AC before the end of a Work Programme year. Taking this into account, the AC is then considered equal to the PV. For more details please see: ISA2 Dashboard — Frequently Asked Questions, 
    <https://ec.europa.eu/isa2/dashboard/faq-page>
    .
:   [(105)](#footnoteref106)
       Needs refer to prerequisites for the efficient delivery of European public services, more specifically, in what concerns the interoperability dimension. Problems consist of specific bottlenecks that hinder the realisation of the needs. The drivers are the underlying causes that lead to the identified problem.
:   [(106)](#footnoteref107)
       European Commission (2010), Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, ‘A Digital Agenda for Europe’, COM(2010)245 final.
:   [(107)](#footnoteref108)
       European Commission (2010), Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions, ‘Towards interoperability for European public services’ and ‘Annex II – EIF (European Interoperability Framework)’, COM(2010) 744 final.
:   [(108)](#footnoteref109)
       European Commission (2015), Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions ‘A Digital Single Market Strategy for Europe’, COM(2015) 192 final.
:   [(109)](#footnoteref110)
       See Section 1.2 of the ISA2 Proposal; Proposal for a DECISION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL establishing a programme on interoperability solutions for European public administrations, businesses and citizens (ISA2) Interoperability as a means for modernising the public sector, COM(2014) 367 final.
:   [(110)](#footnoteref111)
       Kurt Salmon, KPMG (2015), Final evaluation of the ISA programme, Final report v2.06, 1 December 2015.
:   [(111)](#footnoteref112)
       European Commission (2015), Commission Staff Working Document ‘A Digital Single Market Strategy for Europe – Analysis and Evidence’ Accompanying the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions ‘A Digital Single Market Strategy for Europe’, SWD(2015) 100 final.
:   [(112)](#footnoteref113)
       National administrative systems are often characterised by different administrative cultures, which include the beliefs and values on the role of the State and its civil servants. As pointed out by a recent Study published by the Commission, although European public administrations share the values associated with democracy and the rule of law, national administrative cultures show clear differences (Thijs N., Hammerschmid G., Palaric E. (2017), A comparative overview of public administration characteristics and performance in EU28, European Commission).
:   [(113)](#footnoteref114)
       See recitals 34, 36 and 45 of the ISA2 Decision and Section 1.1 of the ISA2 Proposal.
:   [(114)](#footnoteref115)
       European Commission (2017), Better Regulation Guidelines, SWD(2017) 350, Brussels, 7 July 2017.
:   [(115)](#footnoteref116)
       For further details see: 
    <https://joinup.ec.europa.eu/>
:   [(116)](#footnoteref117)
       European Commission (2016), ISA2 Work Programme 2016 – Summary.
:   [(117)](#footnoteref118)
       
    <https://ec.europa.eu/isa2/solutions/testa_en>
:   [(118)](#footnoteref119)
       
    <https://ec.europa.eu/isa2/solutions/open-e-trustex_en>
:   [(119)](#footnoteref120)
       
    <https://ec.europa.eu/isa2/solutions/core-vocabularies_en>
:   [(120)](#footnoteref121)
       
    <https://ec.europa.eu/isa2/solutions/dcat-application-profile-data-portals-europe_en>
:   [(121)](#footnoteref122)
       
    <https://ec.europa.eu/isa2/actions/providing-big-data-opportunities-public-administrations_en>
:   [(122)](#footnoteref123)
       
    <https://ec.europa.eu/isa2/actions/towards-better-financial-data-reporting_en>
:   [(123)](#footnoteref124)
       
    <https://ec.europa.eu/isa2/solutions/re3gistry_en>
:   [(124)](#footnoteref125)
       
    <https://ec.europa.eu/isa2/solutions/open-e-prior_en>
:   [(125)](#footnoteref126)
       
    <https://ec.europa.eu/isa2/solutions/european-single-procurement-document-espd_en>
:   [(126)](#footnoteref127)
       
    <https://ec.europa.eu/isa2/solutions/e-certis_en>
:   [(127)](#footnoteref128)
       
    <https://ec.europa.eu/isa2/solutions/leos_en>
:   [(128)](#footnoteref129)
       
    <https://ec.europa.eu/isa2/solutions/ocs-ecis-0_en>
:   [(129)](#footnoteref130)
       
    <https://ec.europa.eu/isa2/solutions/eusurvey_en>
:   [(130)](#footnoteref131)
       
    <https://ec.europa.eu/isa2/solutions/joinup_en>
:   [(131)](#footnoteref132)
       
    <https://ec.europa.eu/isa2/solutions/nifo_en>
:   [(132)](#footnoteref133)
       
    <https://ec.europa.eu/isa2/news/save-date-semic-2018_en>
:   [(133)](#footnoteref134)
       
    <https://ec.europa.eu/isa2/events/isa%C2%B2-mid-term-conference-linking-public-administrations-businesses-and-citizens_en>
:   [(134)](#footnoteref135)
       
    <https://ec.europa.eu/isa2/dashboard/>
:   [(135)](#footnoteref136)
       Note that the wording of some elements of the evaluation framework mentioned in the body text of the Staff Working Document (like the judgment criteria) has been slightly revised in some cases to improve understanding.
:   [(136)](#footnoteref137)
       The efficiency of the funded actions is already monitored by using the Earned Value Management (EVM) approach.
:   [(137)](#footnoteref138)
       The utility criterion to some extent is similar to the relevance criterion insofar as they both look at stakeholders' needs. However, while the relevance criterion looks at the alignment between the objectives of the programme and the current needs and problems experienced by stakeholders, the utility criterion focuses on how the actual results of the programme have (or do not have) contributed to meeting stakeholders’ needs. Hence, the utility criterion is a proxy for measuring users’ satisfaction.
:   [(138)](#footnoteref139)
       In order to draft policy recommendations, feedback on measures to improve the utility of the ISA² programme’s actions has been collected. However, stricto sensu, future measures are not part of the interim evaluation, which is mostly a backward-looking exercise.
:   [(139)](#footnoteref140)
       Stakeholders related to all 53 actions were invited to answer the online surveys. However, desk research (review of documentary evidence) and interviews focused only on sampled actions.
:   [(140)](#footnoteref141)
     ‘CAPTCHA’ stands for ‘Completely Automated Public Turing Test to tell Computers and Humans Apart’ and it represents a test that is expected to be difficult for machines to complete correctly, but possible for humans to complete correctly.
:   [(141)](#footnoteref142)
     For further details see: 
    [https://ec.europa.eu/eurostat/statistics-explained/index.php/Hourly\_labour\_costs#Source\_data\_for\_tables\_and\_figures\_on\_this\_page\_.28MS](https://ec.europa.eu/eurostat/statistics-explained/index.php/Hourly_labour_costs)
:   [(142)](#footnoteref143)
       Access: better access for consumers and businesses to digital goods and services across Europe.
:   [(143)](#footnoteref144)
       Environment: creating the right conditions and a level playing field for digital networks and innovative services to flourish.
:   [(144)](#footnoteref145)
       Economy & Society: maximising the growth potential of the digital economy.

[Top](#document1)