EUROPEAN COMMISSION Brussels, 23.9.2019 COM(2019) 615 final REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL Results of the interim evaluation of the ISA² programme {SWD(2019) 1615 final}



Table of contents 1. Introduction 2. Background 3. Common understanding of interoperability 4. Methodology 5. Evaluation issues and questions 6. Findings 6.1. Relevance 6.2. Effectiveness 6.3. Efficiency 6.4. Coherence 6.5. EU added value 6.6. Utility 6.7. Sustainability 7. Recommendations 7.1. Awareness-raising beyond national administrations 7.2. From user-centric to user-driven solutions 7.3. Paving the way to increased sustainability 8. Next steps



1.Introduction As provided for in the ISA2 Decision, on 1 January 2016 the five-year programme on interoperability solutions and common frameworks for European public administrations, businesses and citizens (ISA2 programme) was launched as a follow-up to the ISA programme. This report presents the findings and recommendations of the interim evaluation of the ISA2 programme. Under Article 13(3) of the ISA2 Decision, the Commission is required to carry out this evaluation and communicate its results to the European Parliament and the Council by 30 September 2019. The Commission carried out the evaluation using a team of independent experts from a consultancy company. The evaluation was overseen by an interservice steering group. 2.Background The ISA2 programme’s ultimate objective is to promote the ICT-based modernisation of the public sector in Europe and to facilitate addressing the needs of businesses and citizens, via improved interoperability of European public administrations. More specifically the programme aims to do the following: –Facilitate efficient and effective electronic cross-border or cross-sector interaction between European public administrations, businesses and citizens. –Contribute to the development of a more effective, simplified and user-friendly e-administration at the national, regional and local administration levels. –Promote a holistic approach to interoperability in the EU, by identifying, creating and operating interoperability solutions and facilitating their reuse by European public administrations. This will support the implementation of various EU policies and activities. To meet these objectives, the programme builds on the achievements of its predecessor, the ISA programme. It also puts emphasis on fitting smoothly into the wider policy framework related to the digitalisation of public administrations in the EU. In this regard, the ISA2 programme is the main instrument to support the implementation of the European Interoperability Framework (EIF), which has been updated and extended as planned in the Communication on a Digital Single Market Strategy for Europe. Other policy initiatives to which ISA2 contributes include the eGovernment Action Plan 2016-2020, the 2017 Tallinn Declaration on eGovernment and the Single Digital Gateway Regulation. In practical terms, ISA2 runs from 1 January 2016 until 31 December 2020 with a total budget of €130.9 million. It funds actions defined on a yearly basis in the annual rolling work programme. The programme’s management also fosters synergies with other EU programmes, like the Connecting Europe Facility (CEF) or the Structural Reform Support Programme (SRSP). The programme’s primary stakeholders are the European public administrations at all levels: EU, national, regional and local. Yet the programme affects a much wider group of stakeholders, including EU businesses and citizens. ISA2 is open to EU Member States, other countries of the European Economic Area and candidate countries. In addition to the 28 EU Member States, 3 other countries take part in the programme: Iceland, Norway (since 2016) and Montenegro (since 2018). The programme also encourages cooperation with other non-EU countries and with international organisations and bodies. 3.Common understanding of interoperability In line with Article 1(2) of the ISA2 Decision, ‘the ISA2 programme shall ensure a common understanding of interoperability through the EIF and its implementation in Member States’ administrations. The Commission, through the ISA2 programme, shall monitor the implementation of the EIF.’ In this respect, the ISA2 programme is developing an integrated framework for monitoring, assessing and reporting the progress made in implementing the EIF by both the Member States and the Commission. The design and implementation of this monitoring framework are conducted within the remit of the National Interoperability Framework Observatory (NIFO) action, funded by the ISA² programme. The EIF implementation results across the EU will be presented yearly through an online scoreboard to better inform policy-making and to communicate to a wider public in a more visual and intuitive way. In practice, the scoreboard aims to improve communication and capacity building to make Member States better aware of their current status in the EIF’s implementation, and to help them comply more easily with the EIF’s recommendations. To achieve these goals, a toolbox will be made available to the Member States to provide practical guidance on implementing the EIF by means of good practices and solutions. The monitoring framework for the EIF implementation is meant to provide evidence for the EIF’s future evaluation, which was initially planned to take place by the end of 2019. However, in order to leave a realistic timeframe for Member States to adapt their national policies on eGovernment, digital services and interoperability to the EIF and given that the broader scope of the new EIF requires additional internal and external coordination to finalise a commonly accepted and integrated monitoring framework, it appears necessary to postpone the EIF evaluation until 2021. Indeed, these factors resulted in the data collection process only beginning in 2019, thus there is a lack of data to evaluate the EIF’s implementation in the same year.

In addition, by running the EIF’s evaluation back-to-back with the final evaluation of the ISA2 programme in 2021, several synergies can be created, since the programme is the main instrument supporting the EIF’s implementation. 4.Methodology The interim evaluation of the ISA2 programme relied on quantitative and qualitative data collected through various methods, like desk research, public consultation, targeted online surveys, in-depth interviews and a kick-off workshop. The consultation activities allowed for a wide coverage of the different ISA2 stakeholders, ranging from representatives of Member States and Commission departments to citizens and standardisation organisations. In order to better guide data collection, a sample of 20 actions was selected from the 53 actions — grouped into 9 packages — funded by the ISA2 programme up to 2018. Four pre-defined criteria steered this sampling to ensure that the selected actions are largely representative of the programme. Furthermore, the overall data collection was complemented by an expert assessment of the programme carried out by four technical experts in interoperability. 5.Evaluation issues and questions The evaluation focused on the following seven main criteria: –Relevance — to what extent are the objectives of the ISA² programme still pertinent in relation to the evolving needs and problems at both national and EU levels? –Effectiveness — how far are the ISA² programme’s results in the process of achieving the programme’s objectives? Are there aspects that are more or less effective than others, and if so, what lessons can be drawn from this? –Efficiency — to what extent has the programme been cost-effective? Which aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised? How is the programme performing relative to the planned work and budget? –Coherence — to what extent do the ISA² actions form part of a ‘holistic’ approach within the framework of the programme (internal coherence)? To what extent is the ISA² programme coherent with other EU interventions, which have similar objectives, and with global initiatives in the same field (external coherence)? –EU added value — what is the additional value resulting from the ISA² programme, compared to what could reasonably have been expected from Member States acting at national, regional and/or local levels? –Utility — how do the ISA² programme’s actions and results, achieved and anticipated, compare with the needs they are supposed to address? –Sustainability — to what extent is the financial, technical and operational sustainability of the developed solutions — maintained and operated through the ISA² programme — ensured? 6.Findings Based on data collected from 129 consulted stakeholders, extensive desk research, and expert assessments, the interim evaluation confirms that ISA2 performs well in all the evaluation criteria. However, the evaluation is confined to actions funded between 2016 and 2018; hence, the results of ongoing actions as well the longer-term outcomes of ISA2 can only be captured after the programme finishes. 6.1.Relevance The objectives pursued by ISA2 are still pertinent in relation to the evolving needs and problems in the field of interoperability of digital public services, confirming the programme’s relevance. Most of the consulted stakeholders agree that the problem of administrative e-barriers and related needs originally identified by the programme are still valid. However, some of them (46 out of 129) currently have some additional needs (like a more prescriptive approach to design interoperable public services) and problems (including resources constraints), which ISA2 can only be partially address. One additional need, which deserves special attention, is to ensure more collaboration and exchanges with regional and local administrations in order to increase awareness of interoperability and the take-up of ISA2 solutions at the sub-national level. Raising public administrations’ awareness on interoperability was the first of the three recommendations in the final evaluation of the ISA programme. The ISA2 programme acted upon this call by adopting its communication strategy and engagement plan in 2017 and by organising 10 major events between 2016 and 2018, with an average of 211 participants per event. Moreover, ISA2 representatives played an active role in 60 events held over the same period. The interim evaluation shows that these efforts paid off, as most of the consulted stakeholders indicated that they have expert knowledge of interoperability (91 out of 128) and of ISA2 (81 out of 128). In spite of this general awareness, the above-highlighted additional need signals that the programme should continue its awareness-raising activities targeting regional and local administrations and possibly indirect beneficiaries, like citizens and businesses too. 6.2.Effectiveness

The results achieved so far by ISA2 are aligned with the programme’s objectives. Nevertheless, they still do not fully match the expected results, as most of the actions are still ongoing and solutions are still being developed. The duration of the programme also influences the take-up rate of solutions. As such, actions that have been continued from previous editions of the programme have produced solutions that are now more widely used than solutions resulting from actions that have been established under ISA2. There is one specific ISA2 objective where the evaluation found that the programme on its own is less effective: the development of a more effective, simplified and user-friendly e-administration at the national, regional and local level. Here Member States can in fact have a great impact, and complement the EU-level initiatives on interoperability and digitalisation. External factors can improve but also jeopardise the way in which the programme achieves its objectives and delivers its results. The call for common standards and frameworks from public administrations represents an external factor contributing to the programme’s performance. By contrast, institutional complexity could hamper the achievement of interoperability across borders and across sectors. Finally, the evaluation found that the ISA2 actions are largely compliant with the general principles listed in Article 4 of the ISA2 Decision, due to the rolling work programme process, which ensures that the principles are systematically taken into account when the work programme is being prepared. 6.3.Efficiency The programme’s implementation is progressing as planned; all actions are either on track or close to achieving the planned level of work. Nevertheless, the heterogeneity of performance indicators makes it difficult to draw conclusions about the programme’s overall cost-effectiveness. For those ISA2 packages where it was possible to aggregate performance indicators of different actions, costs per end-user (e.g. businesses, citizens) have been estimated as very low. The process for selecting actions funded by ISA2 is considered relatively efficient. The costs incurred to prepare a proposal are rather small, ranging from 0.07% to 0.4% of the potential funds that could be allocated to the proposal once accepted. Nonetheless, the selection process could be further improved by simplifying the rolling work programme (e.g. making the template more flexible) and launching thematic calls for actions, thus ensuring that the selection process is driven by objectives. 6.4.Coherence According to the consulted stakeholders, substantial synergies and limited overlaps characterise the ISA2 actions. This strong internal coherence is in line with the second recommendation of the final ISA evaluation, which called for a holistic approach on interoperability within ISA2. On external coherence, ISA2 followed the third — and last — recommendation of the final ISA evaluation, and built close cooperation with other EU policies and initiatives. It fostered synergies with CEF, Horizon 2020 and the Structural Reform Support Programme. ISA2 also interacts with initiatives such as the single digital gateway, Digital Single Market Strategy, eGovernment Action Plan, Tallinn Declaration on eGovernment, and the rolling plan for ICT standardisation. Nonetheless, some overlaps were perceived between specific solutions under ISA2 and CEF, which could be addressed by better delineating the scope of each solution. 6.5.EU added value The level of coordination ensured by ISA2 is crucial to improving the overall interoperability among European public administrations. In addition, 91 out of 109 respondents emphasised that ISA2 is able to achieve its objectives at a lower cost than comparable national or sub-national initiatives. It is apparent that ISA2 has helped improve cross-border interoperability in the EU: i) it raises awareness about interoperability across the Member States; ii) helps put the topic on national agendas; and iii) creates networks and facilitates exchanges. ISA2 has also helped to further common EU policies: it plays a central role in implementing the EIF and supports the establishment of the digital single market. 6.6.Utility User satisfaction with the ISA2 solutions tends to be positive with only 7 out of 110 stakeholders reporting ‘limited satisfaction’. As the programme is ongoing, it is expected that the take-up of ISA2 solutions will increase, thus improving the way that solutions meet user needs and, in turn, increasing overall user satisfaction. 6.7.Sustainability The sustainability of the ISA2 solutions received mixed feedback from consulted stakeholders. While 66 out of 84 respondents believe that the programme’s results would last in the absence of future funding, over 55 out of 85 respondents consider that the need for operations and the maintenance costs required for the solutions could harm the ability of ISA2 solutions to deliver their results if the programme was terminated. Stakeholders agree though, that ISA2 plays a central role in improving the interoperability landscape in the EU and its absence would jeopardise the efforts of European public administrations to improve interoperability and to foster the ICT-based modernisation of the public sector in Europe. 7.Recommendations The interim evaluation paints a positive picture of the ISA2 programme’s performance so far. However, it also identifies areas for possible improvement. The recommendations below address these aspects of the programme, outlining both short- and longer-term measures. The short-term measures are those, which could be taken on board in the last rolling ISA2 work programme in 2020 and during the transition period leading to the set-up of the new Multiannual Financial Framework (MFF). The longer-term measures would need more years to yield results.