EUROPEAN COMMISSION Brussels, 19.4.2016 SWD(2016) 108 final COMMISSION STAFF WORKING DOCUMENT Implementation and Evaluation report Accompanying the document Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions EU eGovernment Action Plan 2016 - 2020

Accelerating the digital transformation of government {COM(2016) 179 final}

{SWD(2016) 109 final}



Table of contents 1.Introduction 1.1.The purpose of the evaluation 1.2.The scope of the evaluation 2.Background to the EU eGovernment Action Plan 2011 - 20156 2.1.Description of the European eGovernment Action plan 2011 – 2015 and its objectives6 2.2.Instruments8 2.2.1.Political Activities including exchange of experience and Commission Communications8 2.2.2.Funding for eGovernment9 2.2.3.Legislation10 2.3.Intervention logic11 2.4.Situation in 2010/Baseline12 2.4.1.Baseline of target: 50% of citizens (Digital Agenda) and 80% of businesses make use of eGovernment services13 2.4.2.Baseline of target: a number of key cross-border services be offered online by 2015 (Digital Agenda)13 2.4.3.Baselines in the four policy priority areas13 3.Evaluation Questions/Criteria15 4.Method and sources16 4.1.Mid-term evaluation16 4.2.Annual benchmarking reports16

4.3.The online public consultation17 4.4.Limitations18 5.Implementation of the EU eGovernment Action Plan 2011 – 2015 – state of play of the policy priorities19 5.1.General target: 50% of citizens (Digital Agenda) and 80% of businesses make use of eGovernment services19 5.2.General target: a number of key cross-border services be offered online by 2015 (Digital Agenda)19 5.3.The four priorities and 40 actions of the EU eGovernment Action Plan22 5.4.Governance22 6.Answers to the evaluation questions24 6.1.Effectiveness24 6.1.1.Objective on using eGovernment (take-up)24 6.1.2.Key cross-border services online25 6.1.3.Effectiveness of the four policy priorities26 6.2.Efficiency36 6.2.1.Efficiency of the related budget36 6.2.2.Costs and benefits to stakeholders38 6.3.Relevance39 6.3.1.Relevance: User Empowerment40 6.3.2.Relevance: Internal market42 6.3.3.Relevance: "Efficiency and Effectiveness of Governments and Administrations"45 6.3.4.Relevance: Pre-conditions48

6.4.Coherence49 6.5.EU-added value51 7.Conclusions54 8.ANNEXES57 Annex 1.Procedural information57 Annex 2.The Online public consultation59 Annex 3.The individual actions of the eGovernment Action Plan 2011 - 201589 Annex 4.Implementation of the individual actions of the eGovernment Action Plan 2011 – 2015 – State of play of the actions93 Priority: User Empowerment93 Priority: Internal Market96 Priority: Efficiency and Effectiveness of Governments and Administrations98 Priority: Pre-conditions for developing eGovernment100 Annex 5.Acronyms103 Annex 6.REFERENCES AND DATA SOURCES105



1.Introduction 1.1.The purpose of the evaluation In December 2015, European Commission's Directorate-General Communications Networks, Content and Technology launched an implementation and evaluation assessment of the European eGovernment Action Plan 2011 – 2015 ("EU eGovernment Action Plan 2011 – 2015"), adopted on 15th December 2010. The purpose of this evaluation is to describe the progress of implementation of individual actions of the Action Plan and assess its functioning. In the broadest sense, this evaluation exercise will provide answers to how well the European public administrations have progressed towards meeting the Malmö vision (November 2009 Malmö Ministerial declaration), which is included in this eGovernment Action Plan: "by 2015 European public administrations will be recognised for being open, flexible and collaborative in their relations with citizens and businesses. They use eGovernment to increase their efficiency and effectiveness and to constantly improve public services in a way that caters for user's different needs and maximises public value, thus supporting the transition of Europe to a leading knowledge-based economy."

The EU eGovernment Action plan 2011-2015 did not indicate the need to carry out a final evaluation as such. According to the eGovernment Action Plan, the overall progress would be measured annually, using an appropriate mix of instruments and methods. However, according to the Commission's Better Regulation guidelines, retrospective performance evaluations support policy preparation. "Evaluations gather evidence to assess how well a specific intervention has performed (or is working) - -. An evaluation also draws conclusions on whether the EU intervention continues to be justified or should be modified to improve its effectiveness, relevance and coherence or to eliminate excessive burdens or inconsistencies or simply be repealed." This evaluation is carried out in so far as possible in this context, in order to provide a set of lessons learnt to guide the work of the new eGovernment Action Plan 2016 – 2020 announced in the Digital Single Market Strategy. 1.2.The scope of the evaluation The scope of the evaluation is the objectives and measures set out in the EU eGovernment Action Plan 2011 – 2015. It covers the level of progress in eGovernment policy in the EU in the period starting from the beginning of 2011 (Action Plan was adopted 15 December 2010) until and including 2015, subject to the availability of most recent data. The Action Plan 2011 – 2015 aimed to contribute to achieving two important targets of the Digital Agenda in Europe: 1) "By 2015, 50% of EU citizens will have used eGovernment services"; and 2) offering a number of cross-border services online. The Action Plan also aimed that "by 2015, 80% of enterprises will have used eGovernment". Furthermore, the evaluation focuses on the four policy priorities (agreed by Commission and Member States in the Malmö Ministerial Declaration) of the EU eGovernment Action Plan 2011 – 2015: 1.User Empowerment 2.Internal Market 3.Efficiency and Effectiveness of Governments and Administrations 4.Pre-conditions for developing eGovernment The Action Plan included 40 individual actions (listed in Annex 4) divided amongst these four priorities. Five additional actions were related to the governance. The evaluation focuses on the Action Plan as a political instrument to achieve its objectives. It does not evaluate in detail individual actions identified in the Action Plan. An assessment of the completion of actions will be done separately via a study planned for 2016. The results of the study will be used to update to complete this evaluation. Furthermore, since the scope of the evaluation is the policy priorities eGovernment Action Plan 2011 – 2015, it will not cover eGovernment related activities in Member States at all levels of government and all sectors. For the evaluation of the eGovernment policy priorities in the EU it is important to note that eGovernment has also been supported through a number of funding programmes (listed in section 2.2.2). The evaluation of these funding programmes falls outside the scope of this evaluation but their findings are referred to in this evaluation when relevant to address the evaluation criteria. The evaluation criteria used are in line with the Commission's Better Regulation Guidelines and assess the effectiveness, efficiency, relevance, coherence and the EU added value of the EU eGovernment Action Plan 2011 - 2015.



2.Background to the EU eGovernment Action Plan 2011 - 2015 2.1.Description of the European eGovernment Action plan 2011 – 2015 and its objectives The Europe 2020 strategy that was put forward by the Commission on 3 March 2010 set out a strategy for smart, sustainable and inclusive growth to face the immediate challenges resulting from the financial crisis and long-term challenges such as resource scarcity, ageing and globalisation. One of the so-called seven flagship initiatives, the Digital Agenda for Europe, outlined a number of actions supporting the further development of eGovernment in the EU. The actions can be found from the eGovernment chapter of the Digital Agenda Communication and from the eGovernment Action Plan that was the successor of the i2010 Action Plan 2006-2010.

The Action Plan 2011 – 2015 aimed to contribute to achieving two important targets of the Digital Agenda in Europe: 1) that 50% of citizens make use of eGovernment services; and 2) that a number of key cross-border services will be available online by 2015. The Action Plan also aimed that "by 2015, 80% of enterprises will have used eGovernment". Furthermore, the objective was to translate into concrete measures the four political priorities of the Malmö Ministerial Declaration on eGovernment of December 2009. Subsequently, the eGovernment Action Plan had the following four policy priorities (see also Figure 1 ): "1. User Empowerment Empowerment means increasing the capacity of citizens, businesses and other organisations to be pro-active in society through the use of new technological tools. Public services can gain in efficiency and users in satisfaction by meeting the expectations of users better and being designed around their needs and in collaboration with them and third parties whenever possible. Empowerment also means that governments should provide easy access to public information, improve transparency and allow effective involvement of citizens and businesses in the policy-making process. 2. Internal Market Most public online services do not work across borders or involve cumbersome procedures to be accessible. People from one EU country cannot easily apply for public services in another country than the one in which they are established, using for instance national electronic identity cards. This seriously reduces the mobility of businesses and citizens. To support the Internal Market, governments should develop ‘seamless’ services for entrepreneurs to set up and run a business anywhere in Europe and allowing individuals to study, work, reside, receive health care and retire anywhere in the European Union. 3. Efficiency and Effectiveness of Governments and Administrations The aim was to use ICT and enabling organisational changes to deliver better, less intrusive, more sustainable and faster public services, by reducing the administrative burden, improving organisational processes and promoting a sustainable low-carbon economy. 4. Pre-conditions for developing eGovernment A number of technical and legal pre-conditions need to be put in place to enable the implementation of the actions that will enhance eGovernment services in Europe. These include the promotion of interoperability across borders, which would allow - among others - sharing of information, deployments of one-stop-shop approaches, Europe wide use of (national) electronic identity solutions and payment schemes. Interoperability is supported through open specifications and the development of key enablers such as electronic identity management and stimulation of innovation in eGovernment." To achieve these objectives, the Action Plan proposed 40 individual actions (listed in Annex 4 to the present document) distributed under each of the policy priorities. According to the Action Plan, "the actions can also be categorised in three groups, depending on the actors involved and their competence defined in the Treaty: 1.Where Member States are leading and rely on their own resources, the Commission will help by supporting and coordinating activities. The measures proposed will focus on setting targets with the Member States and on how to achieve these targets by means of measures such as exchanging best practice and information, conducting studies and benchmarking. 2.Where the Commission and the Member States work jointly to develop, deploy or improve cross-border services, the Commission will take the lead in activities where joint resources are used, while the Member States will bear the final responsibility for implementing activities using their own resources. The measures proposed will include research and development, pilot projects, collaborative development of services by Member States and transfer of knowledge to the market. 3.Where the Commission can create enabling conditions, the measures proposed will include adopting legal instruments, setting standards, formulating common frameworks, implementing generic tools, providing (re-usable) technical building blocks and ensuring interoperability." Of the 40 actions in the eGovernment Action Plan, the Commission was exclusively responsible for 23 and the Member States for 10. Seven actions were under shared responsibility between the Member States and the Commission. Five additional actions, two for the Commission and three for the Member States, were included for the governance.

2.2.Instruments The instruments used to implement the Action Plan were political, financial and legislative activities. 2.2.1.Political Activities including exchange of experience and Commission Communications The political activities refer to dialogue with Member States, Ministerial meeting, exhibitions, awards and benchmarking. Their aim was to provide political steer, enhance mutual learning through exchange of experience and awareness rising. Dialogue with Member States eGovernment High-Level Expert Group of Member States representatives (activity under governance): The Commission and Member States have engaged in collaborating and in regular dialogue during the eGovernment Action Plan 2011 – 2015 through the eGovernment High-Level Expert Group. In addition, there have been other fora to engage with Member States' representatives: including the ISA (Interoperability Solutions for European Public Administrations) Programme Management Committee. Ministerial meetings Ministerial Conferences (together with a series of eGovernment Presidency events/conferences) helped to take stock of where eGovernment was and provide a vision for where it should go in the coming years, mainly in the form of Ministerial Declarations. The 2009 Malmö Ministerial Declaration resulted in the current eGovernment Action Plan. The conferences also provided exhibitions of projects and awards for best eGovernment projects. The 6th Ministerial meeting was held on November 2011 in Poznan. As well as the conference and the exhibition, it featured an Informal Council of Ministers meeting. Benchmarking Since 2001, the EC has been measuring the progress made in on-line public services. This annual benchmarking has become a worldwide reference for international comparison of the development of eGovernment. Since 2012 a new benchmarking framework has been in place, aligned to the four policy priorities of the current Action Plan. The European Interoperability Framework and the European Interoperability Strategy

The European Interoperability Strategy (EIS) and a European Interoperability Framework (EIF) provide guidance in delivering services across borders and sectors. The EIF, in particular, has been translated into National Interoperability Frameworks in more than 20 Member States. It is now being reviewed and extended in the context of the Digital Single Market Strategy. Cloud computing The Commission Communication "Unleashing the Potential of Cloud Computing in Europe", provided a European strategy designed to speed up and increase the use of cloud computing across all economic sectors. ICT standardisation A number of ICT standardisation activities supported the EU policy activities, including the areas of eGovernment, eProcurement, Public Sector Information, Open Data and Big Data. 2.2.2.Funding for eGovernment The Action Plan did not have funding, but served as a guide to prioritise a number of investments from EU funding programmes: The past Competitiveness and Innovation Framework Programme ICT Policy Support Programme (CIP ICT-PSP) has supported (some projects on-going) piloting of cross-border eGovernment solutions and supported piloting of concepts, like the "cloud of public services"; Under the Connecting Europe Facility (CEF), Digital Services Infrastructures (CEF DSI) deploy digital public services that work across borders; Horizon 2020's Societal Challenge 6 'Europe in a changing world – inclusive, innovative and reflective societies' funds research and innovation actions on ICT-enabled public sector innovation; The past Seventh EU Framework Programme for Research and Technological Development (FP7) supported eGovernment-related projects; The ISA programme (Interoperability solutions for European Public Administrations) provides a framework that allows Member States to work together to create efficient and effective electronic cross-border public services for the benefit of citizens and businesses; The European Structural and Investment Funds (ESIF as of 2014) provide investments in the field of eGovernment in less favoured regions, as this is seen as a strategic component of their economic and social development. One of the thematic priorities for the 2014 – 2020 programming period is enhancing access, use and quality of ICT (thematic objective 2), including strengthening ICT applications for e-government. Another thematic objective is on administrative capacity building (thematic objective 11). The exact amounts invested to policy priorities of the Action Plan are in some cases challenging to measure. Sometimes investments were done directly by the Member States without reporting to the Commission. Furthermore, it is not always obvious what would be considered an eGovernment–related investment in programmes with various information technology or information society projects.

2.2.3.Legislation Legislative instruments have been used to achieve the eGovernment Action Plan's objectives where needed. Some of these legislative instruments existed already in the beginning of the Action Plan and needed amendment, while new legislation was also required, these included: The planned revision of the PSI (Public Sector Information) Directive was indicated in the eGovernment Action Plan. On 12 December 2011 presented a proposal in this respect. The revised Directive was adopted on 26 June 2013. The reuse policy of the Commission was revised by a Decision of 2011. The revision of the eSignature Directive and the proposal for a Decision for mutual recognition of eIdentification and eAuthentication were indicated in the eGovernment Action Plan and led to the eIDAS Regulation on electronic identification and trust services (Regulation 910/2014). The update of the Public Procurement Directives presented by the EC in the end of 2011 proposed a gradual but ambitious transition towards e-procurement in the EU. In February 2014, two Public Procurement Directives were revised and a Directive on concession contracts was adopted. The new rules will enter into force as of 18 April 2016. The CEF regulation in the area of telecommunications infrastructure (CEF Telecom Regulation). The reform of the EU data protection legal framework – the General Data Protection Regulation and the Data Protection Directive for Police and Criminal Justice Authorities. Both will be adopted in early 2016 and followed by a two year transition period. The proposal for a directive on the accessibility of public sector bodies' websites, which has not yet been adopted. The new ISA2 programme adopted by the co-legislators in November 2015 on interoperability between public administrations and extended to local and regional administrations and the links to businesses and citizens is a key instrument for the development of eGovernment in Europe. 2.3.Intervention logic As stated in the Staff Working Document of 2010 accompanying the eGovernment Action Plan 2011 - 2015, this eGovernment Action Plan 2011 – 2015 responded to a call from Member States, made in the Malmö Declaration, for a shared eGovernment policy in the European Union in order to build on past achievements and increase collaboration on eGovernment. This included joint action between Member States and close collaboration with the European Commission The simplified intervention logic figure ( Figure 2 ) explains how the different parts of the Action Plan fit together. 2.4.Situation in 2010/Baseline The 2010 Staff Working Document, referred to above, provides an overview of the situation in 2010, focusing at the time on the availability of basic services online: "The online delivery of basic services has continued to increase steadily in recent years: their full online availability went from 21% in 2001 to 71% in 2009. However, there is a marked difference between services for businesses and services for citizens. Services for businesses reach 83% availability whereas services for citizens are still only at 63% availability."

There were considerable discrepancies between Member States. The gap between the best (with 100% availability of basic services) and worst performer (40% availability) was large. The EU average figure was around 70%. Four Member States had a full online availability for all the 20 basic services considered in 2009 in the benchmark measurement report. 2.4.1.Baseline of target: 50% of citizens (Digital Agenda) and 80% of businesses make use of eGovernment services The Eurostat figure of the percentage of Europeans (EU27) aged 16 to 74 who had used the Internet in the last 3 months for interaction with public authorities was 31 % in 2010. If extending to cover the last 12 months, the Eurostat figure collected in 2010 was 41 %. Regarding advanced modes of interacting with public administrations, the Staff Working Document of the 2011 – 2015 Action Plan noted that in 2009 only 17% of EU citizens had downloaded official forms from a public authority website and only 12% had used the Internet to send back completed forms. In 2010 76% of enterprises used at least one eGovernment service. Take-up was also relatively high for advanced ways of interacting with the public administrations: 54% of enterprises used the Internet to return completed forms and 43% interacted with the public administration entirely using electronic transactions. 2.4.2.Baseline of target: a number of key cross-border services be offered online by 2015 (Digital Agenda) The eGovernment Action Plan 2011 – 2015 Communication stated, describing the situation in 2010, that cross-border eGovernment services are few and, even where eGovernment services were offered, the majority of EU citizens were reluctant to use them. (In the Digital Agenda Communication in 2010 no specific baseline was given for the target of "by 2015 online availability of all the key cross-border public services contained in the list to be agreed by Member States by 2011".) In 2010 the work towards cross-border services had started with three Large Scale Pilots (LSPs) co-funded by the EC under the "ICT Policy Support Programme" as part of the Competitiveness and Innovation Framework Programme (CIP ICT PSP). PEPPOL had started in 2008 to work towards a European system of eProcurement; STORK had started in 2008 and aimed to enable citizens and businesses to identify themselves when interacting with their own and other national administrations over the internet; SPOCS had started in 2009 and was related to the Services Directive. Around 36 million Euros investment had been planned for in these three pilot projects at the time. 2.4.3.Baselines in the four policy priority areas Data from 2010 is not available for the four policy priority areas of the Action Plan, because the benchmarking measurement of the four priorities started only with the new method as of 2012 with a first complete measurement available in 2013. Some data relevant to the Action Plan 2011 – 2015 were given in the eGovernment Benchmarking report of 2010 that reported that Europe had made substantial advances in the 20 basic services. For businesses, a traditional 20 services assessment covered eight services for businesses. Out of these, the services ‘Registration of a company’ and ‘Obtaining an Environmental Permit’ were highly available online and displayed sophistication and full online availability scores of 90%/77% and 78%/63% for the EU27+ respectively. Furthermore, the eGovernment Benchmarking report 2011 measured the availability of different enablers in the EU27+ countries ( Figure 3 below) 3.Evaluation Questions/Criteria

In order to structure the evaluation exercise, the current analysis sets out to answer a series of questions that will provide insight into the effects produced by the EU eGovernment Action Plan 2011-2015. The evaluation criteria used are from the EC’s Better Regulation Guidelines: effectiveness, efficiency, coherence, relevance and EU added value Effectiveness analysis considers how successful the eGovernment Action Plan has been in achieving and progressing towards achieving the targets on the usage of eGovernment services and the availability of cross-border services, in addition to achieving the objectives of the four policy priorities on user empowerment, internal market, efficiency and effectiveness and pre-conditions. This criterion assesses the progress made and the role of the Action Plan in delivering the changes and/or the extent to which progress has fallen short of the target and what factors have influenced why something has not been achieved. Have users' needs been sufficiently catered for under the lifetime of the Action Plan? The EU eGovernment Action Plan 2011 – 2015 did not have funding, but it served as a guide to prioritise investment from a number of funding programmes. How does the development and deployment of these projects support the overall aim of the eGovernment Action Plan? The efficiency analysis aims to, to the extent possible; consider the relationship between the resources used by the Action Plan and the changes generated by the funding intervention. Relevance looks at the relationship between the needs and problems in society and the objectives of the EU eGovernment Action Plan: Was the vision in the Malmö declaration on which the eGovernment Action Plan is based appropriate for developing the EU eGovernment 2011 - 2015? Do the original objectives still correspond to existing needs within the EU, which of the objectives of the Action Plan continue to be still relevant and identifies also some new or changed needs. The evaluation of coherence looks at how well different actions of the eGovernment Action Plan work together (internal coherence) and how the Action Plan relates to activities outside the Action plan (external coherence), including other EU policies. To what extent have Member States developed eGovernment services and policies in line with the overall principles set out by the current eGovernment Action Plan? What is the EU added value resulting from the EU intervention compared with what could be achieved by Member States at national and/or regional levels? This assessment relates to the added value of the EU eGovernment Action Plan, why action at the EU level was necessary and what would have happened without an Action Plan? Which of the eGovernment policy priorities could not have been achieved without an EU-level Action Plan?



4.Method and sources The evaluation has been coordinated by the EC's Directorate-General Communications Networks, Content and Technology with the support of an Inter-Service Group (with representatives of Commission Directorate-Generals Agriculture and Rural Development; Communication; Competition; Informatics; Economic and Financial Affairs; Employment; Social Affairs and Inclusion; Energy; Environment; Internal Market, Industry, Entrepreneurship and SMEs; Migration and Home Affairs, Joint Research Centre; Publications Office; Justice and Consumers; Mobility and Transport; Regional and Urban Policy; Research and Innovation; Health and Food Safety; Secretariat-General; Taxation; and Customs Union and Trade), which steered and monitored progress of the exercise, ensuring the necessary quality, impartiality and usefulness of the evaluation. For the evaluation, a broad literature review was conducted covering various reports, EU studies and a number of international references. Important sources in the desk research were the mid-term evaluation of the eGovernment Action Plan 2011 - 2015 and the annual eGovernment Benchmarking reports. Furthermore, additional information and data (listed in the in the footnotes and in the "References and data sources" part in the end of this document) were used to answer the evaluation questions. Their methods are available in the original referenced reports. The eGovernment factsheets published on the Joinup platform give an overview of eGovernment in each Member State. A 12 week online public consultation was conducted to collect primary data and complemented by statistical data (Eurostat data and Digital Economy and Society Index DESI) and Digital Agenda Scoreboard reports. An effort was made to “triangulate” the data used throughout the analysis. This means that findings presented in the evaluation are supported by evidence from different data sources whenever additional data was available. Any contradicting evidence has been weighed according to its strength and quality before reaching conclusions. 4.1.Mid-term evaluation The first monitoring of the implementation of the EU eGovernment Action Plan was done through the mid-term evaluation that carried out a monitoring and stocktaking exercise (progress assessment), with the purpose of providing an overview of the extent and modalities of implementation of the eGovernment Action Plan, its four priorities, 40 actions and governance chapter. The Mid-term evaluation of the eGovernment Action Plan was carried out between December 2013 and June 2014 by external contractors. The eGovernment Action Plan states that "A midterm evaluation of implementation of this Action Plan will be conducted in 2013. "

4.2.Annual benchmarking reports Since 2001, the Commission measures the progress made in the availability of on-line public services.. In 2012, this benchmarking was aligned with the priorities of the eGovernment Action Plan 2011-2015. The benchmarking method examines a set of indicators related to each of the four political priorities of the Malmö Ministerial Declaration and the Action Plan. The latest benchmarking report dates from June 2015. This 12th eGovernment Benchmark report is the third edition of the measurement made according to the new eGovernment Benchmark Framework 2012-2015 . Unless otherwise specified, the benchmarking report references in this evaluation refer to the "EU28+", which refers to 33 European countries: the EU, Iceland, Norway, Serbia, Switzerland and Turkey. With regard to the methodology used for the 2015 benchmarking report, each of the priority areas of the eGovernment Action Plan was assessed by one or more top level benchmarks. Two methods were used for data collection: Mystery Shopping and User Survey. Benchmarking report Year of data collection eGovernment Benchmarking report 2012 (May 2013): Public Services Online. "Digital by Default or by Detour" 2012 eGovernment Benchmark Report 2014 -11th report (June 2014): Delivering on the European advantage? "How European governments can and should benefit from innovative public services" 2013 eGovernment Benchmarking Report 2015 (June 2015): Future-proofing eGovernment for a Digital Single Market. 2014 eGovernment Benchmarking Report 2016 (to be published) 2015 4.3.The online public consultation The online public consultation gathered input from citizens, businesses and civil servants over 12 weeks from 30 October 2015 till 22 January 2016. Its objective was to inform and help define the new eGovernment Action Plan 2016 - 2020. The consultation was promoted through the eGovernment network (comprising 5000 entities), the EC website and on Twitter. The questionnaire also addressed the EU eGovernment Action Plan 2011 – 2015 as regards the improvement of cross-border eGovernment services and the success of the four political priories. The respondents were asked to provide answers in the form of agreeing to predefined statement on a scale from "not likely" to "very likely", in addition to a few open questions. The results are provided in annex 2. 4.4.Limitations The evaluation faced some limitations in the collection of data, whose impact was mitigated to a maximum possible extent: Measuring the impact of the eGovernment Action Plan is a challenging exercise as it is almost impossible to isolate the impact of the Action Plan from other developments in the public sector, such as ICT-policies or the increased use of the Internet in general. Furthermore, some objectives of the Action Plan, such as increased collaboration, are non-tangible and, as a result, difficult to quantify. The evaluation has used benchmarking to measure the progress in the different policy priority areas of the Action Plan and assessed the effectiveness criteria qualitatively. The Action Plan did not have a dedicated budget but relied on funding from other programmes for a large part of its actions. Other actions were financed by the Member States. As a result, the efficiency analysis (in particular the value for money assessment) was difficult to carry out. Despite being prompted in a number of occasions by the EC, there is no systematic measuring or reporting requirement for the Member States of cross-border public services use. To overcome this, the evaluation used data from other sources, including case studies. Given the multiplicity of the tools used to gather evidence, the results obtained are of different nature (for instance, eGovernment related data stems from the mid-term evaluation of the Action Plan and the benchmarking reports while the online public consultation provided insights in order to prepare for the new eGovernment Action Plan 2016-2020). The evaluation takes into account the inherent limitations of the findings of public consultations. Firstly, as in all surveys, the answers received reflect the views of a sample of relevant stakeholders and not those of the entire population who has a stake in this domain. Secondly, stakeholders' views convey an individual rather than a holistic perspective. The mid-term evaluation did not systematically cover the five evaluation criteria. Furthermore, the methodology had strict requirements. For an action to qualify as "delayed" it was sufficient that one indicator is negative (e.g. one Member State answered ‘no’ to the question, or one action was delayed in one Member State) can sometimes give only a partial overview of progress and it should be understood as an evaluation of the actions as defined by the action plan, not an assessment of the policy priority as such. In addition, the mid-term evaluation did not capture the last 18 months of the Action Plan. This is partly mitigated using the 2015 benchmark study and partly the evaluation questions in the 2015/2016 online consultation on the eGovernment Action Plan and other recent reports and information. Based on the elements above, the evaluation has been carried out on the basis of the best available data. Whenever reliable quantitative data is lacking, this is indicated as appropriate and possibly counter-balanced with qualitative data and considerations.



5.Implementation of the EU eGovernment Action Plan 2011 – 2015 – state of play of the policy priorities The monitoring of the implementation of the Action Plan was carried out by mid-term evaluation and supported by the annual eGovernment benchmarking. An assessment of the completion of actions will be done separately in a planned 2016 study on the assessment of the implementation of the Action Plan. The benchmarking reports are not intended to act as a benchmarking of the implementation of the Action Plan as such, but its results are informative for monitoring the progress of eGovernment in the EU. 5.1.General target: 50% of citizens (Digital Agenda) and 80% of businesses make use of eGovernment services In 2014 eGovernment services were used by 47% of the EU population (38 % in 2009). The Eurostat figure for individuals (aged 16 – 74) having used the Internet for interaction with public authorities within the last 12 months was 46 % in 2015 (41 % in 2010). eGovernment usage by firms (EU 28) has increased steadily from 76% in 2010 to 84% in 2011 and 88 per cent in 2013. 5.2.General target: a number of key cross-border services be offered online by 2015 (Digital Agenda) The eGovernment Action Plan was to contribute to two key objectives of the Digital Agenda: by 2015 a number of key cross-border services will be available on line – enabling a) entrepreneurs to set up and run a business anywhere in Europe independently of their original location, and b) allowing citizens to study, work, reside and retire anywhere in the EU. The Digital Agenda had specified that the key cross-border public services would be contained in a list to be agreed by Member States by 2011. The identification of the key cross-border services was discussed at the Ministerial Roundtable in Poznan (Poland) in 2011. EU Ministers responsible for eGovernment discussed a number of areas of the Digital Single Market from which potential cross-border services could identify for future roll out. The conclusions noted that:

The Ministers concluded that cross–border eServices should be built from components that can be shared and re–used. The Large Scale Pilots have already contributed to this goals by developing building blocks. The Ministers concluded that at least five key cross–border services should be selected and implemented between 2012 – 2014 where possible domains have been put forward for consideration. These cross–border eServices should be built from components that can be shared and re–used. The Member States' eGovernment Expert Group later agreed to endorse a number of focus areas and key enablers for digital cross-border public services. Because of divergent views, Member States did not reach formal consensus on the exact services. However, the work continued through the The Competitiveness and Innovation Programme ICT Policy Support Programme (CIP ICT-PSP) LSPs (STORK, SPOCS, PEPPOL, epSOS, eCodex and eSENS) towards the development of the building blocks that can be used by Member States to produce their own cross-border services according their needs. Most of the Member States have participated with co-funding in one or several of these projects to develop cross-border solutions for the public services. At the mid-term evaluation it was noted that, "while not all Member States have identified their priority key cross-border public service, and not all Member States have implemented or are planning to implement cross-border services, there are several examples of Member States implementing cross-border services." To support the work on cross-border services a detailed studywas concluded by the Commission in 2013. The study identified the cross-border services with the highest potential impact. In determining these services Member States were recommended to refer to the PESTEL (Political, Economic, Social, Technological, Environmental and Legal factors) analysis and estimated volume of users. Furthermore, the study noted that quick wins for Member States can be achieved by re-using the building blocks from the CIP ICT-PSP large-scale pilots.

The CEF Telecom Regulationacknowledges the role of the CIP LSPs such as "PEPPOL, STORK, epSOS, eCODEX or SPOCS, in validating key cross-border digital services in the internal market, based on common building blocks, which are being consolidated by the project eSENS". The regulation's annex specifically identifies a number of building blocks: Electronic identification and authentication Electronic delivery of documents Automated translation Critical digital infrastructures support Electronic invoicing The CEF building blocks that are now available are: eID, eSignature, eDelivery, eInvoicing and Automated Translation. The deployment of eDelivery, eID and eSignatures at the end of 2015 is described in the table below (excluding countries outside the EU and EEA): eDelivery

Countries that have deployed an eDelivery Access Point eID

Countries that are connected to the eID network via STORK eSignatures Austria Belgium Czech Rep. Denmark Finland France Germany Greece Iceland Ireland Italy Netherlands Norway Spain Sweden UK Austria Belgium Czech Rep. Estonia Iceland Italy Latvia Lithuania Luxemburg Portugal Slovakia Slovenia Spain Sweden The trusted list enabling the use of eSignature as part of the trust backbone has been deployed in the 28 Member States (as required by the regulation). Furthermore, the "Cross Border Mobility" benchmark of the eGovernment Benchmarking indicates to what extent European users can use online services in another country. This top level benchmark, which is split into Citizen Mobility and Business Mobility, measures the availability and usability of cross border services in a number of life events under four sub-criteria: availability, usability, ease of use and speed of use. According to the eGovernment Benchmarking Report 2015, across Europe, on average, the score for the indicator of cross-border mobility for business is better than the citizen mobility indicator (58% as opposed to 43% for citizens). There is some improvement from the previous measurement (2012/2013, data not available earlier) from, respectively, 53% and 38%. According to the eGovernment Benchmarking Report 2015, only 57% of the assessed services (life events) are available to cross border businesses and only 41% are available to cross border citizens in 2014. This represents an improvement of 5 points for both groups in the assessed life events (compared to previous measurement 2012/2013, data not available earlier). 5.3.The four priorities and 40 actions of the EU eGovernment Action Plan The 40 actions of the Action Plan (listed and numbered in Annex 4 to this document) were grouped under four priorities. The answers of each Member State (that answered to the survey) for each of the action in the mid-term evaluation of the eGovernment Action Plan are provided in a separate dashboard. The last updates on the dashboard were in the first quarter of 2014. Non-implementation by one Member State was sufficient to mark an action either as delayed or at risk of delay. The mid-term evaluation was finished in June 2014. The detailed implementation status of the 40 actions is provided in Annex 4 to this report. The status is provided on basis of the mid-term evaluation report findings. A number of actions were completed after the end of the evaluation and new progress in Commission actions is reported in Annex 4; detailed data from each Member States' progress after the mid-term evaluation was not available. The mid-term evaluation concluded that of the 40 actions, 16 were completed at the time and six were on track. Three actions were at risk of delay and 15 delayed. The delay meant that an action had not been completed by one or several Member States or Commission within the deadline year and there was thus a delay from one to two years. The reasons and the possible impact (where an analysis was already possible) of these delays are assessed under the evaluation questions in part seven. The large majority of the EC’s 30 Actions were completed or were on track. Of these, there were 23 for which the EC has exclusive responsibility, while, for the remaining seven, responsibility was shared with Member States. 5.4.Governance The High-Level Expert Group of Member State representatives was set up, and rules of procedures were adopted in 2011 (action 41).

According to the mid-term evaluation (2014) of the Action Plan, almost all Member States have a National eGovernment Strategy, which incorporates priorities from the Malmö Declaration and the eGovernment Action Plan (action 42). Only two Member States had not yet updated their national eGovernment strategies. According to the mid-term evaluation, most of the Member States (17) have incorporated the political priorities of the Malmö Declaration and eGovernment Action Plan in their national strategies (action 43) and only four countries indicated in 2014 that this action was delayed in their case. A mid-term evaluation of the Action Plan was carried out and its results and the results of this evaluation accompany the Communication on the EU eGovernment Action Plan 2016 – 2020 (action 44). Furthermore, Member States have informed the EC and the High-Level Expert Group on how the political priorities of the Malmö Declaration have been achieved, while six Member States (of the twenty-four responding in the mid-term evaluation) have not yet done so. Many countries have already provided some lessons learned within the implementation process (action 45).



6.Answers to the evaluation questions 6.1.Effectiveness The effectiveness analysis assesses how successful the EU eGovernment Action Plan has been in achieving or progressing towards its objectives. What was the role of the EU eGovernment Action Plan 2011 – 2015 in delivering the changes? What factors have influenced the progress? To what extent progress fell short of the target? The objectives to be analysed in this part are, first, the two targets of the Digital Agenda in Europe: (1) 50% of citizens make use of eGovernment services by 2015; and (2) that a number of key cross-border services be offered online by 2015. The Action Plan also aimed that "by 2015, 80% of enterprises will have used eGovernment". Secondly, the analysis includes the four policy priorities. The mid-term evaluation of the Action Plan described a wide range of eGovernment effects across the different Member States with various case studies as examples, including examples in the area of user empowerment, open data, interoperability, eID and transparency. However, the eGovernment Action Plan was a strategic plan to give the policies of EU Member States a direction towards commonly agreed goals. Without a detailed individual assessment of each action in each Member State it is not possible to associate the quantitative and qualitative effects on the Member States level directly to the Action Plan. 6.1.1.Objective on using eGovernment (take-up) The 80% target for eGovernment usage by firms (EU 28) was achieved already in in 2011 (84%) and it grew to 88 per cent in 2013. The action plan objective of "By 2015, 50% of EU citizens will have used eGovernment" was nearly achieved. In 2014 eGovernment services were used by 47% of the EU population. The Eurostat figure for individuals (aged 16 – 74) having used the Internet for interaction with the public authorities within the last 12 months was 46 per cent in 2015. The achievement of the full 50 % usage by citizens was still expected in 2011. In the Digital Agenda Scoreboard report 2011 it was assessed that "the take-up of eGovernment in 2010 has reached 41 % of the EU population and, given the current trends, could be quite in accord with the target of the Digital Agenda of 50% of population using it by 2015." At the same time, the rise of the use of eGovernment services has also been driven by the increase of regular Internet users. Finding 1 The Action Plan target of by 2015, 80% of enterprises will have used eGovernment was achieved early into the Action Plan. The target of by 2015, 50% of EU citizens will have used eGovernment was nearly achieved (46% in 2014).

6.1.2.Key cross-border services online During the Action plan duration, cross-border eGovernment services were addressed in the high-level political discussions and in the work of the LSP projects. This was a direct result of the eGovernment Action Plan. The Action Plan played a key role as a basis and political framework for bringing Member States together to discuss and agree on these initiatives. The Action Plan objective was to contribute towards fulfilling the Digital Agenda objective: "By 2015, a number of key cross-border services will be available on line – enabling entrepreneurs to set up and run a business anywhere in Europe independently of their original location, and allowing citizens to study, work, reside and retire anywhere in the European Union." The contribution towards this objective was achieved by LSP projects implemented under the CIP ICT PSP programme. The evaluation of the ICT PSP Programme underlined, among other aspects, its innovative role as its projects design new platforms for innovative cross-border services in areas of public interest, in particular in areas where there are continuing systemic and organisational risks. Finding 2 The Large Scale Pilot projects had an innovative role as projects designing new platforms for cross-border services in areas of public interest. The activities on cross-border services went beyond the Action Plan's objectives. Through the financial support from the CIP ICT-PSP Programme to the LSPs' work on what has become the CEF building blocks, the Action Plan has contributed to achieving also regulatory objectives: the eIDAS Regulation and transition to the electronic public procurement. The eIDAS Regulation got important leverage from the STORK project that piloted an interoperability solution and that is referred to in the Regulation recitals. In the new public procurement directives, e-procurement was one of the important changes introduced. The PEPPOL LSP (Pan-European Public Procurement Online) sought to make it easier for companies to bid for public sector contracts throughout the EU and provided supporting experiences and solutions for cross-border e-procurement. In the context of cross-border services, the Points of Single Contact, required by the Services Directive, cover an increasing number of administrative formalities for businesses. The SPOCS LSP (Simple Procedures Online for Cross-Border Services) sought to improve the cross-border electronic procedures for businesses. In the environmental area, the EC has developed an EU "Shared Environmental Information System" (SEIS). The aim of the Action Plan was to contribute towards the availability of cross-border services. To this date, life events for businesses and citizens are not yet covered by cross-border services and not all Member States have rolled-out the building blocks. For example, the objective that entrepreneurs should be able to set up and run business anywhere in Europe independently of their original location has not been met. In the online public consultation (Annex 2) the majority of the respondents did not know whether the current eGovernment Action Plan has improved cross-border eGovernment services overall or did not agree that the Action Plan has improved them services. Around 25% of the respondents agreed that the Action Plan has improved cross border services overall. Finding 3

The eGovernment Action Plan has shown effectiveness to mobilise resources in particular in areas where the cross-border services objectives have been shared between both EU technical/operational and regulatory measures, e.g. cross-border eID and e-procurement. However, take-up has varied between Member States: entrepreneurs are not yet able to set-up and run a business anywhere in Europe independently of their location nor are citizens able to complete administrative procedures cross-border for all life events on-line. As reported under point 6.2, according to the eGovernment Benchmarking Report 2015, only 57% of the assessed services (life events) are available to cross border businesses and only 41% are available to cross border citizens. 6.1.3.Effectiveness of the four policy priorities The progress in four of the main areas (User empowerment, Internal Market and pre-conditions, key enablers/pre condition) is visible in the below figure ( Figure 6 ) from the 2015 eGovernment Benchmark background report "Future-proofing eGovernment for the Digital Single Market". 6.1.3.1.Effectiveness of the User Empowerment policy priority In the Action Plan, empowerment was referring to increasing the capacity of citizens, businesses and other organisations to be pro-active in society through the use of new technological tools. Governments should provide easy access to public information, improve transparency and allow effective involvement of citizens and businesses in the policy-making process. According to the mid-term evaluation, the actions under the User Empowerment priority require further attention. Important targets have been achieved in implementing inclusive services, in involving citizens and business in the policy-making process, and notable progress has been made in creating collaborative services. There are also good achievements of the Action Plan targets under sub-priority on re-use of public sector information. At the same time, there are delays in general in drawing up and adopting common targets, indicators and measurement frameworks. Details of the achievements under each of the 14 actions under this priority are in Annex 4. Services designed around users' needs and Inclusive Services The Action Plan has supported exchanges of practice in the field of user-centred, inclusive and accessible eGovernment services (through the Joinup platform), though no agreement had been reached with Member States on common targets and evaluation criteria for those services. The Commission adopted its proposal for a directive on the accessibility of public sector bodies' websites on 3 December 2012. The MeAC (Measuring progress of eAccessibility in Europe) reports provided evidence and analysis to help understand and compare the approaches followed by the European countries, with a view to identifying issues and challenges, good practices and future priorities in the web accessibility field. According to the mid-term evaluation report, most Member States have introduced personalised services in their One-Stop-Shops for business and for citizens, as well as in the area of health and tax services. The majority of the Member States that have already introduced personalised services offer them via multiple channels, such as mobile access via mobile apps and helplines. Re-use of Public Sector Information (PSI) Under the Action Plan priority on re-use of public sector information, among other things (details in Annex 4), the Commission has reviewed and subsequently revised the PSI Directive improving re-usability of data held by Member States' authorities and has put in place an Open Data Portal for data of the EU institutions. The portal, launched in 2012, has currently more than 8000 datasets available.

The Action Plan achievements under the sub-priority Re-use of Public Sector Information (see Annex 4 and achievements under Action 6, 7 and 8, including the review of the PSI Directive and the set-up of the Open Data Portal) are good, apart from the formal achievement on agreement on indicators. The Action Plan has supported the achievements and has given them political visibility. There appears to be increased awareness of the importance, including economic importance, of open data as shown, for example, by the adoption of the G8 Open Data Charter and the commitment of various EU Member States to the Open Government Partnership. The study "Open Data Maturity in Europe 2015" reported that 27 countries (of EU28+) have a national Open Data portal and an open data policy is in place in 71% of the countries, often as part of a more generic Digital Strategy or eGovernment programme. It is also worth noting that beyond the objectives of the Action Plan, but supporting the sub-priority on re-use of public sector information, the EU endorsed the G8 Open Data Charter in 2013 and committed to promoting the application of the principles of the G8 Open Data Charter to all EU Member States within the context of a range of ongoing activities. Improvement of Transparency On the sub-priority on Transparency the mid-term evaluation found that the actions under transparency were delayed. Fourteen Member States were using some transparency targets and most Member States provide online access to information on government laws and regulations (see Annex 4 for details). Member States also provide access to citizens' to their personal data, but not all provide it in electronic form as seen in the Action Plan (30% of Member States did at the time of the mid-term evaluation). At the same time, the eGovernment benchmarking report 2015 stated that the transparency benchmark shows improvement from the previous measurement, but is still quite unsatisfactory, as it is at 51%. The Transparency indicator of the benchmarking report examines the extent to which governments are transparent about their own responsibilities and performance, the service delivery process, and the personal data involved. It is also positive that users have gained better access to personal data that is handled on the governments’ websites, but they still face considerable barriers when it comes to the clarity of the service delivery process. The Action Plan supported the delivery of a number of transparency objectives, but this remains an area for continued work. Involvement of citizens and businesses in policy-making processes In the sub-priority Involvement of citizens and businesses in policy-making processes progress is very good according to the mid-term evaluation (see Annex 4 for details). The Action Plan supported and gave a guiding framework for the work for governance and policy modelling under the Seventh EU Framework Programme for Research (FP7) and for eParticipation projects that were supported under the Competitiveness and Innovation Framework programme (CIP). Finding 4 The Action Plan delivered mixed results regarding its contribution to the 4 policy objectives.

While it was effective in achieving and contributing to targets in inclusiveness, re-use of public sector information and in involving citizens and businesses in the policy making processes – particularly where both voluntary and regulatory actions at EU-level shared the same objective such as in re-use of public sector information -, it did not fully meet its objectives under the sub-priority on transparency where the benchmarking report findings indicate that the transparency of governments could be future improved. Not reaching an agreement on common targets was reported by the mid-term evaluation to be an issue requiring further attention in relation to the collaborative production of services, PSI re-use indicators and targets and indicators on transparency. When looking into reasons of why common targets or indicators were not agreed in the different areas, it should be noted that sometimes similar activities were already taking place elsewhere, sometimes making the agreement under the Action Plan unnecessary, or similar issues were discussed elsewhere: The Web Accessibility Directive proposal (2012) aims to approximate the laws, regulations and administrative provisions of the Member States on the accessibility of websites of public sector bodies. A directive proposal goes beyond the (non-binding) Action Plan target of agreeing on common targets. For the collaborative production of services, workshops were organised to discuss recommendations but these did not lead to agreement on common targets for the roll-out. For the PSI re-use indicators, the majority of the Member States agreed as to the usefulness of the indicators of the PSI Scoreboard. However, given the existence of other benchmarking tools (Global Open Data Index, Open Data Barometer) and the fact that the PSI Directive does not mandate adopting specific indicators, it was difficult to ensure a uniform adoption of any given set of indicators for all EU Member States. Furthermore, the Europan Data Portal, has developed its own approach and conducts a 'landscaping study' every year to measure open data maturity across Europe. For transparency, the mid-term evaluation found that some Member States were using the Open Government Partnership indicators. Finding 5 A number of common targets and indicators were not agreed by MS due to similar work and agreements made in other fora. The Action Plan could not be adjusted to take this into account. Looking beyond the specific actions of the eGovernment Action Plan to the broader development of user empowerment in the EU with regards to eGovernment services, the eGovernment Benchmarking Report 2015 indicates that governments’ efforts to improve the quality of the online experience have focused on increasing the availability of their online services and on improving the mechanisms for online support and feedback. Users are also more empowered with medium increases in User Centricity and Transparency. Their performance is weaker at the user friendliness and time efficiency of the online services.

The eGovernment Benchmarking report 2015 indicates that tests of 'user centricity' (to what extent a service is provided online and how it is perceived) show good results with a score of 73 out of a possible 100 points across the EU28+. There is, however, a big difference between the compound indicators of user centricity, with much better performances for usability and online availability of services than for the ease and speed of using those services. While availability was up six points in 2014 (48% of all services are completely online and available through a portal), ease and speed of use did not change between 2012 and 2014. This indicates that many Member States are not focusing enough on the quality of the user’s experience. This again suggests, according to the 2015 benchmark report, a trend of quantity over quality. The differences between citizens and businesses are not too large, suggesting that while there are significantly more information and services available to businesses, the quality of these does not differ much from citizen oriented eGovernment. The online public consultation results reflect low awareness about the work carried out in the User Empowerment area. For User Empowerment, 27% of the respondents assessed that the measures had been successful; 48% said that these measure of the action were not successful and 24% did not know. It should be noted that the Action Plan activities were typically not directed directly towards citizens and businesses, but activities were carried out by the Member States and the Commission with a view to, for example, creating enabling conditions which are not directly visible to individual citizens as eGovernment Action Plan activities. 6.1.3.2.Effectiveness of the internal market policy priority The internal market dimension of the Action Plan covers some of the most relevant accomplishments in the advancement of the cross-border delivery of services for citizens and businesses. The Action Plan supported the internal market objective through activities in "seamless services for businesses", "personal mobility" and in "EU-wide implementation of cross-border services". The detailed results are included in annex 4 of this report. According to the mid-term evaluation of the Action Plan, most of the Member States have incorporated the political priorities of the Malmö Declaration, which is the basis of the Action Plan, in their national eGovernment strategies. As a consequence, aspects of internal market related eGovernment activities have been included in national strategies. The role of the Action Plan in driving the cross-border services was discussed above in 7.1.2. The effectiveness of the work of the LSP projects was acknowledged also in the Commission report on "Evaluations of the Competitiveness and Innovation Framework programme" where the evaluation panel recommended that the top-down policy-driven approach in the LSP projects (Pilot A) should be continued, as it is a working model of how to enhance more widespread uptake of new innovative services at EU level. Also in e-procurement the achievements went beyond the objectives of the Action Plan. In addition to the objective stated in the latter, the mandatory transition to e-procurement is driven by the new procurement directives and facilitated with a wide array of policies and support measures. Finding 6 Due to the lack of flexibility of the Action Plan, it did not adapt to changing or new policy priorities and this reduced its effectiveness in terms of setting new and advanced priorities for the Member States and the European Commission. Consecutive assessments of the Points of Single Contact in 2011/12, 2013 and 2014/15 show continuous but slow progress towards fulfilling the requirements of the Services Directive and becoming an effective tool for further integration of the Single Market. The most visible improvement has been made with regards to the availability of online procedures, although the gap is still significant. Furthermore, efforts to improve general usability of the web portals and accessibility to foreign users are recorded, but still insufficient to produce needed effects. As for the effectiveness of cross-border eEnvironment services, the Mid-Term technical evaluation on the implementation the INSPIRE Directive lists a number of effective eEnvironment cross-border services already at the mid-term implementation stage of INSPIRE. It emphasises that effective cross-border eEnvironment services will depend to a large extent on further ensuring the coherent and timely implementation of INSPIRE across the Member States in the 2016-2020 period. Going beyond the actions of the Action Plan, the eGovernment Benchmarking Report 2015 measured the cross-border mobility of businesses through studying the life event "Starting up a business". Scores on all four indicators (speed and ease of use, online availability and usability) are generally much higher than those for other life events. The score on Usability is excellent at 81% but the scores for Online Availability, Ease of Use and Speed of Use leave room for improvement.

Personal mobility In the Action Plan sub-priority on personal mobility activities have been undertaken to support exchange of good practices and to coordinate the Member States’ efforts to jointly develop and set-up cross-border and interoperable eDelivery services. The Action Plan set-out the framework and commitment for this voluntary work. According to the mid-term evaluation of the eGovernment Action Plan, more than 50% of the Member States already provide or are planning to provide those services. The CEF Telecom programme has financed the deployment of the cross-border eDelivery service. According to the eGovernment Benchmarking Report 2015, eGovernment services are still far from giving adequate support to citizens moving to another EU country. The cross-border mobility compound benchmark (beyond the actions of the Action Plan)_measures the ease and speed of use, online availability and usability of online public services for nationals of another EU country giving a score of maximum 100. This stands at 58% for businesses and at 43% for citizens. Although there is improvement from previous year's benchmarking report (from, respectively, 53% and 38%), these figures are lower than the corresponding ones for domestic services. Cross-border transactional services (where a citizen can complete an entire process online) are rare. At the same time, significant increases of both the Business Mobility and Citizens Mobility score of the benchmarking report indicate that it is gradually becoming easier for people working across borders to access eGovernment services. Finding 7 In the Action Plan's internal market priority, the cross-border delivery of public services for citizens and businesses in a number of sectors identified in the Action Plan was advanced. In e-procurement, the achievements go beyond the objectives of the Action Plan, since similar to findings 2 and 3, both EU voluntary and regulatory actions reinforced each other. However, availability of cross-border public services for citizens and businesses remain still insufficient for a fully functioning Internal Market. 6.1.3.3.Effectiveness of the "Effeciency and Effectiveness" policy priority According to the eGovernment Action Plan mid-term review significant progress was being made in improving organisational processes with a "great deal of activity within Member States both at national and local level, focusing on better coordination and cooperation across different public organisations, and therefore improved governance". The Action Plan played an essential role in supporting the exchanges of experience and the sharing of new approaches through the Expert Group meetings, workshops and the Joinup (ePractice) portal. The programme for staff exchanges between administrations in different Member States was not carried out given the current budgetary cuts due to the economic situation. In the area of the reduction of administrative burden, the Action Plan included a number of actions related to the once-only principle. The sharing of experiences was done in a dedicated workshop and at the High-Level Expert Group meetings. A cost-benefit analysis study was carried out and it gave important input to continuing the work on the once-only principle. The study provided also a roadmap for implementation of the once-only principle. The once-only policy gained further momentum through its inclusion in the Council Conclusions 2013 . This supported further steering of Commission policy and resources towards the once-only principle. The principle was included in the Digital Single Market strategy and a call for a new LSP project focusing on EU-wide once-only principle for businesses was included in the H2020 Work Programme 2016 – 2017 adopted in 2015. The effects of the Action Plan in went beyond the original objectives, and were influenced by the increased political visibility. In the area of Green Government, the Action Plan did not succeed in achieving its objectives in assessing the possibilities of eGovernment in reducing the carbon footprint. The study was not carried out due to insufficient data (study on the potential of eGovernment to reducing carbon footprint of government). The mid-term evaluation also found that although there were interesting developments in the area of sustainability, most Member States did not have indicators or evaluation procedures in "Green Government". An agreement on indicators and evaluation procedures for measuring the reduction of carbon footprint as a result of eGovernment between Member States would not have been realistic. In 2015 the REFIT Fitness Check of the environmental reporting process was launched contributing to the reduction of the carbon footprint of governments. It aims to “Launch a broad review of reporting requirements to see how burdens can be alleviated. This review will have a particularly strong focus on areas where stakeholders have recently indicated their concerns, such as agriculture, energy, environment and financial services." With regard to environment the technical Mid-term evaluation report on the INSPIRE implementation emphasised the need to prioritise in the further implementation period of INSPIRE the connection between EU environmental reporting obligations and INSPIRE so as to reduce overall administrative burden and make reporting easier and more efficient for the Member States. Finding 8 The Action Plan was effective in contributing to significant progress in improving organisational processes. The once-only principle policy gained political visibility from related Council Conclusions, which increased the overall effectiveness of the Action Plan priority. The reduction of carbon footprint continued to have a high visibility during the duration of the Action Plan, but in many Member States it was not linked specifically to eGovernment services and the Action Plan did not reach its objectives in these actions. 6.1.3.4.Effectiveness of the preconditions policy priority The EU eGovernment Action Plan's policy objectives for the preconditions priority supported putting in place technical and legal pre-conditions for eGovernment. The mid-term evaluation noted that coordination of national efforts to support the development of interoperable solutions and key enablers is ongoing. The actions that have been supported by the Action Plan include:

European projects (such as PEPPOL and other LSPs) piloted shared cross-border solutions that many Member States are implementing; The alignment of Member States with the European Interoperability Framework (EIF) (combined with a revision of the European Interoperability Strategy (EIS)); The European Interoperability Framework promotes and supports the delivery of European public services by fostering cross-border and cross-sectoral interoperability and has strongly contributed to the Action Plan objectives in the preconditions priority. The adoption of a Regulation on Mutual recognition of Identification and Authentication; with the eIDAS Regulation and the STORK solution the eGovernment Action Plan proved very effective. To complete the legal framework, the Commission adopted several implementing acts under eIDAS Regulation. Launch of pilot projects for innovative architecture and technologies in eGovernment; under sub-priority Innovative eGovernment, the mid-term evaluation assessed the progress as very satisfactory. On-going work on the deployment of the CEF building blocks. The Action Plan pushed Member States administrations towards the deployment of the pre-conditions, as shown, for example, in the EIF alignment figures ( Figure 7 ) below and in the eID roll-out figures that indicate that 13 Member States are connected to the eID network via STORK (details are in Annex 4 under action 37). However, in the online public consultation, the majority of the respondents did not know the measures in the pre-conditions area or rated them not successful. At the same time, 46% of public administrations that responded to the survey rated the measures related to the preconditions for developing eGovernment successful. As an example of effective work supported by the Action Plan, the Member States national interoperability frameworks were aligned to the EIF ( Figure 7 ). This has helped in achieving a common understanding of interoperability in the public sector and how to achieve it. Some of the above interoperability activities, like the implementation of the EIF, STORK or PEPPOL were supported also by the ISA programme which has also funded various interoperability activities. It has contributed substantially also to the implementation of a number of activities in the Maintenance and Implementation Work Programme of INSPIRE. INSPIRE is an important element in cross-sector interoperability between public administrations. The eGovernment benchmarking report 2015 found that in the key enablers measured in the report (measuring also enablers beyond the Action Plan), there was only a marginal gain in 2013/2014 compared to 2012/2013 in the EU28+. Europe has been slow in terms of adopting key enabling technologies for supporting public online services. Some Member States score well with some of technologies but less for others showing that countries have different priorities in the adoption of these enablers. Most countries showed almost no progress from 2012-13 to 2013-14. The most common of these technologies (eIdentification) was deployed in 63% of the cases examined in the eGovernment benchmarking report in 2014. Finding 9 The European Interoperability Framework (EIF) has strongly contributed to the Action Plan objectives in interoperability. In the Action Plan's key enablers work, a legal framework (eIDAS) was provided on electronic identification and trust services and the eID rollout of eIDAS compatible technical solutions is progressing, but mostly within certain Member States, and only in certain cases across borders. 6.1.3.5.Effectiveness of the "Governance" actions of the Action Plan Important effects of the Action Plan arise from the governance chapter of the eGovernment Action Plan. These included the creation of an eGovernment High-Level Expert Group and the update of national eGovernment strategies incorporating the principles of the Malmö declaration. Actually, in most Member States, the national eGovernment strategies incorporate the Malmö priorities (see above 6.4). The effect has been that in most Member States the national eGovernment strategies thus include Action Plan's political objectives. The mid-term evaluation gives some examples that show how the national eGovernment work has been influenced by the Action Plan. The Expert Group has been a forum for exchange of information and discussion on eGovernment EU-wide.

Finding 10. An EU eGovernment Action Plan (based on the Malmö Declaration) has been a "mobiliser" instrument for national eGovernment policies. This gave further strength and visibility to the objectives of the Action Plan. The Action Plan's governance established an EU-wide forum for exchange of information and discussion on eGovernment. 6.2.Efficiency 6.2.1.Efficiency of the related budget The eGovernment Action Plan had no dedicated budget, therefore efficiency related to budget may only be linked to actions under the Action Plan for which spending has been allocated under an EU funding scheme. For the same reason, no systematic monitoring of resources has been carried out for each of the priorities of the eGovernment Action Plan. Likewise, across the 28 Member States the implementation of the Action Plan has been carried out in a variety of different manners on the national level. eGovernment is supported through a number of funding programmes (see also under 3.2.2.): The past Competitiveness and Innovation Programme ICT Policy Support Programme (CIP ICT-PSP); CEF, Digital Services Infrastrctures (CEF DSI); Horizon 2020's Societal Challenge 6; The past Seventh EU Framework Programme for Research (FP7); The ISA programme; The European Structural and Investment Funds ESIF (ERDF, CF, ESF, YEI and EAFRD); before 2014: Structural Funds. CIP LSP projects The CIP ICT Policy Support Programme funded the following LSPs projects (with the approximate total costs, including the 50 percent EU co-funding): STORK (eIdentification – 26 + 18,6 M EUR), eCodex (e-justice – 24 M EUR), PEPPOL (e-procurement – 30,8 M EUR), SPOCS (Points of Single Contact – 24 M EUR), ePSOS (e-health – 36,5 M EUR) and eSENS (consolidation of results – 27 M EUR). In the analysis below, the eHealth related activities and pilot are excluded as they are covered by a separate eHealth Action Plan. The CIP Implementation reports provide overviews on the LSPs. The total cost of the eGovernment LSPs (excluding ePSOS) is around 150 million EUR. The Commission has co-funded around 50 per cent of this amount the other half was invested by the participating Member States (and countries associated to the CIP programme). The Commission report on "Evaluations of the Competitiveness and Innovation Framework programme" referred to the final evaluation of the CIP programme: "The evaluation underlined the improvements arising from the delegation of the management of substantial parts of CIP to the EACI concerning the efficiency of the programme management, both in relation with costs to the Commission and efficiency of the services provided, as measured in terms of number of contracts signed, the period to contract and payment delays." The results achieved through these pilots in the form of re-usable technical building blocks that are now starting to be deployed in the EU have been described above in 7.1.2. The work contributed towards having key cross-border services online. The participation to these pilots was (and is) voluntary of which reason the costs varied among the Member States. The pilots typically involved between 10 – 20 Member States.

The Action Plan has also led to financing initiatives under the Research and Innovation programmes of the Commission. In particular, research and innovation actions were targeting the understanding the transformation of public administration in engaging more stakeholders in the policy making process, the reduction of administrative burden and the service delivery of public services. Under the FP7 the programme invested over 75M EUR between 2011 and 2013 in co-financed research and innovation actions. Since the launch of the new programme, H2020, over 27M EUR were invested in co-financed research and innovation actions. Finding 11 The Action Plan had no dedicated budget. Despite this, the Action Plan worked by providing a guiding framework for funding. The Action Plan mobilised resources in the Commission and Member States (and countries associated to the CIP programme), that co-funded 50 per cent of the LSPs to develop solutions and common understanding in several cross-border policies. 6.2.2.Costs and benefits to stakeholders The efficiency can also be assessed as the capacity of the Action Plan to mobilise resources (human and financial) versus the potential savings in the intervention domain. Investments in the modernisation of the European public sector have to be seen also in the wider context. In the EU, public expenditure accounts for almost 50% of GDP and the public sector represents about 17% of total employment. Public administrations have a powerful means to pull innovation; in the EU, in 2010, public authorities in the EU spent over € 2 400 billion on supplies, works, and services – amounting to around 19% of EU GDP. The potential of these investments for savings can also be seen in different sectors mentioned in the Action Plan, for example: eIdentification

It has been assessed that replacing the current paper based identification elements used across the private sector could reduce costs by 90% if eID replaced them. In addition, eID is also the most important service to access other digital public services, a key enabler. e-procurement

Contracting authorities and entities that have successfully completed the transition to e-procurement commonly report savings between 5 and 20%; whereby experience also shows that investment costs related to transition to e-procurement can be rapidly recouped. Given the size of the total procurement market in the EU, each 5% saved could return around €100 billion to the public purse. Based on the extrapolation of the national evaluations of benefits, the EC estimates that the adoption of e-invoicing in public procurement across the EU could generate savings of up to €2.3 bn. Switching from paper to fully automated e-invoicing can cut the costs of receiving an invoice from €30-50 to €1. The once-only principle The extension of this principle, in compliance with data protection legislation, would likely generate an annual net saving at the EU level of around EUR 5 billion per year by 2017.

As reported in the mid-term evaluation, for the Member States public administrations, achieving this type of financial gains and improving the efficiency of administrative processes were among the main drivers for eGovernment projects, even more so given the current economic climate and budget pressures. Financial savings are expected to translate into efficiency gains in a variety of ways, including a reduction of administrative burden and savings in opportunity costs. The mid-term evaluation found in the eGovernment case studies that financial impacts are not always clear before implementing a project. In general, the mid-term evaluation found that the decision to implement a project depended in only a few cases on a clear business case. At the same time, the mid-term evaluation found that it was uncommon to have users of the service paying a fee for it. In fact, financial benefits for the public sector are expected to come mostly from savings generated by reducing the production costs for a service and/or by the rationalisation of public expenditure. This meant that for the other stakeholders, citizens and businesses, the Action Plan activities did not incur direct costs and they were able to benefit from, for example, improved cross-border availability of eGovernment services. Finding 12 Different sectors report significant potential savings from the use of eGovernment. Given the important size of public expenditure in these and other sectors, the resources that the Action Plan mobilised towards the modernisation and increasing efficiency of the public administrations seem justified. 6.3.Relevance The public sector represents a significant and unique part of Europe’s economy, has a key responsibility for delivering citizens' quality of life and is also a vital actor in efficiently and effectively delivering policy objectives. Economic and budgetary pressures force governments to be ever more efficient, effective, reduce costs and be more competitive in a multi-polar world . The, contribution of eGovernment to jobs and growth remains a very relevant high-level objective. Repeated analysis during the European Semester has shown that the modernisation of public administrations can contribute to tackling economic challenges. The recent Annual Growth Surveys recognise the importance of a modern and efficient public administration for a business-friendly environment and to ensure fast and high-quality services for citizens. Additionally, in many EU countries the work to implement digital end-to-end services and to achieve cross-border interoperability continues, so further pursuing the eGovernment activities remains relevant. Online public services continue in relevance for reducing business costs and increasing the efficiency and the quality of the services provided to citizens and businesses. According to a recent study, if interactions with public authorities can be made as transparent, as fast and as cost-efficient as in the private sector, then the potential benefits will materialise. 6.3.1.Relevance: User Empowerment As noted earlier, according to the 2015 eGovernment Benchmarking report, user centricity is confirmed as the most advanced indicator. However, many Member States are not focusing enough on the quality of the users' experience. According to the mid-term evaluation (on Action 2), most Member States have introduced personalised services in their One-Stop-Shops for business and for citizens, as well as in the area of health and tax services. The relevance of one-stop-shops continues with new needs for cross-border approaches.. The Action Plan objectives on re-use of public sector information have been found to be relevant. The Action Plan's objectives on user needs, inclusive services, collaborative services, transparency and the involvement of citizens and businesses in policy-making were assessed to be appropriate for user empowerment. This is also due to advancing technology that brings, for example, increased interactions through new tools and social networking. Transparency and participation continue as important principles in the modernisation of public administrations. According to the eGovernment benchmarking study users are provided with information about the duration of the process in only 39% of measured cases. The figure for maximum delivery timelines for government is 46%. At the same time, administrations rarely give an account of their service performance (in 38% of cases). Only 1 in 3 websites inform visitors about their ability to participate in policy-making processes. Users of Member States' justice systems are an example of user empowerment through new technological tools Here, there are currently gaps regarding the deployment of e-justice or more specifically the use of ICT at courts, for example, concerning the possibility to communicate electronically between the courts and the parties as well as the use of ICT in court management system as identified by the EU Justice Scoreboard. The online public consultation also asked how stakeholders should be enabled to contribute to, make proposals on and publicly exchange their views on new initiatives emerging under the eGovernment Action Plan. Most respondents said that using an online social media platform was a good idea. Using a social media platform was viewed as useful to reach out to a large audience, citizens as well as businesses.

Finding 13 The eGovernment Action Plan priorities on user empowerment have been found to be relevant. However the Action Plan could benefit from stronger user empowerment through a mechanism to enable stakeholders to contribute to the initiatives of the Action Plan.

The Action Plan objective on the electronic service to support citizens' initiatives was achieved. However, the general goal of involvement of citizens and the rights of citizens continues to be relevant. According to the Citizenship Report 2013, only one in three citizens (36%) say they are well informed about their EU rights and just under a quarter (24%) feel fairly or very well informed about what they can do when their EU rights are not respected. Building on the achievements of the Action Plan on the sub-priority on the re-use of public sector information further work was relevant and continues to be so. The mid-term evaluation finding of the lack of agreement on the common set of Public Sector Information (PSI) re-use indicators can also feed into future work, including in possible future adaptations of the PSI Directive. The policy evaluations of the PSI Directive and the Infrastructure for Spatial Information in the EU (INSPIRE) Directive demonstrated the need for public authorities to continue improving the sharing and re-use of their data. Despite challenges remaining in implementing the INSPIRE Directive, both the Member States and the consulted stakeholders reported qualitative benefits ranging from more efficient information services to citizens and the improved collaboration between public authorities, as well as building up skills, to cost savings in various areas and improved data quality. Estimates are available for a number of countries , . Other illustrations are more effective environmental risk management , improved cross-border and cross-sector collaboration, spatial planning and lower costs for environmental impact assessments . Moreover, synergies were reported between the EU and national strategies on Open Data and eGovernment where the availability of INSPIRE spatial data and services is used increasingly for a wide range of government-to-government and government-to-public applications . The Commission has put in place an Open Data Portal where re-usable data from the EU institutions is are referenced and has currently more than 8000 datasets available. As a next step to opening up its own data, the Commission is actively promoting the discoverability and re-use of PSI in a cross-border and cross-lingual context, notably by the creation of the European Data Portal, a digital service infrastructure under CEF. The importance of open data and re-use of data continues relevant for contributing to transparency and for economic growth. According to a recent study "Creating Value through Open Data" the market size for Open Data is expected to increase by 36,9 % between 2016 – 2020 to a value of 75,7 bn EUR in 2020. The study also forecasts and a 32 % growth in open data jobs in the 5-year period and important cost savings. The relevance of the Action Plan sub-priority on transparency, including data protection has continued to increase. In addition to the consent-based approaches, users demand to be in control of their data and provide only the amount of data necessary for a certain transaction. The future General Data Protection Regulation on which co-legislators reached a political agreement in December 2015 updates and modernises the Data Protection rules currently contained in the 1995 Data Protection Directive and will have an effect also on the public sector. Finding 14 There are continued needs in the Action Plan priorities on citizens' rights, re-use of public sector information, transparency, participation and data protection. 6.3.2.Relevance: Internal market The Action Plan objectives of seamless services for businesses, personal mobility and EU-wide implementation of cross-border services still correspond to the needs of the single market. The Digital Single Market of over 500 million people cannot be completed without public services. In 2014, there were 17.9 million persons who had been born in a different EU Member State from the one where they were resident. From the perspective of a citizenship, the figure is still high, in 2014 more than 14 million EU citizens lived in another EU Member State than where they had their citizenship. The trends of intra-EU mobility are steadily increasing. According to a 2013 study, over 1,5 million citizens and 314,000 businesses are likely to use cross-border eGovernment services each year in 2020. The same study estimates that the potential volume of the top five public services for businesses is 2,490,000 cross-border users annually. The mid-term evaluation of the Action Plan noted that under half of Member States had rolled out LSP projects, which highlights the need for continued action with the CEF DSIs. There are also increasing needs for cross-border company procedures and the related objectives are still highly relevant, for example in the areas of e-procurement, company law, e-justice, Points of Single Contact and information portals: E-procurement

As outlined above, transition to the e-procurement presents a potential of significant savings enabled by streamlining, simplification and opening up of public procurement procedures. Only adoption of e-invoicing in public procurement across the EU could generate savings of up to €2.3 billion . The work in this area delivered beyond the objectives of the Action Plan. In 2014, the European Commission adopted new public procurement directives, which enter into force in April 2016 and make e-procurement gradually compulsory, with e-Submission being mandatory for all public buyers as of October 2018. The policy aiming at supporting transition to full e-procurement is also building on a 2012 Communication on "A strategy for e-procurement" and a 2013 Communication on "End-to-end procurement to modernise public administration". A wide range of support measures have been put in place to facilitate the transition to e-procurement in all Member States. This will allow the businesses, in particular, SMEs, to benefit as early as possible from new business opportunities opening up at the European Single Market thanks to the electronic public procurement, in particular simplification of the pre-qualification phase. Company law The objective of the 'internal market' priority of the action plan aimed at facilitating the running of companies anywhere in the EU. The Digital Single Market Staff Working Document noted that existing rules under company law do not sufficiently integrate the benefits of digital technologies. Companies are still faced with paper-based formalities. For example, online registration of companies is only possible in 16 Member States. Though a number of measures have been taken at national level, a national approach does not remove the obstacles that companies face if they consider setting up or operating a company across borders. In this context, the Single Market Strategy further mentioned that the Commission will consider further ways of achieving simpler and less burdensome rules for companies — while continuing to act against letterbox companies — including making digital solutions available throughout a company’s lifecycle, in particular in relation to their registration and to the filing of company documents and information. E-justice The action plan foresaw in the 'EU wide implementation of cross border services' activities in the eJustice area as there are continuing needs in cross-border e-justice. The Digital Single Market strategy (Staff Working Document) noted that European e-Justice facilitates access to justice and cross-border judicial proceedings and makes it easier for citizens to find a lawyer/notary in the EU and for businesses to search for insolvent entities through interconnected insolvency registers. However, in a cross-border context issues identified at national level are amplified, as cross-border cooperation in this area entails that the Member States are at equal levels of development. In this context, the work started by the CIP pilot e-CODEX remains highly relevant. Although the pilot project has created a strong backbone for communications between justice administrations, which can be used for many different purposes, extension of this work will be needed in three dimensions. Firstly, the number of Member States which currently participate in this work remains limited, and should be expanded to cover all Member States. Secondly, the e–CODEX backbone provides many opportunities to further support cross-border exchanges, including between judicial professionals such as laywers, notaries and judicial officers. Thirdly, expansion can also take place in terms of the number of judicial cooperation instruments covered. A first pilot project implementing this technology on the European e-Justice Portal will go live in 2016. Further work on this topic could include efforts to realise the cross-border elecronic service of documents. Furthermore, as of 2017, the business registers interconnection system, which will be composed of Member States' business registers, a European central platform, and the European e-Justice portal serving as the European electronic access point will facilitate access to information on companies for citizens and businesses throughout the EU . In addition, the interconnection of national land registers will be developed to go live on the European e-Justice Portal in the course of 2017. Points of Single Contact Extending the scope of the Points of Single Contact as indicated in the Action Plan could further contribute to simplification, savings for public administration and more coherent approach in providing information and e-services to businesses. Information gaps increase costs for business, in particular for SMEs. Procedural streamlining via Points of Single Contact (PSC) could generate up to 0.15% of GDP in the medium run (5-year horizon) and up to 0.21% of GDP in the long run . In line with Action Plan action 18 to transit into "Second Generation" Points of Single Contact (PSC), that would function as fully-fledged eGovernment centres, a first assessment of the PSCs was carried out in 2011-2012. The results revealed a hybrid landscape, with some PSCs more advanced than others in providing the information and services required. This first assessment resulted in the Communication on the implementation of the Services Directive "A partnership for new growth in services 2012-2015”. This Communication encouraged Member States to develop by the end of 2014 Second Generation Points of Single Contacts, which should “1) cover all procedures during the business life cycle, 2) be multilingual, and 3) be more user-friendly”. These recommendations were subsequently included into the PSC Charter, endorsed by the Council in 2013. The Charter encourages the Member States to bring additional features into their PSCs so that the latter respond more effective to the business life cycle of a company in the services sector. It also includes the benchmarks facilitating the assessment of the PSCs with a view to ensure their continuous improvement and ultimately, to provide better public online services to business. The assessment of the PSCs performance against of the criteria set out in the Charter was carried out in 2014 and published in 2015 in a study. The results of the study indicate that PSCs are still far