Table of contents 1 Introduction 1.1 Context of the Web Accessibility Directive 1.2 Purpose and scope of the evaluation 2 What was the expected outcome of the intervention? 2.1 Description of the intervention 2.2 Point(s) of comparison 3 How has the situation evolved over the evaluation period? 3.1 Implementation of the Directive 3.2 National monitoring, reporting and enforcement 3.3 Current state of play and context 4 Evaluation Findings (analytical part) 4.1 To what extent was the intervention successful and why? 4.2 How did the EU intervention make a difference and to whom? 4.3 Is the intervention still relevant? 5 What are the conclusions and lessons learned? 5.1 Conclusions 5.2 Lessons learned Annex I. Procedural Information

Annex II. Methodology and analytical models used Annex III. Evaluation matrix Annex IV. Overview of benefits and costs Annex V. Stakeholder consultation – Synopsis report Annex VI. Additional considerations from national monitoring reports Annex VII. Mapping of UN Sustainable Development Goals (SDGs) Annex VIII. Bibliography Glossary Term or acronym Meaning or definition AS Accessibility Statement DPO Organisation representing people with disabilities EAA European Accessibility Act EEA European Economic Area EPSR European Pillar of Social Rights HEN Harmonised European Standard (specifically EN 301 549) MS Member State(s) PSB Public sector body UNCRPD United Nations Convention on the Rights of Persons with Disabilities WAD Web Accessibility Directive WCAG Web Content Accessibility Guidelines WADEX Web Accessibility Directive Expert Group Introduction This staff working document represents the evaluation of Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the accessibility of the websites and mobile applications of public sector bodies, also referred to as the Web Accessibility Directive (WAD) and its implementing acts. Web accessibility means that ‘all people, particularly people with disabilities and older people, can use websites in a range of contexts of use, including mainstream and assistive technologies; to achieve this, websites need to be designed and developed to support usability across these contexts’ . In the context of the WAD, accessibility should be understood as principles and techniques to be observed when designing, constructing, maintaining, and updating websites and mobile applications so that persons with disabilities can perceive, understand, navigate, and interact with them. In this document, “accessibility” specifically refers to the above definition only. 1.1Context of the Web Accessibility Directive Accessibility to information and communication technologies is an enabler of rights and a prerequisite for the full participation of persons with disabilities on an equal basis with others. The main purpose of web accessibility is to remove the challenges and barriers which emerge for people with disabilities when using digital information and services on the web or in mobile applications. Accessibility – essential for some, and useful for all – is therefore a key driver for the development of a knowledge-based society and makes it possible to pursue economic growth while ensuring social inclusion. Since the internet has become a preferred way for governments to provide information and public services to citizens, people with disabilities across the EU must be supported to access such information and services online. Web accessibility should be considered not only as a set of tools needed to guarantee the participation and inclusion of people with disabilities in society. It should also be considered as a fundamental means of upholding everyone’s human rights in a digitalised society, in line with the provisions of the United Nations Convention on the Rights of Persons with Disabilities (UNCRPD) . This is possible only if digital interfaces are designed, developed and maintained to ensure that they are accessible for a wide range of needs. However, as assessed in the Impact assessment accompanying the document ‘Proposal for a Directive of the European Parliament and of the Council on the accessibility of public sector bodies’ websites’ (“the Impact Assessment”), before the adoption of the WAD, the level of accessibility of websites and mobiles applications in general, including those of public sector bodies, was low. In addition, the market for web accessibility-related products was fragmented . In 2016, 68.1 million persons were estimated as having a moderate disability and 30.5 million as having a severe disability. Taken together, these 98.6 million people accounted for about 24% of people aged 16 and above and living in the EU in private households . In 2019, about a quarter (24%) of the EU population experienced long-standing activity limitations due to health problems . These numbers clearly demonstrate the relevance of the issues at stake at the time of adoption. This is the context in which the WAD entered into force on 22 December 2016, requiring the websites and mobile applications of public sector bodies (PSBs) to meet accessibility requirements. It set the first accessibility requirements in Europe for the online space. The WAD was conceived as a minimum harmonisation directive, in recognition of the fact that some EU countries had already set higher standards or might do so in the future. 1.2Purpose and scope of the evaluation

This staff working document is an evaluation of the WAD and its implementing acts (the intervention), covering the first three years of its application (from the transposition deadline of September 2018 to December 2021). In line with Article 13 of the WAD, the review must take into account: (i) the Member States’ reports on the outcome of the monitoring (Article 8 refers); (ii) the use of the enforcement procedures (Article 9 refers); (iii) an analysis of the technological advancements that could make accessibility easier for excluded types of content. This evaluation is conducted following the guidance provided by the Better Regulation Guidelines and Better Regulation Toolbox. As for all evaluations, this report will assess: ·to what extent was the intervention successful and why (effectiveness, efficiency, coherence); ·how did the EU intervention make a difference and to whom (EU added value); ·is the intervention still relevant (relevance); and will identify ·conclusions and lessons learned, to be considered in future policy developments in the area of web accessibility. As the United Kingdom (U.K.) is no longer a Member State, it is not considered in scope of this evaluation. This document draws on and is accompanied by an external supporting study. The study analysed the evidence available from primary and secondary data sources , including open and targeted consultations, in-depth interviews, a literature review and the assessment of the official national monitoring reports sent by the Member States (MS) to the Commission. Both the study and the evaluation adopted an inclusive approach to consultation activities, involving all relevant stakeholder groups affected by the WAD, with a specific focus on the participation of people with disabilities. In particular, for the first time, an easy-to-read and more accessible questionnaire offered people with disabilities the chance to take part, including those with cognitive impairment . What was the expected outcome of the intervention? The evaluation starts with an explanation of the theoretical framework developed to assess the performance of the WAD. This framework starts from the description of (i) how the WAD was expected to work (section 2.1) and (ii) the points of comparison on which the evaluation is based on (section 2.2). 1.3Description of the intervention The intervention logic of the WAD is presented in Figure 1 below. Problems and needs. The adoption of the WAD aimed to address four major problems in web accessibility: (i) the need to improve the functioning of the internal market providing web accessibility services, given varying legislation in MS; (ii) the very poor level of accessibility of the websites and mobile applications providing public services and information to the public; (iii) the lack of common technical standards, which created a barrier for market players and therefore for end-users; and (iv) digital and social exclusion of people with disabilities caused cumulatively by the above mentioned problems. Regulatory fragmentation across the EU had hindered the potential growth of the market for web accessibility products and services. A wide variation in technical provisions in calls for tenders was a serious issue for web developers who therefore faced barriers in operating across borders. Such variation was the result of different regulations at national – or even regional – level concerning the web accessibility of PSB websites and mobile applications. Such fragmentation was particularly problematic for SMEs, which could not bear the costs of operating in different markets requiring different standards. In addition, providers of tools for web accessibility (such as authoring tools or automatic testing tools) faced the additional barriers of having to implement all these different national specifications in their tools. This in turn caused an interoperability problem for assistive technologies, as these could not be deployed in different MS with different requirements in place. Moreover, the lack of web accessibility also required public administrations to maintain costly alternative channels, such as call centres, to ensure the provision of services to all. In 2007, the study ‘Measuring progress of e-Accessibility in Europe’ (MeAC) identified the level of web accessibility of PSB websites and mobile applications detected as considerably low. This was confirmed by the subsequent studies MeAC 2 (2010) and MeAC 3 (2012) . People with disabilities and older people had limited or no access to websites, and were obliged to seek potentially more costly alternatives to access information and services. The WAD therefore sought to lower these barriers. The lack of common technical standards was one factor behind the market fragmentation described under problem one. Together with the different legal frameworks in the MS, market players faced barriers, as they could not be certain that tools acceptable in one MS would match requirements in another MS. End-users paid the price for the lack of a functioning single market, the price being lower accessibility. Finally, digital and social exclusion, is caused by a cumulative effect of the above-mentioned three problems. Objectives. Given the above context, the WAD has two key objectives:

1)to ensure that the websites and mobile applications of PSBs are made more accessible on the basis of common accessibility requirements ; 2)to harmonise the internal market for web accessibility products and services, overcoming regulatory fragmentation . Inputs. In order to achieve these two objectives, in 2016 the EU adopted the WAD, which requires the MS to ensure that the websites and mobile applications of PSBs meet common accessibility requirements introduced by the Harmonised European Standard, linked to the WAD with the Implementing Decision 2018/2048 Furthermore, to ensure that provisions were implemented in a consistent way across the EU, and that MS regularly monitored the accessibility of public sector websites and mobile applications, the Commission adopted the Implementing Decision 2018/1523 and Implementing Decision 2018/1524. These implementing decisions respectively establish: (i) a model accessibility statement (AS); and (ii) a common methodology for monitoring the conformity of PSBs’ websites and mobile applications with the accessibility requirements of the WAD. Activities. The WAD requires the European Commission to help the MS implement the WAD. To this end, in addition to adopting the Implementing Decisions, the Commission set up the Web Accessibility Directive Expert Group (WADEX) which supports cooperation between MS and with relevant stakeholders. MS were required to assign to specific bodies the responsibilities for: (i) monitoring and reporting as required in Article 8 of the WAD; and (ii) enforcement at the national level, as required in Article 9 of the WAD. In the course of setting up national monitoring processes, MS had to involve and consult end-users with disabilities. The WAD also obliges MS to take the necessary measures to raise awareness of accessibility requirements and to promote and facilitate training programmes on accessibility. Finally, PSBs were required by the WAD to provide and regularly update a detailed, comprehensive and clear accessibility statement (AS) on the compliance of their websites and mobile applications. Expected outputs. Four main outputs were expected: (i) a more coherent and consistent legislative framework on web accessibility across the EU; (ii) the appointment of monitoring, reporting and enforcement bodies at national level; (iii) the publication of an AS on the websites and mobile applications of each public sector body; (iv) feedback from end users to PSBs on the accessibility of their websites or mobile applications, leading to virtuous improvement cycles. Expected results. In line with the two key objectives mentioned above, the implementation of the WAD was expected to lead to two types of results. First, a harmonisation of the internal market for web accessibility products and services, overcoming the regulatory fragmentation existing prior to 2016. In the identified intervention logic, the overall results were divided into three measurable outcomes: (i) increased cross-border sales of accessible information communication and technology (ICT) products and services; (ii) more companies offering accessibility expertise; and (iii) a reduction in prices for web accessibility products and services due to increased competition in the market. Second, an increase in EU Member States and EEA countries of: (i) the number of accessible PSB websites and mobile applications (according to the WAD’s requirements) and (ii) the use of PSB websites and mobile applications by people with disabilities. Expected impacts. The WAD aimed at increasing digital and social inclusion by improving access for people with disabilities to public sector websites and mobile applications. Greater public participation was envisaged, measured through the use of online public services and the greater ‘co-design’ of public policies (i.e. where policymakers and the public work together to design public policies). Wider impacts were also expected in the market due to legislative harmonisation and the reduction of regulatory barriers across the EU. The two most significant market results expected were: (i) a more efficient digital internal market; and (ii) web accessibility market growth, contributing to economic growth and job creation. The final expected impact of the WAD was greater digitalisation-led efficiency in the delivery of public sector services. 1.4Point(s) of comparison The Impact Assessment estimated that the EU policies and actions available before the introduction of the WAD would not have been sufficient to solve the problems and needs explained in section 2.1. The market would have remained fragmented, and the low level of web accessibility in the public sector would have persisted. Members of the public unable to interact with online public services would have continued to experience ‘digital exclusion’ and limitations on their social, economic, and civil participation. Given that the Impact Assessment was drafted in 2012, the data gathered at the time of the initial proposal has limited relevance today, due to the rapid increase in digitalisation since then. Consequently, the evaluation re-conducted the analysis to the five main evaluation criteria suggested by the Better Regulation Guidelines: effectiveness, efficiency, coherence, relevance and EU added value. In light of these considerations, the intervention logic described in section 2.1 explains how the inputs and activities required to meet the objectives of the WAD are expected to generate outcomes, namely short-term outputs, medium-term results and long-term impacts. These elements will be used as the main points of comparison to assess the performance of the WAD. Such points are built on the evaluation framework provided in the Impact Assessment and are directly linked to the evaluation matrix presented in Annex III: ·Point 1: relation between the observed results (medium-term) and impacts (long-term) and the objectives of the WAD. To what extent have the two main objectives of the WAD been achieved – measured through the effectiveness criterion (see sections 4.1 and 5.1.1 for the results of the evaluation). The analysis considered (i) the most relevant success factors for the application of the WAD and (ii) the gaps and challenges hindering the achievement of the objectives. ·Point 2: relation between the inputs and activities put in place for the implementation of the WAD and the observed outcomes. The costs borne to achieve the outcomes of the WAD – measured through the efficiency criterion, i.e. to what extent did the provisions of the WAD produce outputs at reasonable costs (see sections 4.1.10 and 5.1.2). ·Point 3: relation between the inputs and activities conducted to implement the WAD and the problems to be addressed. To what extent is the WAD complementary with other legislative initiatives at the European and national level on web accessibility of public services – measured through the coherence criterion, i.e., the coherence of the WAD with other legislative interventions in the area of web accessibility and inclusion, at national, international and EU level, identifying any complementarities or inconsistencies (see sections 4.1.11 and 5.1.3). ·Point 4: the difference made by the EU in achieving the outcomes of the WAD. To what extent has an EU-level intervention reached outcomes which could not have been achieved by MS on their own – measured through the EU added value criterion (see sections 4.2 and 5.14). ·Point 5: relation between the inputs and activities conducted to implement the WAD and the objectives. To what extent has the WAD been relevant to achieve its initial objectives – measured through the relevance criterion (see sections 4.3 and 5.1.5). For each of these five points, the evaluation matrix presented in Annex III both explains the key questions set to perform an evidence-based assessment, and describes the related indicators and data sources used to judge the performance of the WAD.

How has the situation evolved over the evaluation period? 1.5Implementation of the Directive The WAD was implemented through the following milestones: on 22 December 2016, the WAD entered into force. By 23 September 2018 (transposition deadline), MS were required to transpose the WAD into their national legislation and designate bodies responsible for monitoring and reporting functions, as well as for enforcement of the WAD. MS were to apply national legislation transposing the WAD in practice according to the following timeframe: ·from 23 September 2019 for websites published after 22 September 2018, i.e. for ‘new websites’; ·from 23 September 2020 for all other websites of PSBs, i.e. for all websites, old and new; ·from 23 June 2021 for mobile applications of PSBs. In line with Article 8 of the WAD, all MS are required to submit every 3 years a report (the ‘monitoring report’) of the results of monitoring and enforcement procedures. The first submission was due by 23 December 2021. At European level, in 2016 the European Commission set up the WADEX group to support uniform application of the WAD at national level. WADEX is expected to achieve this goal by fostering cooperation and the exchange of best practices on web accessibility between MS and relevant stakeholders. Three implementing decisions were adopted in 2018, to support a consistent implementation of the WAD across the EU. These implementing acts are directly applicable and need not be transposed into national law. ·Implementing Decision (EU) 2018/1523 laid down a model accessibility statement which PSBs must follow to ensure that their websites and mobile applications comply with the WAD (WAD Article 7 refers). ·Implementing Decision (EU) 2018/1524 set up a monitoring methodology and arrangements for the reporting every 3 years required from MS (WAD Article 8 refers). ·Implementing Decision on the harmonised standard (2018/2048) for websites and mobile applications. This implementing decision provided for the presumption of conformity with the accessibility requirements recognised by the WAD (WAD Article 6 refers). Subsequently amended by Commission Implementing Decision 2021/1339, which refers to the latest version of the Harmonised European Standard EN 301 549 V3.2.1 (2021-03). All MS have officially transposed the WAD into national legislation. Six MS completed the transposition within the requested timeline: Slovenia, the Netherlands, Denmark, France, Italy and Spain. The U.K., still a Member State at the time, also transposed the WAD on time. The MS that did not transpose fully by the deadline can be divided into three groups: ·12 MS notified the transposition measures after the deadline: Ireland, Greece, Croatia, Cyprus, Latvia, Luxembourg, Malta, Poland, Portugal, Romania, Finland, Sweden; ·6 MS notified pre-existing measures within the deadline, without providing evidence of full compliance with the WAD: Bulgaria, Czechia, Estonia, Hungary, Lithuania, Slovakia; ·3 MS with federal systems which did not adopt measures in all regions within the deadline: Austria, Belgium, Germany. 1.6National monitoring, reporting and enforcement MS were required to designate monitoring, reporting and enforcement bodies by 23 September 2018. The three bodies must perform the functions set out below. Monitoring bodies must annually monitor the accessibility of public sector websites and mobile applications and provide information to PSBs on how those websites and mobile applications are complying with the accessibility requirements. They may also check for the compliance with the WAD of accessibility statements published on the websites and mobile applications of PSBs.

Enforcement bodies must: (i) ensure the availability of an adequate and effective enforcement procedure to guarantee the compliance of websites and mobile applications with the accessibility requirements; (ii) investigate when a PSB affirms that making content accessible would be an excessive burden for them (the so-called “disproportionate burden”) ; and (iii) ensure that feedback given by users is effectively handled (both notifications about content that is not accessible and requests for content excluded from the accessibility obligations). Reporting bodies are required to publish in an accessible format and to submit reports on the results of monitoring and enforcement activities to the Commission every 3 years. All Member States (MS) met the obligation to appoint these bodies. In 12 MS monitoring, reporting and enforcement are performed by the same authority, while 15 MS entrusted these functions to separate bodies. Austria, Belgium, Germany, Spain and the Netherlands assigned responsibility for enforcement to the relevant regional or local level of public administration bodies. The full list of bodies appointed by MS and notified to the European Commission is published on a dedicated web page. The assignment of multiple or single function to the bodies concerned does not appear to have any impact on their efficiency and is in line with the WAD requirements. Currently there are 25 national reports available, covering all EU Member States except France and Cyprus. All the monitoring reports are published on a dedicated Commission website, as well as on national websites. According to these reports, total of 10 412 websites and 298 mobile applications have been tested for accessibility by MS during the first monitoring period, in what has been described as the ‘world’s largest accessibility test’. 1.7Current state of play and context We cannot consider the period under evaluation (2018-2021), without mentioning COVID, and how it accelerated the digital transformation. The pandemic made PSB crisis information essential, and obliged use of online PSB services, to an unprecedented extent. Part of the Commission’s response has been the Digital Decade policy , which sets ambitious targets for further digitalisation of the public sector. This context helps explain some of the apparent contradictions in the public feedback, which identified progress but also the need for further improvements. To the challenge facing MS at the start, of identifying and improving accessibility of all existing PSB websites and mobile applications, were added the sudden need to digitise more, to limit, organise or replace face to face transactions, and the greater use of online PSB information and services across all government departments and levels, due to the pandemic. The 2022 eGovernment Benchmark 2022 report found that only 16% of evaluated eGovernment websites passed all of the selected 8 accessibility criteria, and expressed a need for more accessible public sector websites . However, looking closer at the source data and test results per criteria, 28% of websites did not fail any criterion (e.g. test passed or inapplicable). Although this pilot indicator , measured with automated tools on selected criteria, does not determine compliance with the WAD, it helps determine if web content meets accessibility guidelines. This pilot gives an insight into the state of accessibility, but the results cannot be compared directly to earlier studies (e.g. MeAC ) or even national monitoring reports, due to different testing methodologies and the fact that for a complete evaluation of website’s accessibility, additional manual evaluations are needed to verify full compliance. While administrations across Europe largely rose to the challenge and accelerated digital service transformation in the face of the coronavirus pandemic, the eGovernment Benchmark 2022 report suggested to prioritise user-centric design to ensure that eGovernment services are inclusive to users in all their diversity, including users with poor digital skills or those living with disabilities. The 2020 Berlin Declaration on digital society and value-based digital government acknowledges the public sector as an essential element for the European Single Market and emphasises the importance of digital public services in our everyday lives. According to the first progress report on the monitoring of the Berlin Declaration , in 2022 the EU average score was 60% in policy area 2 on ‘social participation and digital inclusion to shape the digital world’ . The revised European Interoperability Framework looks at principle of inclusion and accessibility, and observed an improvement of EU average scores since 2019 . By 2021, the share of individuals using internet in the EU had risen to 90%, some 20 percentage points higher than in 2011 . In 2022, the share of individuals interacting online with public authorities in the EU had risen to 63% (for 25 to 64 years old) and 38% (for 65 to 74 years old) from 48% and 17% respectively in 2012. This trend is likely to continue, as explained further in sections 4.3 and 5.1.5 on relevance. Evaluation Findings (analytical part) This section discusses the analytical findings of the evaluation, encompassing all five evaluation criteria (effectiveness, efficiency, coherence, relevance and EU added value). As described in section 1.2, the analysis draws from a supporting study and extensive consultation activities, including: (i) a public consultation using both a standard and an easy-to-read questionnaire; (ii) targeted surveys for different stakeholder groups; (iii) in-depth interviews conducted with relevant stakeholders; and (iv) analysis of Member States’ reports on the outcome of the monitoring and the use of the enforcement procedures (see Annexes II, V, VI). 1.8To what extent was the intervention successful and why? The assessment of the intervention’s success level is based on the first three points of comparison described in section 2.2. 1.8.1Compliance with the Directive and conformity with accessibility requirements are not identical

At the outset, it should be mentioned that conformance to the accessibility requirements does not equate to compliance with the WAD. This distinction was sometimes overlooked, as observed throughout the evaluation, in national monitoring activities and reports, and in the wording used by monitoring tools. For a website or mobile application to be compliant with the WAD, it must fulfil two criteria. First, it must conform to the accessibility requirements (perceivable, operable, understandable and robust), for which the Harmonised European Standard EN 301 549 gives a presumption of conformity . Secondly, a website or mobile application must have a detailed, comprehensive and clear accessibility statement (AS). The AS, based on the model provided by the Commission implementing decision, must include information on (i) compliance status, (ii) any inaccessible elements and (iii) accessible alternatives, (iv) how to report or request any inaccessible content, and (v) a link to the complaint mechanism if the response is inadequate. Thus, a website or mobile application without an AS is not compliant with the WAD, even if it is technically fully accessible. Conversely, a website or mobile application that, for example, cannot be navigated with a screen reader, does not conform to accessibility requirements, as defined in the standard. However, it may still be compliant with the WAD if the inaccessible element is exempt, for example due to legitimate disproportionate burden, and this is clearly explained in the AS, with information on accessible alternatives . The box below presents the findings from the monitoring reports related to the accessibility statements. Most of the Member States (20 out of 25 that submitted reports) checked the availability of accessibility statements, but five MS did not. The evidence submitted shows that accessibility statements (AS) are either missing or were not checked on websites or mobile applications at different rates across the MS. It is not an obligation to monitor AS availability per se, but the web page with the accessibility statement should be part of the in-depth monitoring according to the Implementing Decision (EU) 2018/1524 . As became apparent during the interviews, a lack of AS may be correlated with lower level of feedback from end-users to PSBs. These statements offer information on how to report accessibility issues, and to pursue accessibility requests, ensuring accessibility statement as standard – as required under the WAD – is essential. For the next monitoring period, this issue of encouraging feedback by virtue of AS should become a focus for the monitoring agencies, as publishing an AS is an important obligation of the WAD. Only five MS compared the content of the AS with monitoring results to validate the AS claims. Although this comparison exercise is not mandatory, it would not require much extra effort, and would show that AS matter. Such a comparison would assess the quality of the AS, and ensure that the PSBs (or their suppliers) are aware of any mismatch between the AS, and what it presents as accessible and inaccessible content and alternatives on the website or mobile application. If the monitoring agencies find recurring differences between the AS and the monitoring results, this may be a reason to focus monitoring and/or training activities on specific functions or areas. From the end-user’s perspective, it is, of course, important that the AS is complete, correct and reliable. Inaccessible content flagged by monitoring tools may be legitimately exempt (e.g. disproportionate burden) or not in scope of the WAD . The monitoring tools, especially automated ones, mostly assess the accessibility requirements set out in the WCAG (sometimes also parts of HEN), but do not consider the possible legitimate exceptions and alternatives to inaccessible content. In this context, the lack of an explicit obligation on the MS to consider the accessibility statement when monitoring a website or mobile application is not ideal. The research literature shows that although there was already a slight improvement in the accessibility of public sector bodies’ websites and mobile applications (objective 1) in the early years following the transposition of the WAD, there is still a notable lack of accessible websites across the EU in both public and private sector. However, the research in the existing literature is of limited relevance, as it is mostly based on automated accessibility checking, so excluding accessibility statements, and covers typically national level PSBs. As noted earlier, the 2007 benchmarking study Measuring e-Accessibility (MeAC) found only 5% of government websites passed automatic and manual accessibility checks. In 2011, the follow-up study, using a different method, found that 34% of the EU ‘web content’ (public and private) was accessible. However, given the rapid digitalisation of the public sector since then and evolution of automatic testing , these figures also are no longer particularly relevant. Although the 2022 eGovernment Benchmark report found that only 16% of the eGovernment websites passed all of the selected web accessibility criteria, the conformance rate per criteria is more encouraging: out of all checks 76% passed, 17% failed, 7% were inapplicable . Rather than giving an accessibility score per website, the modern automatic accessibility testing tools count the number of criteria that pass, fail or are inapplicable, and thus allow testers to identify conformity per criteria. For example, the eGovernment Benchmark report suggests that some accessibility criteria are better or more easily implemented than others . This is in line with the findings of the 2022 WebAIM Million report. After examining the accessibility of the top 1,000,000 home pages worldwide, the report noted that while accessibility errors have decreased, no website is completely error-free. 1.8.2Progress and compliance – surprisingly hard to demonstrate More than half of respondents (57%) to the easy to read survey found government websites easier to use in the last three years, while in the standard survey 41% reported experiencing accessibility issues never or less often than three years ago. About a third of respondents in both surveys reported no changes in their experiences. Looking closer at content types or services offered by public sector bodies, 72% of respondents to the standard consultation observed slight or significant improvements when accessing online information (respectively 115 and 27 out of 198) over the last three years, as shown in the chart below. This opinion was shared by 52% of people with disabilities, older people, or people taking care of people with disabilities in targeted surveys. As to the role of WAD, 50% organisations representing people with disabilities, older people or consumers agreed that the WAD had contributed to greater accessibility of online public services. Yet about the same number of respondents to the public consultation (57%) do not think that ‘people with disabilities can use online information just as well as others can’. This apparent contradiction can be explained – PSB web accessibility has increased from a low starting point, but persons with disabilities cannot yet access all online public services on equal terms with everyone else at a time when public services are progressively more online. The majority of respondents (79%) affirmed that their use of online public services has increased in the last three years, and 65% expected more improvements in the future. The main accessibility issues identified when using public services online were navigation and accessing forms, followed by video content (though live time-based media is not in scope of the WAD) . From a geographical perspective, there is slight difference in experience, as measured by the surveys. More respondents in northern and southern European countries reported experiencing issues less often than before or never (50% in North and 54% in South) compared to central and eastern European countries (33% in Central and 37% in East). One difficulty for MS in demonstrating progress in making websites and mobile applications “more accessible”, is the compliance status as defined in the Implementing Decision (EU) 2018/1523: fully, partially or non-compliant. The Harmonised European Standard EN 301 549 (HEN) sets out the means to determine a presumption of conformity with the accessibility requirements of the WAD: full compliance is achieved by meeting all the applicable requirements. Testing a website or mobile application item can thus result in: pass, fail, or not applicable. In practice this means that if any one item on a relevant page or screen fails the test, the website or mobile application can only be considered partially compliant.

Inevitably, given the wide and varied nature of these requirements, and the many different actors responsible in each MS, most of the websites and mobile applications sampled for monitoring fall in the category of partially compliant. However, partial compliance covers a wide range of situations, from nearly perfect accessibility to only just non-accessible, with no consideration of the impact of errors on the end-user. Failing some criteria (for example, no alternative text for images) would render a page technically non-compliant but usable, whereas failing other criteria (for example, a keyboard trap that prevents a user with visual impairment or limited mobility from completing a task online) would create a serious usability issue, in effect blocking access Unfortunately, there is no agreed methodology amongst accessibility experts at present, neither at EU level nor internationally, on how to quantify partial accessibility as a percentage , or to weight the accessibility requirements by impact. It should also be noted that the HEN has been updated once since 2020, and will continue to evolve. At present the HEN itself acknowledges areas for further work: its Annex D states ‘It is evident that people with limited cognitive, language and learning abilities have diverse accessibility needs and preferences and that there is a need for further guidelines and standards. Research in this area is ongoing’ . Results from the national monitoring reports, however, show there is ample room for improvement in compliance as most of the websites and mobile applications monitored did not respect all of the requirements. Nonetheless, purely binary outcomes of monitoring of websites and mobiles applications (pass/fail of conformity requirements) should be taken with caution, as they do not reflect the full measure of compliance with the WAD. According to the national monitoring reports, there were only a handful of fully compliant websites or mobile applications, but no completely non-compliant website or mobile application. This does not contradict the findings of the public consultation, as there is ample space for relevant improvements within the “partially compliant” category, without reaching full compliance. The interpretation of such results should consider some factors. On the one hand, (i) some MS did not always follow the Implementing Decision (EU) 2018/1524 and defined compliance according to their own methodology, which sometimes was stricter for non-compliant websites, but also sometimes more lenient regarding full compliance. On the other, (ii) when the compliance assessment followed the Implementing Decision, the reporting of “pass” or “fail” results was inconsistent across MS, either because not applicable requirements were handled differently or because the MS also considered (and weighted) the level of breach in different ways. Therefore, it is difficult to draw a meaningful comparison between the MS from the evidence provided in the monitoring. In the second monitoring period, however, this will be addressed by requiring a recurring sample of at least 10 % of websites and mobile applications monitored in the previous monitoring period and at least 50 % that were not monitored in the previous one . 1.8.3Cross-border monitoring tools are in use, although commercial tools are not perfect As objective 2 of the WAD was to harmonise the single market, the fact that some European tools used in the monitoring exercise are being used cross-border is very encouraging. The industry players consulted identified the most relevant drivers of success in the application of the WAD at market level as: (i) the spread of knowledge about accessibility solutions across businesses; and (ii) the increase in demand for accessibility tools. Common technical requirements, such as the Harmonised European Standard EN 301 549, were a first essential step in removing barriers in the internal market and have been useful for making public sector websites accessible. According to industry players, common requirements produced a positive impact on their business, driving demand of products and services in the national markets and an increase in opportunities to penetrate foreign markets . Several MS developed tools themselves, some of which have been also used in other countries: Belgium (Scan.accessibility.belgium.be), Finland (Salvia, using QualWeb), Luxembourg (simplA11yMonit, simplA11yGenReport, simplA11yPDFCrawler), Portugal (AccessMonitor, also used in Slovakia), and Spain (OAW Tracker, also used in Sweden). Coupled with this, many MS used commercial tools, with at least 4 such tools used by at least 3 MS, and 2 by at least 6. More than 10 tools that help manual testing were used. For mobile applications, MS altogether used at least 8 automated tools, with a further 8 tools helping manual inspection. All MS used (to a greater or lesser extent) automated tools in the simplified monitoring, with the exception of Croatia. However, looking at the least tested criteria, the list clearly shows that the vast majority of the automated testing tools used have been created to fit the US market, where WCAG 2.0 (published in 2008) is still relevant. Conversely, the EU legislation uses WCAG 2.1 (published in 2018) as the reference for presumed conformity in Harmonised European Standard EN 301 549. These automated testing tools can only check for 20-30% of success criteria , while some leading accessibility software tools claim 57% of digital accessibility issues . For the in-depth monitoring, the list of least tested success criteria may indicate that multimedia content is either omitted from the page samples or does not feature in the websites and applications selected for monitoring . This would require further investigation, as multimedia content is very important for large user groups. This issue may serve as a possible direction for further investments in the market and/ or research funding, given the expansion of accessibility requirements that will come into effect in 2025 under the European Accessibility Act (EAA) . Regarding harmonisation of the internal market for web accessibility products and services, overcoming regulatory fragmentation (second WAD objective), the evidence collected indicates that the WAD helped to improve the internal market for web accessibility services . Interviews with stakeholders from the private sector indicated that the WAD had lowered entry barriers to other European markets for software providers, facilitating cross-border sales. The industry players consulted agreed that both the demand and supply of web accessibility products and services had increased after the introduction of the WAD. The WAD facilitated an increase in the number of industry actors specialising in accessibility tools in the EU market . This growth of the sector is likely to further improve the internal market in the coming years. The ongoing implementation of the WAD and the upcoming EAA requirements will continue to increase demand, while the in-depth interviews with the national monitoring bodies indicated that the supply of accessibility tools is still insufficient, both in quantity, and quality (not fully covering the requirements set in the HEN). 1.8.4Member States reports are difficult to compare The first monitoring reports provide insights on the different use and interpretation of monitoring methodologies by MS. The WAD and Implementing Decisions deliberately do not impose a particular monitoring methodology, to allow innovation and appropriate choices at national level. MS in practice both monitored and reported on different timeframes, to different versions of the technical standard, and some did not monitor the minimum sample size required. While reserving the right to pursue actions of non-compliance, it should be noted that this was the first such monitoring exercise, and came relatively soon after the practical deadlines for implementation (15 months for websites, and six months for mobile applications). In this context a range of interpretation could be expected. The most significant of these differences concerns the standard used as the baseline for presumption of conformity, with the complexity of monitoring exacerbated by the difficulties encountered in interpreting the guidelines of the WAD. Work will continue to ensure this is fully understood, especially as this standard will be further revised to reflect the requirements of the European Accessibility Act. Although some MS monitored a smaller sample size , all MS who reported carried out both simplified (all but one using automated tools) and in-depth monitoring. Only 2 MS have not reported to the Commission . Some MS committed to test more than is required, according to analysis of findings from the Member States going beyond the scope of the WAD in the monitoring exercise . For example, Germany also tested three requirements, on sign language, easy-to-read language, and the ‘UE’ standard for PDF, not covered by the HEN. Both Portugal and Finland checked WCAG 2.1 AAA criteria in their monitoring exercise, although not an obligation under the WAD. The results were not shared in the monitoring report. On the procedural side, in Greece it is obligatory to publish self-assessment reports.

From analysis of the reports, the following points emerge: ·It also emerged in the interviews that many MS are unaware of the accessibility criteria set out in the Harmonised European Standard EN 301 549 (HEN), and so they only focused on WCAG 2.1 criteria for monitoring. 16 MS monitored only WCAG success criteria, omitting the clauses 5, 6, 7 and 12 of the EN 301 549 (HEN). Only four MS monitored all the relevant requirements of the HEN. This is a significant problem, as while Annex A of the HEN sets the criteria for the presumption of compliance, some MS may use different baselines to judge conformity. Several MS mentioned that the requirements of WCAG 2.1 and HEN for mobile applications were hard to interpret, making it difficult to assess compliance. ·MS conducted the monitoring in different periods within the set timeframe. Implementing Decision (EU) 2018/1524 asked for the first monitoring report to perform the monitoring ‘within’ the period between 1 January 2020 and 22 December 2021 for websites, and 23 June 2021 and 22 December 2021 for mobile applications. In five countries , the monitoring exercise was conducted only in 2021. While the legal text is open to different interpretations, it should be noted that all MS conducted monitoring “within” the period defined in the WAD. ·The scope of the monitoring exercise varied. Under the WAD and Implementing Decision (EU) 2018/1524, MS should select a sample of websites and mobile applications that (i) is as representative as possible (considering the different administrative levels, the type of services provided by the public sector, and the geographical distribution); (ii) is proportionate to the population of the country, and (iii) takes into account the views of stakeholders, in particular organisations representing persons with disabilities and older people . ·The differing interpretations of standards, accessibility conformance rules and testing methods can lead to inconsistent or even conflicting results. This makes it difficult or even impossible to reliably compare monitoring results produced by different tools. To address uniform interpretation, for example, of the WCAG and provide consistent results, the W3C has produced a recommendation – Accessibility Conformance Testing (ACT) rules, which are constantly evolving . It should be noted that Bulgaria reported that iOS-based applications are not used by a significant number of people with disabilities, because no adequate speech synthesiser is currently available in Bulgarian for that platform. During the interviews, several MS stated that for the next monitoring period they would welcome clearer guidance on how to report results of the monitoring activities and a standard template at the EU level. This could substantially improve the outcomes’ reporting comparability and results’ analysis. Another concern that emerged in the interviews is the high burden that the current monitoring process imposes on monitoring bodies, which often do not have enough expert staff to carry out these activities. When asked about their reasons for not monitoring all requirements, some monitoring agencies stated they were not to be aware of the current requirements, and others said that the external experts hired to carry out the audits did not have the skills to do so. One possible reason for the difference in scope may be that the automated tools used to support the testing cater to WCAG and do not contain all the HEN requirements. The Commission had flagged this as a topic for discussion in the WADEX meetings and has published detailed information about this topic online . Some MS reported that their current list of digital public services is incomplete and/or unreliable as mapping and counting all public sector websites and mobile applications is impractical, if not impossible, particularly given the many different administrations involved at national, regional and local level. This limits the MS’ capacity to consider the entirety of websites and mobile applications in scope of the legislation. 1.8.5Member States have committed to training and awareness-raising Article 7(4) of the WAD states that MS should facilitate training programmes and awareness-raising activities on the WAD and its main provisions, asking for the relevant information in a dedicated section of the monitoring reports. All MS which delivered the report provided this information, with the exception of Ireland and Portugal. The monitoring report from Lithuania only shared the country’s plans for future training in this area. The remaining 22 countries stated that they had implemented the provisions of Article 7(4) mainly through three channels: (i) setting up dedicated events and workshops at the national and sub-national level; (ii) organising training programmes for different stakeholder categories ; and (iii) drafting technical guidelines and materials . The analysis of the literature reveals several other factors influencing the adoption and implementation of web accessibility standards in PSBs . These include: (i) assigning clear responsibilities; while stakeholders claim to be in full control over their part of the process, once confronted with non-conformance of the end result, they pointed to others for responsibility; (ii) management decisions; in municipalities where managerial commitment to the website and online services is high, stakeholders perceive fewer problems; (iii) perceived benefits; if management does not perceive sufficient benefits from better accessibility, priorities move away from accessibility; and (iv) legislation; although it does not directly guarantee the proper implementation of accessibility standards, it has a positive influence on the process. Opinions differed over the involvement of people with disabilities in the implementation of the WAD obligations. Several engagement activities and procedures were launched by national authorities to consult relevant stakeholders about the implementation of the WAD. Some MS even set up permanent committees with representatives from DPOs, industry, academia and public authorities. Furthermore, many of them involved end-user groups and relevant organisations in the process of selecting websites and mobile applications to be tested for the first monitoring activities, an obligation under the Implementing Decision . The involvement of people with disabilities ranged from consulting on strategy and advising on the mechanisms implemented by the WAD to being involved in user testing, providing the views and knowledge of the group benefitting most from the proper implementation of measures and monitoring, following the principle “nothing about us without us”. The facilitation of training and awareness raising in MS can be considered a success factor in the implementation of the WAD. Since the beginning of the monitoring period, a variety of training initiatives have been promoted by most of the MS. Several countries introduced training on accessibility topics, mostly targeting employees in PSBs. For instance, Czechia is launching a new course for PSBs on web accessibility, which will provide a certificate to participants who complete the course. Some MS (e.g. Slovakia and Spain) reported that new training courses were devised based on the results of the monitoring activities. However, a 2019 survey conducted by the European Disability Forum showed significant dissatisfaction with the level of engagement from relevant stakeholders. Respondents criticised the lack of consultation by MS before the adoption of the national laws and denounced the poor guidance and lack of support for stakeholder organisations. In addition, the results of the WAD review survey targeting DPOs show that 60% of DPOs (29 out of 48) reported little or no involvement in either the implementation of the WAD or in the selection of websites and mobile applications to be monitored. 1.8.6Feedback and enforcement procedures under-reported and under-used The Implementing Decision (EU) 2018/1524 requires MS to describe and detail their use of the enforcement procedure set up at the national level, to be used when initial feedback has failed. MS may also (but are not obliged to) give data on the feedback received by the PSBs through the first-level feedback mechanism laid down in Article 7(1)(b) of the WAD. On the enforcement procedure, 18 monitoring reports describe the procedure in place, while only two countries (Germany and Finland) reported quantitative information on the complaints they had received. The enforcement procedures of various MS include the possibility of using administrative fines for PSBs that do not comply with the WAD . In general, countries did not give any quantitative information about the feedback mechanism, apart from Finland, which shared results from a survey it had conducted on the mechanism. On the issue of feedback, several countries filled in the section devoted to feedback mechanisms with information that seemed unrelated. Some countries reported only on complaints or on feedback given to the monitoring body by PSBs or experts. Such shortcomings in the reports suggest that the optional reporting requirements on feedback mechanism were either unclear or left room for interpretation to the MS. In addition, from the public consultation it appears that both these procedures are little used. While the lack of complaints could be seen as very positive (e.g. no accessibility issues found), among the reasons reported for not complaining were the inaccessibility of feedback forms due to use of the online ‘captcha’ tools and a perception that a complaint would not change anything. Some individuals who had complained reported negative experiences, as enforcement bodies did not always reply satisfactorily to their complaint, nor seem to be aware of the mechanism. This lack of complaints, however, should not be fully interpreted as an indication of a malfunctioning instrument. It may also be the case that the answers given or actions taken in the feedback mechanism, are satisfactory in not giving rise to further formal complaints.

The awareness of these mechanisms should continue to be monitored, as further research on the needs and barriers for end-user possible feedback and complaint may be needed, given the current low level of use. The box below presents the findings from the monitoring reports on the enforcement procedures. The enforcement procedures of most of the MS were described in the monitoring reports. These vary country by country depending on their constitutional structures. For instance, in Austria, enforcement is performed at the federal level, while in Germany each state has its own enforcement body, where the Conciliation Body acts at the federal level. In most of the other MS, both the examination of the complaints and the enforcement is carried out at the central level by the agency or the ministerial department which works on digital accessibility. Despite some differences, there are some similarities in the overall procedure, as most MS follow a three-staged approach to enforcement. Firstly, the non-compliance complaints are collected and examined by the responsible body, which decides whether to start a procedure. Secondly, the body may issue binding instructions if a breach of the accessibility requirements is found, specifying a deadline for compliance. The deadlines are usually between 1 and 3 months. Thirdly, if the organisation fails or refuses to comply, the responsible body will proceed with enforcing the regulations. While in some countries (e.g. Germany and Luxembourg) there are mediation procedures in place, in others (e.g. Estonia), disciplinary proceedings can be conducted against an official if there are infringements. In most cases, the enforcement body has the power to decide on the infringement and issue an order to the PSB. However, in some other cases, when complaints are handled through mediation, the complainer may need to turn to the courts if the parties cannot reach an agreement. Finally, in a few cases (mainly in some states in Germany), the right to complain is limited to people with disabilities, people whose rights were infringed, or their representative organisations, limiting the personal scope of the complaint mechanism laid down in the WAD. Some MS made it possible to impose administrative fines, but reports did not provide data about the PSBs fined or about the intention of using fines in practice. To conclude, enforcement mechanisms are in place, and while differences still exist in terms of detail, the MS appear to be taking this seriously within the framework of national legislation. 1.8.7Accessibility experts are needed, as job opportunities exist MS highlighted the lack of accessibility expertise both internally in the public sector and available to hire. This was a particular problem when MS needed to hire external auditors (testers) for the monitoring activities. The public consultation also identified the most relevant challenges as: (i) PSB web managers often lack skills in web accessibility; (ii) technical difficulties in making existing websites/mobile applications compliant; and (iii) a lack of resources (both staff and money). The WAD has shown beneficial effects in stimulating the job market for web accessibility services and capacity building. This has created new job opportunities and profiles (e.g. web accessibility specialist, web accessibility coordinator, web accessibility software engineer, web accessibility analyst) , and changed existing ones (e.g. the work of front-end engineers, user-interface developers, etc.). These findings are also confirmed by the available literature on the topic , as testing for accessibility requires distinct skills, as well as the use of specific tools. However, despite the increased number of private service providers in the EU after the implementation of the WAD, MS monitoring bodies reported a shortage and/or high cost of experts to conduct accessibility audits. A Wall Street Journal article from September 2021 cited LinkedIn data that the number of job listings with “accessibility” in the title had grown 78% in the year August 2020 – July 2021. In September 2022, the EURES job database listed over 1,000 job vacancies with accessibility in the title . As manual accessibility checks must be done locally in the local language(s) it is important to ensure there is sufficient accessibility expertise in all regions in all the MS. The accessibility industry representatives interviewed expect that in the coming years the job market will develop even more, driven by higher demand from PSBs and by the new legal obligations for the private sector set out in the European Accessibility Act . Digital accessibility therefore needs to feature far more prominently in information technology courses, as well as usability design training. For more than a decade, the Digital Competence Framework for Citizens (DigComp) , the EU-wide framework for developing and measuring digital competence, has provided a common understanding, across the EU and beyond, of what constitutes “digital competence”. In 2022, the Digital Competence Framework – DigComp 2.2 – was updated to explicitly mention ‘digital accessibility’ across several knowledge, skill and attitude competencies, including the awareness of legal requirements such as accessibility statements. This should ensure accessibility is recognized as a basic part of digital competence and is taught throughout the MS. This fits also the targets and policy of the Digital Decade which is about making digital technology work for people and businesses and with the target of 20 million ICT specialists in the EU by 2030 , with significant funding under the DIGITAL programme to bridge the digital skills gap . There is clearly a high need for digital skills when it comes to integrating digital technologies in the work processes, including digital accessibility skills for both civil servants, and for contractors. Skills and competences of civil servants could be systematically boosted through specialised training. One should emphasize the opportunities for people with disabilities in this growing job market, particularly given the surge in remote work , which may help some. In most EU countries, the percentage of remote jobs has increased nearly 5 times compared with the pre-COVID period, especially in the Information & Communication Technology sector . People with disabilities could be involved as professional accessibility auditors (testers) of websites and mobile applications, which can be stepping-stone jobs into the workplace. This is an important issue, as the unemployment rate for persons with disability is significantly and persistently higher than that of the general population . MS also identified a lack of official skill certification systems . Currently, the IAAP (International Association of Accessibility Professionals) offers the main internationally recognised professional qualification, with four types of certifications for online accessibility. In practice, auditors and external experts are generally hired based on their professional experience and references they can provide, rather than on professional qualifications. 1.8.8Accessibility is increasing, but more is needed, especially at regional and local levels As the public sector continues to digitise its delivery of public services, the expectation increasingly is for the public to use online services by default, also in line with the Digital Decade target to have 100% of key public services online by 2030. According to targeted consultations organisations representing persons with disability, consumers and older people, the general opinion is that the WAD has been more effective at improving the accessibility of PSB websites at the national level, and, to a lesser extent, at the regional and local level in all MS . 51% of these respondents believed that, since the application of the WAD, there has been an improvement in online information available on websites at national level. The statistics suggest a slightly lower impact on the accessibility of websites at the regional (45% reported an improvement) and local level (38%). This may be related to the size of public administrations, as larger PSBs tend to have more budget, staff and experience to enforce web accessibility. It is hoped and expected that as PSBs progress to the digital decade targets, accessibility will be embedded by design from the start – e.g., as observed in national recovery and resilience plans that feature prominently measures for digitalising public services and introducing or improving e-government solutions to modernise public administration. To close the digital divide in rural and remote areas, the EU supports investments in an unprecedented manner. For example, around EUR 16 billion Recovery and Resilience Facility (RRF) reforms and investments have already been approved to roll out digital connectivity networks in the next four years, especially in rural regions . On the adoption of similar standards in the private sector, as encouraged by Recital 34 of the WAD, web accessibility has been – and still is – a low priority in the private sector (see below Box 3). Some of the examples mentioned at Recital 34 will be covered by the EAA. Industry players reported that the WAD, together with the EAA, should help change this situation, with the scope of accessibility obligations expanded under the EAA, so introducing incentives for private actors to develop accessibility solutions (see section 4.1.11). In this context, many MS went beyond the scope of the WAD in their national laws, as detailed below. Examples of national legislation that goes beyond the scope of the WAD. Legislation that partially covers the private sector in addition to PSBs: In France, the relevant legislation also covers private enterprises with a commercial turnover of more than EUR 250 million. In Finland, various specific private companies also fall under the scope of the national legislation when their services are intended to provide public services (e.g. companies providing public services within the water, energy, transport or postal service sectors; insurance companies ). Finland did not monitor these private sector websites and mobile applications in the first round of monitoring, but plans to include them in the next one. Many MS opted to included schools, kindergartens and nurseries in their legislation Other MS limited the technical exemptions: Finland and the city of Bremen (Germany) set accessibility requirements for public procurement. Czechia and Latvia do not exempt file formats from the Microsoft Office suite of software products. Latvia also does not exempt time-based media (both pre-recorded and live) and third-party content. Slovenia generally does not allow exemptions if the content is necessary for administrative services, while Slovakia does not mention any exclusions in the national legislation. Poland created an obligation to publish an accessible version of any PSB content published on a third-party website if it is not accessible, and in Sweden content published by a PSB via a third party needs to be made as accessible as possible.

1.8.9Technological advancements could make accessibility easier for content types not covered by WAD Article 1 of the WAD allows for the temporary or permanent exclusion from its scope of some types of content, some websites, and some mobile applications. Technological advancements in digital accessibility could make accessibility easier for content types currently excluded from the scope of the WAD due to lack of automated or efficient and easy-to-implement means at the time of adoption of the WAD . Analysis on the technologies that have significantly evolved in the last years shows that automation through artificial intelligence and machine learning is increasingly used in tools relevant for web accessibility . Respondents to the public consultation identified the following six technological advances as being the most relevant for the WAD: ·assistive technologies driven by artificial intelligence (AI); ·accessible authoring tools; ·AI and machine learning applied to monitoring; ·accessibility monitoring and testing tools; ·the ‘internet of things’ for accessible products and services; and ·biometrics for identification and security aspects. The experts interviewed agreed that automation is increasingly being used in different tools relevant for web accessibility. Based on the level of maturity, relevance for the WAD and size of the affected user groups, the study prioritized 13 technologies for deeper analysis (e.g., speech or sign language recognition, automated accessibility testing, web authoring tools) in scientific fields such as machine learning, natural language processing, deep learning, computer vision, etc . To highlight some of the most relevant, the market offers automatic captioning for videos or online videoconferencing tools, built on speech recognition technologies, but only in major languages. While the quality of captions (subtitles) is incrementally improving, so is language coverage. This is a welcome development for WAD, as accurate automatic captions in official EU languages (and others) will make it cheaper and easier to subtitle live time-based media, currently excluded from the WAD. This will help PSBs to comply with the requirement to subtitle pre-recorded time-based media, which are in scope of the WAD. There are promising developments in automated text simplification and summarisation to support people with cognitive disabilities or reading difficulties . Research continues into digital access to sign languages , and speech / text multimodal conversion . While all promising, there is not sufficient maturity in all languages to modify the current exemptions. Except for English and, to a certain extent, German, Spanish and French, at present all European languages are massively under-resourced in terms of data, tools and technologies . It should be noted that the context is evolving in favour of accessibility outside the WAD obligations. Social media platforms, for instance, make some such tools available by default. This evolving context is changing expectations and raising awareness, which may have a positive impact also on PSB websites and applications even before any potential revision of the WAD. Given the likely spill-over effect of the increased accessibility requirements of the EAA from 2025 onwards, there appears to be no pressing need to revise the WAD exceptions in the near term. 1.8.10Efficiency, adequacy and proportionality of the resources This section presents evidence-based analysis on (1) cost-effectiveness of the WAD and (2) adequacy and proportionality of the resources. Overall, the supporting study suggests that the WAD was designed and implemented efficiently, and the use of EU resources was adequate and proportionate . However, a detailed assessment of economic, social, and environmental impacts is limited, mainly due to the difficulties in quantifying and monetising web accessibility, inclusion and non-discrimination, as well as due to the limited availability of relevant data. The social impact of the activities for persons with disability in the digital sphere cannot be measured in the absence of comprehensive inclusion and equality data. The main costs imposed by the WAD are those for PSB to comply with the accessibility standards and those for the monitoring activities borne by MS. However, it is difficult to measure such costs, due to a lack of quantitative data. The evidence collected shows that accessibility related costs have not constituted a major barrier to implementing the WAD, as the national monitoring reports show that there was limited use of the disproportionate burden exemption clause. The benefits can therefore be mainly measured in qualitative terms. According to the supporting study the main economic benefits are an increase in the demand for web accessibility products and services in the national markets, driven by PSBs, and an increase in opportunities for accessibility service providers to penetrate foreign markets.

With little information publicly available about the costs, it is not surprising that the majority of respondents (58%) were neutral on whether the objectives of the Directive had been achieved at a reasonable cost (out of 311 respondents, 81 neither agreed nor disagreed, 101 were not sure). However, a majority of the respondents (184 or 59%) agreed with the statement that the benefits arising from the availability of accessible websites and mobile applications outweigh the costs of implementing the WAD. According to 50% of the PSBs consulted (31 out of 62) in a targeted survey, the main benefit of the WAD has been the possibility of providing services to a larger pool of people because of improved accessibility. Moreover, consulted PSBs appreciated the role of the WAD in encouraging organisations to provide clearer accessibility requirements in procurement procedures. Some of the stakeholders interviewed identified as benefits increased usability and fewer complaints about services. 68% of the PSBs consulted believe that the WAD introduced additional economic costs for their organisations. The main sources of recurring costs were: (i) improving the accessibility of digital services; (ii) training staff; and (iii) drawing up accessibility statements. Despite this, only a few MS reported that the WAD imposes high costs on PSBs to implement its requirements. In fact, MS reported only very few cases of using the disproportionate burden exemption clause. The monitoring bodies provided information on the costs related to monitoring activities. However, the information provided and the figures varied significantly, which makes it difficult to provide a high-level overview across all MS. The supporting study contains some of these figures. Costs were reported only by few MS and range from a minimum cost of €65,000 up to a maximum cost of €650,000. The costs of additional activities, such as awareness-raising and training, were not reported by the MS. According to the industry players replying to the targeted survey, the WAD introduced new costs, associated mainly with: (i) hiring new specialists; (ii) updating the products and services to meet the new accessibility requirements; and (iii) expanding internal capacities to meet the new, higher volume of demand. However, according to 60% of industry players, the common accessibility requirements introduced by the WAD had a positive impact on their business. The main benefits highlighted by the respondents were an increase in the demand for web accessibility products and services in the national markets and an increase in opportunities to penetrate foreign markets. The stakeholders interviewed reported that there is still a lack of industry players offering accessibility services. However, they expect a significant increase in service companies and organisations working in this area in the coming years. This represents an opportunity for the technology and educational sectors as new digital skills will be required by the market. The Digital Skills and Jobs Platform may be an important EU-wide instrument to support companies looking for specialists. Finally, the supporting study also suggests that EU funds invested in the implementation of the WAD were adequate in promoting research on web accessibility and accessible technology, and in the deployment of new solutions, as well as in supporting accessibility and social inclusion for persons with disabilities . The EU supports MS through a wide range of instruments, from strategic and operational analysis to the design and the financing of reforms and investments. As mentioned in section 4.1.8, the EU supports investments in digital transformation of the public sector in an unprecedented manner. According to a recent Commission report to the European Parliament and the Council, measures supporting the digitalisation of public services and e-government represent more than a third of the digital expenditure financed by the Recovery and Resilience Facility (RRF). The share of ‘E-government, digital public services and local digital ecosystems’ in RRF expenditure supporting the digital transformation is 36% (EUR 47 billion) . A third of that amount (around EUR 16 billion) has already been approved to roll out digital connectivity networks in the next four years, especially in rural regions. Moreover, the EU aims to also leverage connectivity investments through the new Cohesion Funds, the EAFRD, InvestEU and EIB loans, and, last but not least, through CEF Digital . Also the Digital Europe Programme recognizes that digitalisation can facilitate and improve barrier-free accessibility to all, including older people, people with reduced mobility or with disabilities, and people in remote or rural areas . It is worth noting that public administrations at central, regional and local level implement directly about 35% of the EU budget. In some MS, more than 50% of public investments were funded under the EU cohesion policy. These investments in e-government, and the digital transformation in general, should directly improve web accessibility. The Common provisions regulation explicitly spells out that operations must comply with applicable Union law and national law, which includes WAD and its transposed rules in national law. The EU has co-financed many national projects through its structural funds, which have produced several relevant outcomes, both in terms of new tools/solutions and in knowledge sharing. These can be considered a return on investment for the achievement of the two main objectives of the WAD. The Quality of Public Administration Toolbox is a summary of Commission knowledge on improving public administration, which has a dedicated knowledge base on digitalisation and accessibility. The Technical Support Instrument (TSI) aims to provide tailor-made expertise (technical support) on the ground to help MS design, develop and implement growth-enhancing reforms in a wide range of policy areas, including for digital public administration. The Commission also provides knowledge services like the Knowledge4Policy platform that brings together policymakers and scientists across the EU for evidence-informed policymaking. These centres support policymaking by monitoring the uptake and impact of artificial intelligence in the public sector. To illustrate some more detailed examples related to WAD, the EU has allocated resources to implement the WAD in two further ways. Firstly, the WADEX group, using it to facilitate the exchange of best practices between the MS, as well as with relevant stakeholders and experts where appropriate. Secondly, the European Commission funded, through the Horizon 2020 programme, pilot projects and preparatory actions, such as (e.g. WAI-Tools, WADcher, WAI-Guide, and We4Authors) in the areas of web accessibility, accessible technology, and the deployment of solutions. Most of the MS considered that their engagement in the implementation of the WAD had contributed to achieving its objectives. The WADEX group has been very useful for these stakeholders, especially by making possible to obtain and exchange technical information and to meet and engage with other people working on web accessibility. Many member appreciated the meetings as a forum to exchange views on the standards, the monitoring methodologies and other practical examples. 1.8.11Coherence with other policies and instruments The supporting study confirmed that the WAD is coherent internally, and with relevant EU and international policies and actions, as well as with the UNCRPD, the EU Charter of Fundamental Rights and the European Pillar of Social Rights. This is based on feedback provided by stakeholders during consultation activities and the review of relevant EU and international policies. The supporting study shows that the WAD is coherent internally with the overarching Commission strategies on disability, the European disability strategy for 2010-2020 and the Strategy for the rights of persons with disabilities for 2021-2030. The current strategy itself refers to this evaluation and commits to include accessibility and inclusiveness in the reinforced EU digital government strategy, focusing on human-centric and user-friendly digital public services across Europe that respond to the needs and preferences of European citizens, including the needs of persons with disabilities. Effective use of digital technologies requires the removal of accessibility barriers for persons with disabilities and investing in their digital skills. As announced in the Digital Education Action Plan 2021-2027, MS will be supported in securing assistive technologies and in providing an accessible digital learning environment and content. To promote disability-inclusive education, MS can use the opportunities offered by EU funding, including the Erasmus+ and the European Solidarity Corps programmes, setting out dedicated inclusion measures. Furthermore, for schools and educational buildings, MS can also address accessibility through the Renovation Wave aiming to make liveable environments accessible to everyone. The European Accessibility Act (EAA) complements the WAD in both scope and coverage. It has a strong digital relevance that aims to improve the functioning of the internal market for accessible products and services, by removing barriers created by divergent rules in MS. As the WAD, the EAA establishes accessibility requirements for a range of products and services that have been identified as rather important for persons with disabilities while having diverging accessibility requirements across EU countries. This includes, among others, e-commerce, banking, and transport services, concerning both public and private sectors. The MS had to transpose the EAA by 28 June 2022 and apply in practice as of 28 June 2025, so businesses across the EU need to ensure that the services they provide and products they deal with comply with the EAA’s accessibility requirements. The scope EAA is complementary to WAD, covering also websites and applications of the private sector. Presumption of conformity with the EAA will be based on the same (updated) European standard EN 301 549 harmonised under the WAD for websites and mobile applications. In addition, two other updated standards (EN17161 and EN17210) and three new ones will be developed to support specific elements of the EAA.

The WAD is also coherent with other relevant EU legislation in the field of digital policies. Most recently, the updated set of rules for all digital services within the Digital Services Act also contains an obligation for the Commission to encourage and facilitate the drawing up of Codes of Conduct. Such a code will ensure that services provided are designed in an understandable and robust way in order to ensure they are accessible to persons with disabilities, and it is publicly explained how accessibility requirements are met. Providers of services as well as civil society representatives and recipients of the services should be involved in their drafting. The Audiovisual Media Services Directive (AVMSD) obliges MS to adopt rules to impose accessibility requirements on providers of audiovisual media services. The European Electronic Communications Code (EECC) , which aims to modernise EU telecoms rules, covers service providers and requires that the services they provide be accessible. EECC does not cover content and does not lay down accessibility requirements for what is placed on the internet, but aims for equivalent access for end-users with disabilities, addressing the need for available and affordable equipment and services and the access to emergency services. The Directive and the Regulation implementing the Marrakesh Treaty share the same objectives of accessibility as the WAD but cover different types of products. The Marrakesh Treaty sets the legal framework for changes needed in national copyright regimes to implement legal copyright exemptions for the benefit of blind, visually impaired, and otherwise print disabled persons that would enable the beneficiaries or authorised entities to make accessible format copies of books and other written material. The Treaty also facilitates the exchange of accessible format copies across national borders between authorised entities or with the beneficiaries. Finally, the eIDAS Regulation, which ensures that people and businesses can use their own national electronic identification schemes (eIDs) to access public services available online in other EU countries, is also coherent with the objectives of the WAD, by requiring that trust services and end-user products used in the provision of those services shall be made accessible for persons with disabilities, where feasible. Accessible eID and trust services are highly relevant for completing processes on the web, important in public services covered by the WAD, so accessible eID and trust services should lead to increased use of such online PSB services. The requirements in the WAD together with the measures above establish a coherent regulatory system that facilitates digital accessibility both in the public and private sector. Almost all monitoring bodies interviewed during the consultation activities reported no overlaps or inconsistencies between the WAD and other relevant national measures related to web accessibility. German national law also sets another standard for checking the accessibility of PDF files (the ‘PDF UE’ standard), but the two standards, although having different requirements, do not contradict each other . The WAD fits into the aims and objectives of major treaties and instruments addressing issues of digital and social inclusion for people with disabilities, to which the Commission is party, both at European and international level, i.e. the EU Charter of Fundamental Rights and the European Pillar of Social Rights (EPSR) , the UNCRPD and the UN Sustainable Development Goals (SDGs) . The WAD is coherent with the relevant articles of the EU Charter of Fundamental Rights, including: (i) on increased social participation for older people and people with disabilities (Articles 25 and 26); (ii) on the right of access to documents (Article 42); (iii) on removing any form of discrimination, including disability (Article 21); and (iv) on freedom of expression and information (Article 11). Regarding the EPSR, coherence with the objectives of the WAD is evident especially: (i) on inclusion of people with disabilities, including through services that enable them to participate in the labour market and in society (Principle 17); and (ii) on access to essential services which states that everyone has the right to access essential services of good quality, including digital communications (Principle 20). The WAD is also consistent with and supports the proposed digital principle that everyone should have access to technology and to all key public services online across the Union. The WAD has significant synergies with the UNCRPD, currently the most relevant piece of international law in the area of disability and is coherent with its objectives. Its two most relevant articles are: Article 9, which focuses specifically on accessibility, and Article 21, which focuses on freedom of expression and opinion, as well as on the freedom to retrieve and access information. As society becomes more digitalised and most administrative services are available online, a number of other UNCRPD articles show synergies with the WAD . As a precondition of human rights, accessibility indirectly links also to the exercise of rights of persons with disabilities under the UNCRPD provisions . Finally, all of the UN Sustainable Development Goals (SDGs) are coherent with the achievement of more inclusion of people with disabilities in society. The provisions of the WAD and its implementing acts are, in particular, relevant for achieving the objectives of SDGs 1, 3, 4, 8, 9, 10, 16, and 17 as flagged by the UN Flagship Report on Disability and Sustainable Development Goals. Achieving such goals requires the accessibility of PSBs’ digital administrative procedures, including services of general interest. The list of all relevant SDGs is provided in Annex VII with a description of how the key provisions of WAD and its implementing acts contribute to these . 1.9How did the EU intervention make a difference and to whom? As described above, especially at sections 4.1.7 and 4.1.8, the WAD has directly impacted persons with disabilities, accessibility solution providers, as well as the public sector itself. The analysis considered to what extent the value of the EU intervention can be considered additional to what could have been achieved by MS acting alone. In particular, it sought to assess to what extent the WAD made a difference in removing internal market barriers and in improving digital access to persons with disabilities and older people. Finally, the analysis evaluated to what extent the WAD generated additional value compared to national measures implementing the UNCRPD, the EU Charter of Fundamental Rights and the EPSR. Most of the respondents to the public consultation agreed that there is an EU added value of the WAD in improving online accessibility for public services and in increasing the digital inclusion of users. Although information pre- and post-Directive on the accessibility of PSBs’ websites and mobile applications is limited, interviewed monitoring bodies reported a significant increase in the awareness of web accessibility among PSBs since the implementation of the WAD. In particular, the monitoring activities have been an important occasion for awareness raising. Many PSBs addressed accessibility problems after the start of the monitoring period. The other added value reported by monitoring bodies is that, for 22 MS, the adoption of the WAD meant the introduction of national laws on web accessibility that did not previously exist. The general finding from the in-depth interviews was that the MS alone would not have reached this level of awareness and web-accessibility related activity without the WAD. Concerning the EU added value in removing barriers across the internal market, most respondents across different stakeholder categories agreed that the main benefit has been the harmonisation of national legislation, which has brought tangible results in the internal market, particularly the common standard which facilitates cross-border provision of web accessibility products and services. Industry players noted that several barriers to the cross-border provision of web accessibility services had been removed. Most PSBs consulted (79%, 49 out of 62) believed that the WAD had contributed to making national laws on web accessibility requirements more harmonised. Similarly, 72% of the organisations consulted, representing persons with disabilities, older people and consumers (35 out of 48 such organisations consulted) agreed that the WAD had contributed to the harmonisation of the legislative framework on web accessibility across EU countries. In addition, 54% of the organisations also believed that the WAD enhanced cross-border cooperation among different associations of people with disabilities. The main contributions of the WAD to the national initiatives of MS have been: (i) increasing the demand for web accessibility tools from public authorities (reported by 65% of the respondents); and (ii) spreading awareness about accessibility solutions among web professionals (reported by 62% of the respondents). 61% of industry players agreed with the statement that both the demand for – and supply of – web accessibility products and services increased after the introduction of the WAD. In addition, 44% of the respondents emphasised the impact of the WAD in incentivising private actors to develop accessibility solutions. The available literature agrees that the introduction of harmonised standards is a first essential step in removing barriers in the internal market. In addition, some authors have also shown the beneficial effects of the WAD in stimulating the job market for web accessibility services . 1.10Is the intervention still relevant? The supporting study and the online surveys find that the WAD is still relevant, particularly considering the evolving European and international policy context .

As can be seen above, the overwhelming majority of respondents to the public consultation consider it still relevant for PSBs to provide accessible content and services both online (97%) and on mobile devices, including apps (95%). It is still highly relevant for WAD to contribute to digital inclusion (97%) and to reduce regulatory differences among MS in the field of web accessibility (87%). The WAD remains relevant for all main stakeholders, as the increasing use of PSB websites and online services shows the continuing need for more accessible websites and mobile applications. The amount of online information provided by public sector organisations has increased in the last three years according to 85% of respondents (265 out of 311). Similarly, most respondents reported an increase of interactive services (84%), mobile applications (78%) and online documents (72%) . PSBs overall have not reacted to the new accessibility obligations by limiting their online presence. The surveys and interviews noted only a few examples of limiting online presence, such as reducing the number of published pdf documents, removing originally live-streamed videos after 2 weeks, or publishing videos on third-party/social media platforms. Overall, MS have not widely reduced online content in order to avoid the WAD’s requirements. Indeed many MS have gone beyond the scope of the WAD in their national laws, as detailed at section 4.1.8, which implies a genuine commitment to digital inclusion for persons with disabilities. Regarding the policy context, the WAD objectives are directly relevant to the implementation of the UNCRPD. On the UN policy framework, the WAD objectives are also still relevant to several global SDGs, including SDG 10 on reducing inequality. The WAD objectives are still pertinent to the Charter of Fundamental Rights of the European Union. The fact that the WAD does directly not apply to EU institutions was considered a limitation by some of the stakeholders interviewed. The Commission has already announced measures to address this in the new Strategy for the rights of persons with disabilities 2021-2030 , and has adopted an action plan on web accessibility to ensure its web presence is compliant with the provisions of the WAD . Such intervention is also relevant in the aftermath of COVID-19 which highlighted how important it is that people with disabilities can access public sector online crisis information on equal terms . Tackling the digital divide is even more important, given the historic shift to using digital technologies in work, education, communication and public services. During the COVID-19 pandemic, digital interaction has been the public administration’s main (and often the only) communication channel with citizens, enabling it to ensure business continuity. The responses to the public consultation show that almost all respondents considered the objectives of the WAD to still be relevant today. Up to 68% of people with disabilities or older people reported using online public services of national, regional, and local PSBs at least monthly. Looking at changes since the adoption of the WAD, 50% of the respondents (142 out of 311) to the public consultation affirm that their use of government websites increased slightly (23%) or significantly (27%) in the last 3 years. Regarding mobile applications, 78% agreed that the number of applications provided by PSB has increased. On the kind of public services used online, health services were identified as the most accessed service (12%), followed by money and tax (8%), utilities (8%), benefits (7%), employment (7%), disability (6%), and citizenship services (6%). In terms of scope, almost half of the respondents (49%) to the public consultation (151 out of 311) agreed that the WAD adequately covers online public services to ensure the full participation of people with disabilities in digital society. On subject and content types not covered by the WAD, most of the respondents favoured removing the current exemptions and/or expanding the scope of the WAD to other areas. For example, full inclusion of universities in the requirements of the WAD was supported by 81% of respondents, while full inclusion of schools was supported by 70%. There was also strong support for full inclusion of NGOs (56%), online mapping services (57%), live videos (53%), extranets/intranets (51%) and third-party content (47%) given technological advances. Finally, on the Harmonised European Standard EN 301 549 (v2.1.2), many respondents (63%) to the public consultation consider that the standard has been useful for making public sector websites accessible. Around one third of the respondents (32%) agree that the standard still covers all relevant end-user groups, with half of respondents (51%) not sure or neutral on the matter. In the interviews, a few MS mentioned that services for people with cognitive impairments should be better covered. What are the conclusions and lessons learned? 1.11Conclusions The adoption of the WAD in 2016 aimed to address two major problems in web accessibility: the need to improve (i) the functioning of the internal market for the provision of web accessibility; and (ii) the poor level of accessibility of websites and mobile applications providing public services. All MS integrated the requirements of the WAD into national legislation and embarked on activities: (i) to make PSB websites and mobile applications more accessible; (ii) to ensure monitoring, reporting and enforcement of measures; (iii) to promote training programmes for PSBs on the accessibility of websites and mobile applications; and (iv) to raise awareness of both the accessibility requirements introduced by the WAD and the possibility for users to give feedback if they encounter cases of non-compliance with the accessibility requirements. 1.11.1Effectiveness Since the adoption of the WAD in 2016, progress has been made concerning both the accessibility of PSB websites and mobile applications The WAD is going in the right direction towards the achievement of its objectives, as both cross-border demand and supply of web accessibility products and services increased after the adoption of the WAD. Since PSBs still face challenges in complying with the requirements established by the WAD, there is still room for improvement. The Directive has helped to improve the internal market for providing web accessibility services. Most importantly, the harmonisation of national legislation throughout the EU has substantially improved the opportunities for p