COMMISSION STAFF WORKING DOCUMENT EVALUATION Review of the application of Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the accessibility of the websites and mobile applications of public sector bodies (Web Accessibility Directive)
Date: 2022-12-07
Institutions: Directorate-General for Communications Networks, Content and Technology, European Commission
Subjects: European standard, Internet site, access to information, computer applications, digital technology, facilities for people with disabilities, harmonisation of standards, mobile phone, public institution, software

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 7.12.2022  SWD(2022) 410 final  COMMISSION STAFF WORKING DOCUMENT  EVALUATION  Review of the application of Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the accessibility of the websites and mobile applications of public sector bodies (Web Accessibility Directive) {SWD(2022) 411 final}  
 
 
Table of contents 

1 

INTRODUCTION .......................................................................................................... 1 

1.1 
1.2 

Context of the Web Accessibility Directive ....................................................... 1 
Purpose and scope of the evaluation .................................................................. 2 

2  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? .............................. 3 

2.1 
2.2 

Description of the intervention ........................................................................... 4 
Point(s) of comparison ....................................................................................... 7 

3  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? ................... 8 

3.1 
3.2 
3.3 

Implementation of the Directive ......................................................................... 8 
National monitoring, reporting and enforcement ............................................. 10 
Current state of play and context ...................................................................... 11 

4  EVALUATION FINDINGS (ANALYTICAL PART) ......................................................... 13 

4.1 
4.2 
4.3 

To what extent was the intervention successful and why? ............................... 13 
How did the EU intervention make a difference and to whom? ...................... 39 
Is the intervention still relevant? ...................................................................... 40 

5  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? ....................................... 43 

5.1 
5.2 

Conclusions ...................................................................................................... 43 
Lessons learned ................................................................................................ 46 

ANNEX I. PROCEDURAL INFORMATION ........................................................................... 50 

ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED ........................................ 53 

ANNEX III. EVALUATION MATRIX ................................................................................... 58 

ANNEX IV. OVERVIEW OF BENEFITS AND COSTS ............................................................ 70 

ANNEX V. STAKEHOLDER CONSULTATION – SYNOPSIS REPORT .................................... 74 

ANNEX VI. ADDITIONAL CONSIDERATIONS FROM NATIONAL MONITORING REPORTS .. 89 

ANNEX VII. MAPPING OF UN SUSTAINABLE DEVELOPMENT GOALS (SDGS) .............. 92 

ANNEX VIII. BIBLIOGRAPHY ........................................................................................... 94 

 
 
 
GLOSSARY 

Term or acronym 

Meaning or definition 

AS 

DPO 

EAA 

EEA 

EPSR 

HEN 

MS 

PSB 

UNCRPD 

WAD 

WCAG 

Accessibility Statement 

Organisation representing people with disabilities 

European Accessibility Act 

European Economic Area 

European Pillar of Social Rights 

Harmonised European Standard (specifically EN 301 549) 

Member State(s) 

Public sector body 

United  Nations  Convention  on  the  Rights  of  Persons  with 
Disabilities 

Web Accessibility Directive 

Web Content Accessibility Guidelines 

WADEX 

Web Accessibility Directive Expert Group 

 
 
1 

INTRODUCTION 

This staff working document represents the evaluation of Directive (EU) 2016/2102 of the 
European Parliament and of the Council of 26 October 2016 on the accessibility of the 
websites  and  mobile  applications  of  public  sector  bodies 1,  also  referred  to  as  the  Web 
Accessibility Directive (WAD) and its implementing acts 2. 

Web accessibility means that ‘all people, particularly people with disabilities and older 
people, can use websites in a range of contexts of use, including mainstream and assistive 
technologies;  to  achieve  this,  websites  need  to  be  designed  and  developed  to  support 
usability  across  these  contexts’ 3.  In  the  context  of  the  WAD,  accessibility  should  be 
understood  as  principles  and  techniques  to  be  observed  when  designing,  constructing, 
maintaining,  and  updating  websites  and  mobile  applications  so  that  persons  with 
disabilities can perceive, understand, navigate, and interact with them 4. In this document, 
“accessibility” specifically refers to the above definition only. 

1.1  Context of the Web Accessibility Directive 

Accessibility to information and communication technologies is an enabler of rights and a 
prerequisite for the full participation of persons with disabilities on an equal basis with 
others 5. The main purpose of web accessibility is to remove the challenges and barriers 
which emerge for people with disabilities when using digital information and services on 
the web or in mobile applications. Accessibility – essential for some, and useful for all 6 – 
is therefore a key driver for the development of a knowledge-based society and makes it 
possible to pursue economic growth while ensuring social inclusion. 

Since the internet has become a preferred way for governments to provide information and 
public services 7 to citizens, people with disabilities across the EU must be supported to 
access such information and services online. Web accessibility should be considered not 
only as a set of tools needed to guarantee the participation and inclusion of people with 
disabilities in society. It should also be considered as a fundamental means of upholding 

1  Directive (EU) 2016/2102 (OJ L 327, 2.12.2016). See also legislative summary of the Directive. 
2 

Implementing  Decision  (EU)  2018/1523  (OJ  L  256,  12.10.2018)  establishing  a  model  accessibility 
statement; Implementing Decision (EU) 2018/1524 (OJ L 256, 12.10.2018) establishing a monitoring 
methodology and arrangements for reporting; and Implementing Decision (EU) 2018/2048 (OJ L 327, 
21.12.2018) establishing a Harmonised European standard. 
Petri H. et al. (2015): Towards a unified definition of web accessibility. W4A '15: Proceedings of the 
12th International Web for All Conference. May 2015. Article No: 35, pp. 1-13. 

3 

4  Recital 2 of the WAD, cited above (note 1). 
5  Communication from the Commission to the European Parliament, the Council, the European Economic 
and Social Committee and the Committee of the Regions – Union of Equality: Strategy for the Rights 
of  Persons  with  Disabilities  2021-2030,  COM(2021)  101  final,  3.3.2021.  See  also  its  Monitoring 
framework. 

6  European  Commission  (2019).  Shaping  Europe’s  digital  future,  Accessibility:  Essential  for  Some, 

Useful for All, Factsheet/Infographic (PDF), 13 May 2019. 

7  Commission Staff Working Document  – Supporting Public Administrations in EU Member States to 

Deliver Reforms and Prepare for the Future, SWD(2021)101 final, 9.4.2021. 

1 

 
                                                 
everyone’s human rights in a digitalised society, in line with the provisions of the United 
Nations Convention on the Rights of Persons with Disabilities (UNCRPD) 8. 

This is possible only if digital interfaces are designed, developed and maintained to 
ensure that they are accessible for a wide range of needs. However, as assessed in the 
Impact assessment accompanying the document ‘Proposal for a Directive of the European 
Parliament and of the Council on the accessibility of public sector bodies’ websites’ 9 (“the 
Impact  Assessment”),  before  the  adoption  of  the  WAD,  the  level  of  accessibility  of 
websites and mobiles applications in general, including those of public sector bodies, was 
low 10. In addition, the market for web accessibility-related products was fragmented 11. 

In 2016, 68.1 million persons were estimated 12 as having a moderate disability and 30.5 
million as having a severe disability. Taken together, these 98.6 million people accounted 
for about 24% of people aged 16 and above and living in the EU in private households 13. 
In 2019, about a quarter (24%) of the EU population experienced long-standing activity 
limitations due to health problems 14. These numbers clearly demonstrate the relevance of 
the issues at stake at the time of adoption. 

This is the context in which the WAD entered into force on 22 December 2016, requiring 
the websites and mobile applications of public sector bodies (PSBs) to meet accessibility 
requirements. It set the first accessibility requirements in Europe for the online space. The 
WAD was conceived as a minimum harmonisation directive, in recognition of the fact that 
some EU countries had already set higher standards or might do so in the future 15. 

1.2  Purpose and scope of the evaluation 

This staff working document is an evaluation of the WAD and its implementing acts (the 
intervention),  covering  the  first  three  years  of  its  application  (from  the  transposition 

8 

Ferri, D., & Favalli, S. (2018). Web Accessibility for People with Disabilities in the European Union: 
Paving the Road to Social Inclusion. Societies, 8(2), 40. 

9  Commission Staff Working Document – Impact Assessment, Accompanying the document ‘Proposal 
for a Directive of the European Parliament and of the Council on the accessibility of public sector bodies’ 
websites’,  SWD(2012) 401 final (PDF); Executive  summary of the Impact  Assessment,  SWD(2012) 
402 final (PDF); Proposal for a Directive, COM(2012) 721 final (PDF). 

10  According  to  the  2007  benchmarking  study  ‘Measuring  progress  of  eAccessibility  in  Europe  – 
Assessment of the Status of eAccessibility in Europe’ (MeAC 1), Main report (PDF), p. 63, only 5,3% 
of government websites passed automatic and manual accessibility checks. In 2011, the follow-up study 
‘Monitoring  e-accessibility  in  Europe:  2010-2011’  (MeAC  2),  Annual  report  (PDF),  p.  115,  using  a 
different method, found that 34% of the EU ‘web content’ (public and private) was accessible. 
Impact Assessment, cited above (note 9), p. 25. It should be noted the original Proposal for the Directive, 
COM(2012) 721 final (PDF), p. 19, covered only 12 types of public sector bodies’ websites, and did not 
cover mobile applications. 

11 

12  We refer here to what is officially reported by respondents in household surveys. It should be noted that 

the actual number of people with disabilities can vary between different data sources. 

13  Grammenos, S., 2018. European Comparative Data On Europe 2020 & People With Disabilities. Centre 

de politique sociale et économique européenne Asbl (CESEP). 

14  Eurostat (2020), Functional and activity limitations statistics (update planned in December 2022). See 

also overview of Disability statistics. 

15  As provided in Article 288 of the TFEU. See also legislative summary about EU directives. 

2 

 
                                                 
deadline of September 2018 to December 2021). In line with Article 13 of the WAD, the 
review  must  take  into  account:  (i)  the  Member  States’  reports  on  the  outcome  of  the 
monitoring (Article 8 refers); (ii) the use of the enforcement procedures (Article 9 refers); 
(iii) an analysis of the technological advancements that could make accessibility easier for 
excluded types of content. 

This  evaluation  is  conducted following the  guidance provided by the  Better  Regulation 
Guidelines and Better Regulation Toolbox 16. As for all evaluations, this report will assess: 

 

to  what  extent was the intervention successful  and why (effectiveness, efficiency, 
coherence); 

  how did the EU intervention make a difference and to whom (EU added value); 

 

is the intervention still relevant (relevance); 

and will identify 

  conclusions and lessons learned, to be considered in future policy developments in 

the area of web accessibility. 

As the United Kingdom (U.K.) is no longer a Member State, it is not considered in scope 
of this evaluation. 

This document draws on and is accompanied by an external supporting study 17. The study 
analysed  the  evidence  available  from  primary  and  secondary  data  sources 18,  including 
open and targeted consultations, in-depth interviews, a literature review and the assessment 
of  the  official  national  monitoring  reports  sent  by  the  Member  States  (MS)  to  the 
Commission 19. 

Both the study and the evaluation adopted an inclusive approach to consultation activities, 
involving all relevant stakeholder groups affected by the WAD, with a specific focus on 
the participation of people with disabilities. In particular, for the first time, an easy-to-read 
and more accessible questionnaire offered people with disabilities the chance to take part, 
including those with cognitive impairment 20. 

2  WHAT WAS THE EXPECTED OUTCOME OF THE INTERVENTION? 

The evaluation starts with an explanation of the theoretical framework developed to assess 
the performance of the WAD. This framework starts from the description of (i) how the 

16  European Commission (2021), Better regulation: guidelines and toolbox. 
17  Study supporting the review of the application of the Web Accessibility Directive (WAD), Final Report, 

PwC, Intellera Consulting, Open Evidence, Funka (2022). 

18  See Annex I for procedural information and Annex II for details on the methodology. 
19  Member State  monitoring reports are published on  national  websites and  made  available also on the 
Commission’s website Web Accessibility Directive – Monitoring reports. See Annex V for stakeholder 
consultations and synopsis report. 

20  80% of people with disabilities responding to public consultation replied to easy-to-read questionnaire 

(Annex V Synopsis report). 

3 

 
                                                 
WAD was expected to work (section 2.1) and (ii) the points of comparison on which the 
evaluation is based on (section 2.2). 

2.1  Description of the intervention 

The intervention logic of the WAD is presented in Figure 1 below. 

Relevance 

Coherence 

Efficiency 

EU added value 

Effectiveness 

Problems & 
Needs

Different legislative 
approaches to Web 
Accessibility 
(regulatory 
fragmentation)

Low level of 
accessibility of 
public sector 
websites and mobile 
applications

Different web 
accessibility 
technical standards 
(barrier for market 
players and users)

Digital and social 
exclusion

Objectives

Inputs

Activities

Outputs

Results

Impacts

Harmonised, 
coherent and 
consistent legal 
framework across 
the EU

Monitoring, 
reporting and 
enforcement bodies 
set up in all MS

Published 
accessibility 
statements

Feedback from end-
users

Public sector 
websites and mobile 
applications 
compliant with WAD 
minimum 
requirements

Increased use of 
public sector 
websites and mobile 
applications by 
persons with 
disabilities

Cross-border sales 
of accessible ICT 
products and 
services

Increased efficiency 
of public service in 
line with needs and 
preferences of 
citizens, in particular 
persons with 
disabilities

Increased citizen 
participation

Increased digital and 
social inclusion

Better digital single 
market efficiency

Competition and 
lower prices

Job creation and 
productivity growth

The provision of 
accessible online or 
mobile content and 
services by public 
sector bodies in EU 
Member States and 
EEA countries

Provisions for MS to 
ensure public sector 
websites and mobile 
applications meet 
the accessibility 
requirements

Reduction of EU 
single market 
fragmentation 
relating to 
accessibility of 
websites and mobile 
applications

Technical 
specifications in a 
European standard 
as presumed 
conformance to the 
minimum 
requirements of the 
Directive

Monitoring 
methodology for MS

Model Accessibility 
Statement for use 
by public sector 
bodies

Commission

• Support MS for 
implementation

• Facilitate 

cooperation among 
MS, and between MS 
and relevant 
stakeholders

Member States

• Transpose WAD 

provisions
• Enforcement 
procedure

• Monitor compliance, 
in line with Art. 7 
WAD

• Facilitate training and 

raise awareness

Public Sector Bodies

• Make websites and 
mobile applications 
(more) accessible in 
line with WAD, 
involving persons 
with disabilities 

• Produce accessibility 
statements with:
• Feedback 

mechanism

• Link to complaint 

mechanism

Figure 1. Intervention logic of the Web Accessibility Directive 21 

Problems and needs. The adoption of the WAD aimed to address four major problems in 
web accessibility: (i) the need to improve the functioning of the internal market providing 
web  accessibility  services,  given  varying  legislation  in  MS;  (ii)  the  very  poor  level  of 
accessibility  of  the  websites  and  mobile  applications  providing  public  services  and 
information to the public; (iii) the lack of common technical standards, which created a 
barrier for market players and therefore for end-users; and (iv) digital and social exclusion 
of people with disabilities caused cumulatively by the above mentioned problems. 

Regulatory fragmentation across the EU had hindered the potential growth of the market 
for web accessibility products  and services. A wide variation in  technical provisions  in 
calls for tenders was a serious issue for web developers who therefore faced barriers in 
operating across borders 22. Such variation was the result of different regulations at national 
– or even regional – level concerning the web accessibility of PSB websites and mobile 
applications. Such fragmentation was particularly problematic for SMEs, which could not 
bear the costs of operating in different markets requiring different standards. In addition, 
providers of tools for web accessibility (such as authoring tools or automatic testing tools) 

21  Source: Study, cited above (note 17). The relationship of the five key evaluation criteria and intervention 

logic is simplified for illustration purposes 
Impact Assessment, cited above (note 9), p. 13. 

22 

4 

 
                                                 
faced  the  additional  barriers  of  having  to  implement  all  these  different  national 
specifications in their tools. This in turn caused an interoperability problem for assistive 
technologies, as these could not be deployed in different MS with different requirements 
in place. Moreover, the lack of web accessibility also required public administrations to 
maintain  costly  alternative  channels,  such  as  call  centres,  to  ensure  the  provision  of 
services to all. 

In 2007, the study ‘Measuring progress of e-Accessibility in Europe’ (MeAC) 23 identified 
the  level  of  web  accessibility  of  PSB  websites  and  mobile  applications  detected  as 
considerably low. This was confirmed by the subsequent studies MeAC 2 (2010) 24 and 
MeAC 3 (2012) 25. People with disabilities and older people had limited or no access to 
websites,  and  were  obliged  to  seek  potentially  more  costly  alternatives  to  access 
information and services. The WAD therefore sought to lower these barriers. 

The lack of common technical standards was one factor behind the market fragmentation 
described  under  problem  one.  Together  with  the  different  legal  frameworks  in  the  MS, 
market players faced barriers, as they could not be certain that tools acceptable in one MS 
would  match  requirements  in  another  MS.  End-users  paid  the  price  for  the  lack  of  a 
functioning single market, the price being lower accessibility. Finally, digital and social 
exclusion, is caused by a cumulative effect of the above-mentioned three problems. 

Objectives. Given the above context, the WAD has two key objectives: 

1)  to ensure that the websites and mobile applications of PSBs are made more accessible 

on the basis of common accessibility requirements 26; 

2)  to  harmonise  the  internal  market  for  web  accessibility  products  and  services, 

overcoming regulatory fragmentation 27. 

Inputs. In order to achieve these two objectives, in 2016 the EU adopted the WAD, which 
requires the MS to ensure that the websites and mobile applications of PSBs meet common 
accessibility requirements introduced by the Harmonised European Standard, linked to the 
WAD  with  the  Implementing  Decision  2018/2048 28.  Furthermore,  to  ensure  that 
provisions were implemented in a consistent way across the EU, and that MS regularly 
monitored  the  accessibility  of  public  sector  websites  and  mobile  applications,  the 

23  European  Commission  (2007).  Measuring  progress  of  eAccessibility  in  Europe  –  Assessment  of  the 

Status of eAccessibility in Europe, Main report (PDF). 

24  European Commission (2011). Monitoring e-accessibility in Europe: 2010-2011, Annual report (PDF). 
25  European Commission (2013). G3ict, Study on Assessing and Promoting e-Accessibility, Final report. 
The web accessibility monitoring methodology used by Member States in recent reporting is different 
from MeAC studies and therefore the statistical results are not easily directly comparable (see below 
section 4.1). 

26  Recital 9 and Article 1 of the WAD, cited above (note 1). 
27  Recital 8 and Article 1 of the WAD, cited above (note 1). 
28  The harmonised standard was updated by Commission Implementing Decision (EU) 2021/1339 of 11 
August 2021 amending Implementing Decision (EU) 2018/2048 as regards the harmonised standard for 
websites and mobile applications (OJ L 289, 12.8.2021). The new harmonised standard is applicable 
since 12 February 2022, therefore it is not observed in this document. 

5 

 
                                                 
Commission adopted the Implementing Decision 2018/1523 29 and Implementing Decision 
2018/1524 30.  These  implementing  decisions  respectively  establish:  (i)  a  model 
accessibility  statement  (AS);  and  (ii)  a  common  methodology  for  monitoring  the 
conformity of PSBs’ websites and mobile applications with the accessibility requirements 
of the WAD. 

Activities. The WAD requires the European Commission to help the MS implement the 
WAD. To this end, in addition to adopting the Implementing Decisions, the Commission 
set  up  the  Web  Accessibility  Directive  Expert  Group  (WADEX) 31  which  supports 
cooperation between MS and with relevant stakeholders. 

MS were required to assign to specific bodies the responsibilities for: (i) monitoring and 
reporting as required in Article 8 of the WAD; and (ii) enforcement at the national level, 
as  required  in  Article  9  of  the  WAD.  In  the  course  of  setting  up  national  monitoring 
processes,  MS  had  to  involve  and  consult  end-users  with  disabilities.  The  WAD  also 
obliges MS to take the necessary measures to raise awareness of accessibility requirements 
and to promote and facilitate training programmes on accessibility. 

Finally,  PSBs  were  required  by  the  WAD  to  provide  and  regularly  update  a  detailed, 
comprehensive and clear accessibility statement (AS) on the compliance of their websites 
and mobile applications. 

Expected outputs. Four main outputs were expected: (i) a more coherent and consistent 
legislative  framework  on  web  accessibility  across  the  EU;  (ii)  the  appointment  of 
monitoring, reporting and enforcement bodies at national level; (iii) the publication of an 
AS on the websites and mobile applications of each public sector body; (iv) feedback from 
end users to PSBs on the accessibility of their websites or mobile applications, leading to 
virtuous improvement cycles. 

Expected  results.  In  line  with  the  two  key  objectives  mentioned  above,  the 
implementation of the WAD was expected to lead to two types of results. 

First, a harmonisation of the internal market for web accessibility products and services, 
overcoming  the  regulatory  fragmentation  existing  prior  to  2016.  In  the  identified 
intervention  logic,  the  overall  results  were  divided  into  three  measurable  outcomes:  (i) 
increased  cross-border  sales  of  accessible  information  communication  and  technology 
(ICT) products and services; (ii) more companies offering accessibility expertise; and (iii) 

29  Commission  Implementing  Decision  (EU)  2018/1523  of  11  October  2018  establishing  a  model 
accessibility statement in accordance with Directive (EU) 2016/2102 of the European Parliament and of 
the Council on the accessibility of the websites and mobile applications of public sector bodies (OJ L 
256, 12.10.2018). 

30  Commission  Implementing  Decision  (EU)  2018/1524  of  11  October  2018  establishing  a  monitoring 
methodology and the arrangements for reporting by Member States in accordance with Directive (EU) 
2016/2102 of the European Parliament and of the Council on the accessibility of the websites and mobile 
applications of public sector bodies (OJ L 256, 12.10.2018). 

31  Register of Commission expert groups and other similar entities (E03475). See also its activities on Web 

Accessibility Directive Expert Group (WADEX). 

6 

 
                                                 
a  reduction  in  prices  for  web  accessibility  products  and  services  due  to  increased 
competition in the market. 

Second,  an  increase  in  EU  Member  States  and  EEA  countries  of:  (i)  the  number  of 
accessible PSB websites and mobile applications (according to the WAD’s requirements) 
and (ii) the use of PSB websites and mobile applications by people with disabilities. 

Expected  impacts.  The  WAD  aimed  at  increasing  digital  and  social  inclusion  by 
improving  access  for  people  with  disabilities  to  public  sector  websites  and  mobile 
applications.  Greater  public  participation  was  envisaged,  measured  through  the  use  of 
online  public  services  and  the  greater  ‘co-design’  of  public  policies  (i.e.  where 
policymakers and the public work together to design public policies). Wider impacts were 
also expected in the market due to legislative harmonisation and the reduction of regulatory 
barriers across the EU. The two most significant market results expected were: (i) a more 
efficient digital internal market; and (ii) web accessibility market growth, contributing to 
economic  growth  and job creation. The final  expected impact  of the WAD was  greater 
digitalisation-led efficiency in the delivery of public sector services. 

2.2  Point(s) of comparison 

The  Impact  Assessment  estimated that the EU policies and actions available before the 
introduction of the WAD would not have been sufficient to solve the problems and needs 
explained in section 2.1. The market would have remained fragmented, and the low level 
of  web  accessibility  in  the  public  sector  would  have  persisted.  Members  of  the  public 
unable to interact with online public services would have continued to experience ‘digital 
exclusion’ and limitations on their social, economic, and civil participation. Given that the 
Impact Assessment was drafted in 2012, the data gathered at the time of the initial proposal 
has  limited  relevance  today,  due  to  the  rapid  increase  in  digitalisation  since  then. 
Consequently, the evaluation re-conducted the analysis to the five main evaluation criteria 
suggested  by  the  Better  Regulation  Guidelines:  effectiveness,  efficiency,  coherence, 
relevance and EU added value. 

In light of these  considerations, the intervention  logic described in  section 2.1 explains 
how the inputs and activities required to meet the objectives of the WAD are expected to 
generate  outcomes,  namely  short-term  outputs,  medium-term  results  and  long-term 
impacts.  These  elements  will  be  used  as  the  main  points  of  comparison  to  assess  the 
performance of the WAD. Such points are built on the evaluation framework provided in 
the Impact Assessment and are directly linked to the evaluation matrix presented in Annex 
III: 

  Point  1:  relation  between  the  observed  results  (medium-term)  and  impacts  (long-
term) and the objectives of the WAD. To what extent have the two main objectives 
of  the  WAD  been  achieved  –  measured  through  the  effectiveness  criterion  (see 
sections 4.1 and 5.1.1 for the results of the evaluation). The analysis considered (i) 
the most relevant success factors for the application of the WAD and (ii) the gaps 
and challenges hindering the achievement of the objectives. 

7 

 
  Point 2: relation between the inputs and activities put in place for the implementation 
of the WAD and the observed outcomes. The costs borne to achieve the outcomes of 
the  WAD  –  measured  through  the  efficiency  criterion,  i.e.  to  what  extent  did  the 
provisions of the WAD produce outputs at reasonable costs (see sections 4.1.10 and 
5.1.2). 

  Point 3: relation between the inputs and activities conducted to implement the WAD 
and the problems to be addressed. To what extent is the WAD complementary with 
other legislative initiatives at the European and national level on web accessibility of 
public services – measured through the coherence criterion, i.e., the coherence of 
the WAD with other legislative interventions in the area of web accessibility and 
inclusion, at national, international and EU level, identifying any complementarities 
or inconsistencies (see sections 4.1.11 and 5.1.3). 

  Point 4: the difference made by the EU in achieving the outcomes of the WAD. To 
what extent has an EU-level intervention reached outcomes which could not have 
been achieved by MS on their own – measured through the EU added value criterion 
(see sections 4.2 and 5.14). 

  Point 5: relation between the inputs and activities conducted to implement the WAD 
and the objectives. To what extent has the WAD been relevant to achieve its initial 
objectives – measured through the relevance criterion (see sections 4.3 and 5.1.5). 

For each of these five points, the evaluation matrix presented in Annex III both explains 
the key questions set to perform an evidence-based assessment, and describes the related 
indicators and data sources used to judge the performance of the WAD. 

3  HOW HAS THE SITUATION EVOLVED OVER THE EVALUATION PERIOD? 

3.1 

Implementation of the Directive 

The WAD was implemented through the following milestones: on 22 December 2016, the 
WAD  entered  into  force.  By  23  September  2018  (transposition  deadline),  MS  were 
required  to  transpose  the  WAD  into  their  national  legislation  and  designate  bodies 
responsible  for  monitoring  and  reporting  functions,  as  well  as  for  enforcement  of  the 
WAD. MS were to apply national legislation transposing the WAD in practice according 
to the following timeframe: 

 

 

 

from  23  September  2019  for  websites  published  after  22 September 2018,  i.e.  for 
‘new websites’; 
from 23 September 2020 for all other websites of PSBs, i.e. for all websites, old and 
new; 
from 23 June 2021 for mobile applications of PSBs. 

In line with Article 8 of the WAD, all MS are required to submit every 3 years a report 
(the  ‘monitoring  report’)  of  the  results  of  monitoring  and  enforcement  procedures.  The 
first submission was due by 23 December 2021. 

8 

 
At European level, in  2016 the European Commission  set  up the WADEX  group 32  to 
support uniform application of the WAD at national level. WADEX is expected to achieve 
this goal by fostering cooperation and the exchange of best practices on web accessibility 
between MS and relevant stakeholders. 

Three  implementing  decisions  were  adopted  in  2018,  to  support  a  consistent 
implementation  of  the  WAD  across  the  EU.  These  implementing  acts  are  directly 
applicable and need not be transposed into national law. 

  Implementing  Decision  (EU)  2018/1523 33  laid  down  a  model  accessibility 
statement  which  PSBs  must  follow  to  ensure  that  their  websites  and  mobile 
applications comply with the WAD (WAD Article 7 refers). 

  Implementing Decision (EU) 2018/1524 34 set  up a monitoring  methodology  and 
arrangements  for  the  reporting  every  3  years  required  from  MS  (WAD  Article  8 
refers). 

  Implementing Decision on the harmonised standard (2018/2048) 35 for websites 
and mobile applications. This implementing decision provided for the presumption 
of  conformity  with  the  accessibility  requirements  recognised  by  the  WAD  (WAD 
Article  6  refers).  Subsequently  amended  by  Commission  Implementing  Decision 
2021/1339 36, which refers to the latest version of the Harmonised European Standard 
EN 301 549 V3.2.1 (2021-03) 37. 

All MS have officially transposed the WAD into national legislation 38. 

Six  MS  completed  the  transposition  within  the  requested  timeline:  Slovenia,  the 
Netherlands, Denmark, France, Italy and Spain. The U.K., still a Member State at the time, 
also transposed the WAD on time. The MS that did not transpose fully by the deadline can 
be divided into three groups: 

  12  MS  notified  the  transposition  measures  after  the  deadline:  Ireland,  Greece, 
Croatia, Cyprus, Latvia, Luxembourg, Malta, Poland, Portugal, Romania, Finland, 
Sweden; 

32  Meeting minutes available in Register of Commission expert groups and other similar entities (E03475). 
Since 2016, the expert group met in total 32 times. From 2020 annual in-person meetings were replaced 
by series of online meetings and webinars. WADEX members collaborate on  Futurium platform and 
Microsoft Teams. 
33  Cited above (note 29). 
34  Cited above (note 30). 
35  Commission Implementing Decision (EU) 2018/2048 of 20 December 2018 on the harmonised standard 
for websites and mobile applications drafted in support of Directive (EU) 2016/2102 of the European 
Parliament and of the Council (OJ L 327, 21.12.2018). 

36  Commission Implementing Decision (EU) 2021/1339, cited above (note 28). 
37  EN 301 549 V3.2.1 (2021-03) (PDF), Harmonised European Standard, ‘Accessibility requirements for 

ICT products and services’, ETSI, CEN, CENELEC. 

38  EUR-Lex portal: list of national transposition measures communicated by the Member States concerning 

Directive (EU) 2016/2102. 

9 

 
                                                 
  6 MS notified pre-existing measures within the deadline, without providing evidence 
of full compliance with the WAD: Bulgaria, Czechia, Estonia, Hungary, Lithuania, 
Slovakia; 

  3 MS with federal systems which did not adopt measures in all regions within the 

deadline: Austria, Belgium, Germany. 

3.2  National monitoring, reporting and enforcement 

MS  were  required  to  designate  monitoring,  reporting  and  enforcement  bodies  by  23 
September 2018. The three bodies must perform the functions set out below. 

Monitoring  bodies  must  annually 39  monitor  the  accessibility  of  public  sector  websites 
and  mobile  applications  and  provide  information  to  PSBs  on  how  those  websites  and 
mobile  applications  are  complying  with  the  accessibility  requirements.  They  may  also 
check  for  the  compliance  with  the  WAD  of  accessibility  statements  published  on  the 
websites and mobile applications of PSBs. 

Enforcement  bodies  must:  (i)  ensure  the  availability  of  an  adequate  and  effective 
enforcement procedure to guarantee the compliance of websites and mobile applications 
with  the  accessibility  requirements;  (ii)  investigate  when  a  PSB  affirms  that  making 
content accessible would be an excessive burden for them (the so-called “disproportionate 
burden”) 40;  and  (iii)  ensure  that  feedback  given  by  users  is  effectively  handled  (both 
notifications about content that is not accessible and requests for content excluded from 
the accessibility obligations). 

Reporting bodies are required to publish in an accessible format and to submit reports on 
the results of monitoring and enforcement activities to the Commission every 3 years. 

All Member States (MS) met the obligation to appoint these bodies. In 12 MS monitoring, 
reporting and enforcement are performed by the same authority, while 15 MS entrusted 
these functions to separate bodies. Austria, Belgium, Germany, Spain and the Netherlands 
assigned  responsibility  for  enforcement  to  the  relevant  regional  or  local  level  of  public 
administration bodies. The full list of bodies appointed by MS and notified to the European 
Commission is published on a dedicated web page 41. The assignment of multiple or single 
function to the bodies concerned does not appear to have any impact on their efficiency 
and is in line with the WAD requirements. 

39  Except  for  the  first  monitoring  period,  which  was  two  years  for  websites  (1 January  2020  and 
22 December 2021) and six months for mobile applications (23 June 2021 and 22 December 2021). 
40  According  to  Recital  39  of  the  WAD,  disproportionate  burden  means  that  the  necessary  measures 
“would  impose  an  excessive  organisational  or  financial  burden  on  a  public  sector  body,  or  would 
jeopardise the body's capacity to either fulfil its purpose or to publish information needed for or relevant 
to its tasks and services, while taking into account the likely resulting benefit or detriment for citizens, 
in particular persons with disabilities”. 

41  List of Member States’ bodies in charge of monitoring, reporting and enforcement under the Directive. 

10 

 
                                                 
Currently there are 25 national reports available, covering all EU Member States except 
France and Cyprus 42. All the monitoring reports are published on a dedicated Commission 
website 43,  as  well  as  on  national  websites.  According  to  these  reports,  total  of  10 412 
websites and 298 mobile applications have been tested for accessibility by MS during the 
first  monitoring period, in  what  has been described as the  ‘world’s largest  accessibility 
test’ 44. 

3.3  Current state of play and context 

We cannot consider the period under evaluation (2018-2021), without mentioning COVID, 
and  how  it  accelerated  the  digital  transformation.  The  pandemic  made  PSB  crisis 
information essential, and obliged use of online PSB services, to an unprecedented extent. 
Part  of  the  Commission’s  response  has  been  the  Digital  Decade  policy 45,  which  sets 
ambitious targets for further digitalisation of the public sector. 

This  context  helps  explain  some  of  the  apparent  contradictions  in  the  public  feedback, 
which identified progress  but  also  the need for further improvements. To the challenge 
facing  MS  at  the  start,  of  identifying  and  improving  accessibility  of  all  existing  PSB 
websites and mobile applications, were added the sudden need to digitise more, to limit, 
organise or replace face to face transactions, and the greater use of online PSB information 
and services across all government departments and levels, due to the pandemic. 

The  2022  eGovernment  Benchmark  2022  report  found  that  only  16%  of  evaluated 
eGovernment websites passed all of the selected 8 accessibility criteria 46, and expressed a 
need for more accessible public sector websites 47. However, looking closer at the source 
data 48  and  test  results  per  criteria,  28%  of  websites  did  not  fail  any  criterion  (e.g.  test 
passed or inapplicable). Although this pilot indicator 49, measured with automated tools on 

42  This reflects the situation in September 2022, and all Member States that have not yet submitted reports 
have committed to do so. Although the WAD applies to EEA countries, transposition in these countries 
is not yet completed, and the EEA countries have not yet had an obligation to report. 

43  List of Member States’ Monitoring reports (in original language and automated translations in English). 
44  Wilco Fiers (Deque.com), EU Runs World’s Largest Accessibility Test, 1.02.2022. 
45  2030 digital compass: the European way for the Digital Decade, COM(2021) 118 final, 9.3.2021. See 

46 

47 
48 

49 

also Europe’s Digital Decade: digital targets for 2030. 
eGovernment  Benchmark  (2022),  Insight  Report  (PDF),  p.  9.  The  benchmark  compares  how 
governments across Europe deliver digital public services. The method update of 2020 has led to a break 
in the series, which makes comparisons with earlier reports impossible. 
Ibid., p. 18. 
eGovernment  Benchmark  (2022),  Non-scored  indicators  (.xlsx).  The  pilot  tested  8  out  of  50  Web 
Content Accessibility Guidelines (WCAG) 2.1 success criteria. A ‘pass’ means that content element is 
present on the web page (e.g. image) and it passes alternative text check. ‘Fail’ means it did not pass the 
check. ‘Inapplicable’ means that there is no image to test on the web page. In the sample, the total of all 
checks 76% passed, 17% failed, 7% were inapplicable. 83% websites passed five or more criteria. 
eGovernment Benchmark (2022), Background Report (PDF) – Synchronising Digital Governments, pp. 
20-22. Web accessibility foundations indicator pilot means that it is not included in the scoring of the 
current benchmark. The compliance rate of websites with the selected criteria ranged between 40,9% 
(WCAG:  1.4.3  colour  contrast)  and  99,5%  (WCAG:  4.1.2  name,  role,  value),  suggesting  that  some 
accessibility criteria are better implemented than others. 

11 

 
                                                 
selected criteria, does not determine compliance with the WAD, it helps determine if web 
content meets accessibility guidelines. 

This pilot gives an insight into the state of accessibility, but the results cannot be compared 
directly  to  earlier  studies  (e.g.  MeAC 50)  or  even  national  monitoring  reports,  due  to 
different  testing methodologies and the  fact  that for a complete  evaluation of website’s 
accessibility, additional manual evaluations are needed to verify full compliance. 

While administrations across Europe largely rose to the challenge and accelerated digital 
service  transformation  in  the  face  of  the  coronavirus  pandemic,  the  eGovernment 
Benchmark  2022  report  suggested  to  prioritise  user-centric  design  to  ensure  that 
eGovernment services are inclusive to users in all their diversity, including users with poor 
digital skills or those living with disabilities. 

The  2020  Berlin  Declaration  on  digital  society  and  value-based  digital  government 51 
acknowledges the public sector as  an essential element for the European Single Market 
and emphasises the importance of digital public services in our everyday lives. According 
to the first progress report on the monitoring of the Berlin Declaration 52, in 2022 the EU 
average score was 60% in policy area 2 on ‘social participation and digital inclusion to 
shape the digital world’ 53. 

 The revised European Interoperability Framework 54 looks at principle of inclusion and 
accessibility, and observed an improvement of EU average scores since 2019 55. 

By 2021, the share of individuals  using internet  in  the EU had risen to 90%, some 20 
percentage  points  higher  than  in  2011 56.  In  2022,  the  share  of  individuals  interacting 

50  Cited above (notes 23-25). For example, 2010 MeAC 2 study, cited above (note 24), p. 118, evaluated 
conformity with WCAG 1.0/2.0, as WCAG 2.1 was only released in 2018 and reflected in EN 301 549 
V2.1.2 (2018-08). 

51  Berlin Declaration on Digital Society and Value-based Digital Government (2020). 
52  Report on the monitoring of the Berlin Declaration (PDF), Directorate General for Informatics, (2022), 

pp. 92-94. 

53  MS policy action 2.2 “Ensure that the digital transformation is inclusive of and accessible for persons 
with disabilities and elderly persons and increase our efforts to make public services and information 
fully  digitally  accessible  in  accordance  with  the  Web  Accessibility  Directive  and  the  European 
Accessibility Act” and KPI 9 refer. 

54  Communication from the Commission to the European Parliament, the Council, the European Economic 
and  Social  Committee  and  the  Committee  of  the  Regions  –  European  Interoperability  Framework  – 
Implementation  Strategy,  COM(2017)  134  final,  23.3.2017.  See  also  The  European  Interoperability 
Framework. 
Inclusion and accessibility (principle 7) measured through  KPI-28 on compliance  with the European 
accessibility standards – score improved from 2.25 (2019) to 2.61 (2021), EIF Monitoring dashboard. 
See  also  ‘State-of-play  report  on  digital  public  administration  and  interoperability,  2021  (PDF), 
Directorate General for Informatics, pp. 23, 55. According to the report ‘20 different countries created 
or upgraded 29 public services in the period from 2020 to 2021’, p. 13. 

55 

56  Eurostat (2022), Code: ISOC_CI_IFP_IU, Last internet use: in the last 12 months, all individuals. See 

also an overview of Digital economy and society indicators. 

12 

 
                                                 
online with public authorities in the EU had risen to 63% (for 25 to 64 years old) and 
38% (for 65 to 74 years old) 57 from 48% and 17% respectively in 2012. 

This trend is likely to continue, as explained further in sections 4.3 and 5.1.5 on relevance. 

4  EVALUATION FINDINGS (ANALYTICAL PART) 

This  section  discusses  the  analytical  findings  of  the  evaluation,  encompassing  all  five 
evaluation criteria (effectiveness, efficiency, coherence, relevance and EU added value). 

As  described  in  section  1.2,  the  analysis  draws  from  a  supporting  study  and  extensive 
consultation  activities, including:  (i) a public consultation using both  a standard and an 
easy-to-read questionnaire; (ii) targeted surveys for different stakeholder groups; (iii) in-
depth interviews conducted with relevant stakeholders; and (iv) analysis of Member States’ 
reports on the outcome of the monitoring and the use of the enforcement procedures (see 
Annexes II, V, VI). 

4.1  To what extent was the intervention successful and why? 

The  assessment  of  the  intervention’s  success  level  is  based  on  the  first  three  points  of 
comparison described in section 2.2. 

4.1.1  Compliance with the Directive and conformity with accessibility requirements 

are not identical 

At the outset, it should be mentioned that conformance to the accessibility requirements 
does  not  equate  to  compliance  with  the  WAD.  This  distinction  was  sometimes 
overlooked, as observed throughout the evaluation, in national monitoring activities and 
reports, and in the wording used by monitoring tools. 

For a website or mobile application to be  compliant with the WAD, it must fulfil two 
criteria.  First,  it  must  conform  to  the  accessibility  requirements  (perceivable,  operable, 
understandable and robust), for which the Harmonised European Standard EN 301 549 58 
gives a presumption of conformity 59. 

Secondly, a website or mobile application must have a detailed, comprehensive and clear 
accessibility statement (AS). The AS, based on the model provided by the Commission 
implementing  decision,  must  include  information  on  (i)  compliance  status,  (ii)  any 
inaccessible elements  and (iii) accessible alternatives, (iv) how to  report or request  any 
inaccessible  content,  and  (v)  a  link  to  the  complaint  mechanism  if  the  response  is 
inadequate. Thus, a website or mobile application without an AS is not compliant with the 
WAD, even if it is technically fully accessible. 

57  Eurostat  (2022),  Code:  ISOC_CIEGI_AC,  Internet  use:  interaction  with  public  authorities,  last  12 

months; the share for all individuals was 58% (up from 44% in 2012). 

58  Cited above (note 37), see also above, p. 9, regarding implementing decisions on harmonised standard. 
59  Websites  and  mobile  applications  can  be  exempted  from  accessibility  requirements  only  if  those 
requirements impose a disproportionate burden on the public sector body. Disproportionate burden must 
be assessed before being applied, and must be explained in the relevant accessibility statement. 

13 

 
                                                 
Conversely, a website or mobile application that, for example, cannot be navigated with a 
screen reader, does not conform to accessibility requirements, as defined in the standard. 
However, it may still be compliant with the WAD if the inaccessible element is exempt, 
for example due to legitimate disproportionate burden, and this is clearly explained in the 
AS, with information on accessible alternatives 60. 

The box below presents the findings from the monitoring reports related to the accessibility 
statements. 

Box 1. Analysis of findings from the monitoring reports on the accessibility statements 
Most of the Member States (20 out of 25 that submitted reports) checked the availability 
of  accessibility  statements,  but  five  MS  did  not.  The  evidence  submitted  shows  that 
accessibility  statements  (AS)  are  either  missing  or  were  not  checked  on  websites  or 
mobile applications at different rates across the MS. It is not an obligation to monitor AS 
availability per se, but the web page with the accessibility statement should be part of 
the in-depth monitoring according to the Implementing Decision (EU) 2018/1524 61. As 
became apparent during the interviews, a lack of AS may be correlated with lower level 
of feedback from end-users to PSBs. These statements offer information on how to report 
accessibility issues, and to pursue accessibility requests, ensuring accessibility statement 
as standard – as required under the WAD – is essential. For the next monitoring period, 
this issue of encouraging feedback by virtue of AS should become a focus for the 
monitoring agencies, as publishing an AS is an important obligation of the WAD. 

Only five MS compared the content of the AS with monitoring results to validate the 
AS  claims. Although this  comparison  exercise is not  mandatory, it would not  require 
much extra effort, and would show that AS matter. Such a comparison would assess the 
quality of the AS, and ensure that the PSBs (or their suppliers) are aware of any mismatch 
between  the  AS,  and  what  it  presents  as  accessible  and  inaccessible  content  and 
alternatives  on  the  website  or  mobile  application.  If  the  monitoring  agencies  find 
recurring differences between the AS and the monitoring results, this may be a reason to 
focus monitoring and/or training activities on specific functions or areas. From the end-
user’s perspective, it is, of course, important that the AS is complete, correct and reliable. 

Inaccessible  content  flagged  by  monitoring  tools  may  be  legitimately  exempt  (e.g. 
disproportionate burden) or not in scope of the WAD 62. The monitoring tools, especially 
automated  ones,  mostly  assess  the  accessibility  requirements  set  out  in  the  WCAG 
(sometimes also parts of HEN), but do not consider the possible legitimate exceptions and 
alternatives to inaccessible content. In this context, the lack of an explicit obligation on the 

60  Other exceptions are listed in Article 1(4) of the Directive, for example, office file formats published 
before  2018,  audio  or  video published  before  2020,  live  audio  or  video, online  mapping,  third-party 
content, reproductions of heritage items, extranet and intranet content published before 2019, archives. 
61  Cited  above  (note  30),  Annex  I,  point  3(2)(c)  states  that  ‘for  the  in-depth  monitoring  method  the 
following pages and documents, if existing, shall be monitored: (c) the pages containing the accessibility 
statement or policy and the pages containing the feedback mechanism’. 

62  See section 4.1.1 for a discussion on compliance with WAD. 

14 

 
                                                 
MS  to  consider  the  accessibility  statement  when  monitoring  a  website  or  mobile 
application is not ideal. 

The research literature shows that although there was already a slight improvement in the 
accessibility of public sector bodies’ websites and mobile applications (objective 1) in 
the  early  years  following  the  transposition  of  the  WAD,  there  is  still  a  notable  lack  of 
accessible  websites  across  the  EU  in  both  public  and  private  sector 63.  However,  the 
research in the existing literature is of limited relevance, as it is mostly based on automated 
accessibility checking, so excluding accessibility statements, and covers typically national 
level  PSBs.  As  noted  earlier,  the  2007  benchmarking  study  Measuring  e-Accessibility 
(MeAC) 64  found  only  5%  of  government  websites  passed  automatic  and  manual 
accessibility  checks.  In  2011, the follow-up study, using a different  method, found that 
34% of the EU ‘web content’ (public and private) was accessible. However, given the rapid 
digitalisation of the public sector since then and evolution of automatic testing 65, these 
figures also are no longer particularly relevant. 

Although  the  2022  eGovernment  Benchmark  report  found  that  only  16%  of  the 
eGovernment  websites  passed  all  of  the  selected  web  accessibility  criteria,  the 
conformance  rate  per  criteria  is  more  encouraging:  out  of  all  checks  76%  passed,  17% 
failed, 7% were inapplicable 66. Rather than giving an accessibility score per website, the 
modern automatic accessibility testing tools count the number of criteria that pass, fail or 
are inapplicable, and thus allow testers to identify conformity per criteria. For example, 
the eGovernment Benchmark report suggests that some accessibility criteria are better or 
more easily implemented than others 67. 

This is in line with the findings of the 2022 WebAIM Million 68 report. After examining 
the accessibility of the top 1,000,000 home pages worldwide, the report noted that while 
accessibility errors have decreased, no website is completely error-free. 

4.1.2  Progress and compliance – surprisingly hard to demonstrate 
More  than  half  of  respondents  (57%)  to  the  easy  to  read  survey 69  found  government 
websites easier to use in the last three years, while in the standard survey 41% reported 
experiencing accessibility issues never or less often than three years ago. About a third of 
respondents in both surveys reported no changes in their experiences. 

63  Barricelli et al., 2018; Sala, Arrue, Perez & Valencia, 2019; Pribeanu, 2019; Sabev, Georgieva-Tsaneva 
& Bogdanova, 2020. The prevailing assessment method used in the literature was the use of automated 
checking tools, which has its limitations. 

64  Cited above (notes 10, 23-25), see sections 1.1 and 2.1 of this report. 
65  For example, 2010 MeAC 2 study, cited  above (note 24), p. 118, evaluated conformity  with WCAG 
1.0/2.0, as WCAG 2.1 was yet to be released in 2018 and reflected in EN 301 549 V2.1.2 (2018-08). 
66  See above section 3.2 in relation to eGovernment Benchmark (2022) Non-scored indicators (.xlsx) and 

67 

notes 48, 49. 
Ibid., WCAG 1.4.3 colour contrast criteria passed 40,9%, while WCAG 4.1.2 name, role, value criteria 
passed 99,5%. 

68  The WebAIM Million. The 2022 report on the accessibility of the top 1 000 000 home pages. 
69  Used by 80% of total respondents self-identifying as persons with disability. 

15 

 
                                                 
Do you think government websites 
have become easier to use in the last 
3 years? (Easy to read survey, n=656)

Compared with the situation 3 years ago, 
how often are you experiencing accessibility 
issues? (Standard survey, n=198)

7%

9%

8%

28%

48%

Much easier
Little bit easier
No difference
More difficult
Don’t know

Source: Public consultation 

4%

15%

11%

37%

33%

Never
Less often
As often
More often
Not sure

Looking  closer  at  content  types  or  services  offered  by  public  sector  bodies,  72%  of 
respondents to the standard consultation observed slight or significant improvements when 
accessing  online  information  (respectively  115  and  27  out  of  198)  over  the  last  three 
years,  as  shown  in  the  chart  below.  This  opinion  was  shared  by  52%  of  people  with 
disabilities,  older  people,  or  people  taking  care  of  people  with  disabilities  in  targeted 
surveys. 

Compared with the situation 3 years ago, how did your experience change
when accessing the following public sector content or services?

0%

Online
information
Mobile
applications
Interactive
online services

Online
documents

27

17

24

20

25%

64

62

50%

75%

100%

115

35

12

7

2

59

46

7

5

93

45

22

9

70

26

15

5

5

Improved significantly

Improved slightly No change Not sure Worsened slightly Worsened significantly

Source: Public consultation (standard survey, n=198; not all respondents were asked this question) 

As  to  the  role  of  WAD,  50%  organisations  representing  people  with  disabilities,  older 
people or consumers agreed that the WAD had contributed to greater accessibility of online 
public services. Yet about the same number of respondents to the public consultation (57%) 
do not think that ‘people with disabilities can use online information just as well as others 
can’. This apparent contradiction can be explained – PSB web accessibility has increased 
from a low starting point, but persons with disabilities cannot yet access all online public 
services on equal terms with everyone else at a time when public services are progressively 
more online. The majority of respondents (79%) affirmed that their use of online public 
services has increased in the last three years, and 65% expected more improvements in the 
future. 

The main accessibility issues identified when using public services online were navigation 
and accessing forms, followed by video content (though live time-based media is not in 
scope of the WAD) 70. 

70  See more in Annex V Synopsis report, section 3.1.1. 

16 

 
                                                 
From a geographical perspective, there is slight difference in experience, as measured 
by  the  surveys.  More  respondents  in  northern 71  and  southern 72  European  countries 
reported experiencing issues less often than before or never (50% in North and 54% in 
South) compared to central 73 and eastern 74 European countries (33% in Central and 37% 
in East). 

One  difficulty  for  MS  in  demonstrating  progress  in  making  websites  and  mobile 
applications “more accessible”, is the compliance status as defined in the Implementing 
Decision (EU) 2018/1523:  fully, partially or non-compliant. The Harmonised European 
Standard EN 301 549 (HEN) sets out the means to determine a presumption of conformity 
with the accessibility requirements of the WAD: full compliance is achieved by meeting 
all  the  applicable  requirements.  Testing  a  website  or  mobile  application  item  can  thus 
result  in:  pass,  fail,  or  not  applicable.  In  practice  this  means  that  if  any  one  item  on  a 
relevant  page  or  screen  fails  the  test,  the  website  or  mobile  application  can  only  be 
considered partially compliant. 

Inevitably, given the wide and varied nature of these requirements, and the many different 
actors responsible in each MS, most of the websites and mobile applications sampled for 
monitoring fall in the category of partially compliant. However, partial compliance covers 
a wide range of situations, from nearly perfect accessibility to only just non-accessible, 
with no consideration of the impact of errors on the end-user. Failing some criteria (for 
example, no alternative text for images) would render a page technically non-compliant 
but usable, whereas failing other criteria (for example, a keyboard trap that prevents a user 
with visual impairment or limited mobility from completing a task online) would create a 
serious  usability  issue,  in  effect  blocking  access  Unfortunately,  there  is  no  agreed 
methodology  amongst  accessibility  experts  at  present,  neither  at  EU  level  nor 
internationally, on how to quantify partial accessibility as a percentage 75, or to weight the 
accessibility requirements by impact. 

It should also be noted that the HEN has been updated once since 2020, and will continue 
to evolve. At present the HEN itself acknowledges areas for further work: its Annex D 
states ‘It is evident that people with limited cognitive, language and learning abilities have 
diverse accessibility needs and preferences and that there is a need for further guidelines 
and standards. Research in this area is ongoing’ 76. 

Results from  the national  monitoring reports, however, show there is  ample room for 
improvement in compliance as most of the websites and mobile applications monitored 
did  not  respect  all  of  the  requirements.  Nonetheless,  purely  binary  outcomes  of 

71  Denmark, Ireland, Finland, Sweden. 
72  Greece, Spain, Italy, Cyprus, Malta, Portugal. 
73  Belgium, Germany, France, Luxembourg, the Netherlands, Austria. 
74  Bulgaria, Czechia, Estonia, Croatia, Latvia, Lithuania, Hungary, Poland, Romania, Slovenia, Slovakia. 
75  As  examples,  the  Netherlands  have  implemented  a  system  of  indicating  compliance  status  in  the 
accessibility  statements  (A:  completely,  B:  partially,  C:  first  measures  taken,  etc).  Portugal  offers 
usability and accessibility seal on websites (gold, silver, bronze) to showcase examples of compliance 
with best practices. 

76  EN 301 549 V3.2.1 (2021-03) (PDF), cited above (note 37), p. 181. 

17 

 
                                                 
monitoring  of  websites  and  mobiles  applications  (pass/fail  of  conformity  requirements) 
should be taken with caution, as they do not reflect the full measure of compliance with 
the WAD. 

According  to  the  national  monitoring  reports,  there  were  only  a  handful  of  fully 
compliant websites or mobile applications, but no completely non-compliant website or 
mobile application. This does not contradict the findings of the public consultation, as there 
is  ample  space  for  relevant  improvements  within  the  “partially  compliant”  category, 
without reaching full compliance. 

The interpretation of such results should consider some factors. On the one hand, (i) some 
MS  did  not  always  follow  the  Implementing  Decision  (EU) 2018/1524 77  and  defined 
compliance according to their own methodology, which sometimes was stricter for non-
compliant websites, but also sometimes more lenient regarding full compliance. On the 
other,  (ii)  when  the  compliance  assessment  followed  the  Implementing  Decision,  the 
reporting  of  “pass”  or  “fail”  results  was  inconsistent  across  MS,  either  because  not 
applicable requirements were handled differently or because the MS also considered (and 
weighted)  the  level  of  breach  in  different  ways.  Therefore,  it  is  difficult  to  draw  a 
meaningful comparison between the MS from the evidence provided in the monitoring. In 
the  second  monitoring  period,  however,  this  will  be  addressed  by  requiring  a  recurring 
sample  of  at  least  10  % of  websites  and  mobile applications  monitored  in  the  previous 
monitoring period and at least 50 % that were not monitored in the previous one 78. 

4.1.3  Cross-border monitoring tools are in use, although commercial tools are not 

perfect 

As  objective  2  of  the  WAD  was  to  harmonise  the  single  market,  the  fact  that  some 
European tools used in the monitoring exercise are being used cross-border is very 
encouraging. 

The  industry  players  consulted  identified  the  most  relevant  drivers  of  success  in  the 
application of the WAD at market level as: (i) the spread of knowledge about accessibility 
solutions across businesses; and (ii) the increase in demand for accessibility tools. 

Common technical requirements, such as the Harmonised European Standard EN 301 549, 
were  a  first  essential  step  in  removing  barriers  in  the  internal  market 79  and  have  been 
useful  for  making  public  sector  websites  accessible.  According  to  industry  players, 
common requirements produced a positive impact on their business, driving  demand of 
products and services in the national markets and an increase in opportunities to penetrate 
foreign markets 80. 

77 

Implementing Decision (EU) 2018/1524 establishing a monitoring methodology and the arrangements 
for reporting by Member States, cited above (note 30). 
Ibid., Annex I, point 2.4. 

78 
79  Rajšp, A., Kous, K., Kuhar, S., Sumak, B., & Sorgo, A. (2019). Preliminary review of jobs, skills and 
competencies  for  implementation  of  Digital  Accessibility.  Proceedings  of  the  Central  European 
Conference on Information and Intelligent Systems. 

80  Study, cited above (note 17), pp. 77, 81. 

18 

 
                                                 
Several  MS  developed  tools  themselves,  some  of  which  have  been  also  used  in  other 
countries:  Belgium  (Scan.accessibility.belgium.be),  Finland  (Salvia,  using  QualWeb), 
Luxembourg (simplA11yMonit, simplA11yGenReport, simplA11yPDFCrawler), Portugal 
(AccessMonitor, also used in Slovakia), and Spain (OAW Tracker, also used in Sweden). 

Coupled with this, many MS used commercial tools, with at least 4 such tools used by at 
least 3 MS, and 2 by at least 6. More than 10 tools that help manual testing were used. For 
mobile applications, MS altogether used at least 8 automated tools, with a further 8 tools 
helping manual inspection. 

All MS used (to a greater or lesser extent) automated tools in the simplified monitoring, 
with the exception of Croatia. However, looking at the least tested criteria, the list clearly 
shows that the vast majority of the automated testing tools used have been created to fit 
the US market, where WCAG 2.0 (published in 2008) is still relevant. Conversely, the EU 
legislation uses WCAG 2.1 (published in 2018) as the reference for presumed conformity 
in Harmonised European Standard EN 301 549. These automated testing tools can only 
check for 20-30% of success  criteria 81, while some leading accessibility software tools 
claim 57% of digital accessibility issues 82. 

For  the  in-depth  monitoring,  the  list  of  least  tested  success  criteria  may  indicate  that 
multimedia  content  is  either  omitted  from  the  page  samples  or  does  not  feature  in  the 
websites  and  applications  selected  for  monitoring 83.  This  would  require  further 
investigation, as multimedia content  is  very important  for large user  groups. This  issue 
may serve as a possible direction for further investments in the market and/ or research 
funding,  given the expansion of accessibility requirements that will  come into effect  in 
2025 under the European Accessibility Act (EAA) 84. 

Regarding  harmonisation  of  the  internal  market  for  web  accessibility  products  and 
services,  overcoming  regulatory  fragmentation  (second  WAD  objective),  the  evidence 
collected  indicates  that  the  WAD  helped  to  improve  the  internal  market  for  web 
accessibility services 85. Interviews with stakeholders from the private sector 86 indicated 
that the WAD had lowered entry barriers to other European markets for software providers, 
facilitating cross-border sales. The industry players consulted agreed that both the demand 
and supply of web accessibility products and services had increased after the introduction 
of the WAD. 

81  Automated  Accessibility  Testing  Tools:  How  Much  Do  Scans  Catch?  (Essentialaccessibility.com, 

2020). 

82  Automated Testing Identifies 57 Percent of Digital Accessibility Issues (Deque.com, 2021). 
83  Study, cited above (note 17), p. 49. 
84  Directive  (EU)  2019/882  of  the  European  Parliament  and  of  the  Council  of  17  April  2019  on  the 

85 

accessibility requirements for products and services (OJ L 151, 7.6.2019, p. 70-115). 
In the public consultation, 30% of the PSB respondents (19 out of 62) agreed or strongly agreed that the 
Directive had increased the cross-border provision of products and services for web accessibility. Only 
10% disagreed or strongly disagreed with the statement, and most of the remaining respondents (60%) 
neither agreed nor disagreed. 

86  See Annex V Synopsis report, section 4 List of interviewees. 

19 

 
                                                 
The  WAD  facilitated  an  increase  in  the  number  of  industry  actors  specialising  in 
accessibility tools in the EU market 87. This growth of the sector is likely to further improve 
the internal market in the coming years. The ongoing implementation of the WAD and the 
upcoming  EAA 88  requirements  will  continue  to  increase  demand,  while  the  in-depth 
interviews with the national monitoring bodies indicated that the supply of accessibility 
tools is still insufficient, both in quantity, and quality (not fully covering the requirements 
set in the HEN). 

4.1.4  Member States reports are difficult to compare 

The  first  monitoring  reports  provide  insights  on  the  different  use  and  interpretation  of 
monitoring methodologies by MS. 

The WAD and Implementing Decisions deliberately do not impose a particular monitoring 
methodology, to allow innovation and appropriate choices at national level. MS in practice 
both monitored and reported on different timeframes, to different versions of the technical 
standard, and some did not monitor the minimum sample size required. While reserving 
the right to pursue actions of non-compliance, it should be noted that this was the first such 
monitoring  exercise,  and  came  relatively  soon  after  the  practical  deadlines  for 
implementation (15 months for websites, and six months for mobile applications). In this 
context  a  range  of  interpretation  could  be  expected.  The  most  significant  of  these 
differences concerns the standard used as the baseline for presumption of conformity, with 
the complexity of monitoring exacerbated by the difficulties encountered in interpreting 
the  guidelines  of  the  WAD.  Work  will  continue  to  ensure  this  is  fully  understood, 
especially  as  this  standard  will  be  further  revised  to  reflect  the  requirements  of  the 
European Accessibility Act. 

Although some MS monitored a smaller sample size 89, all MS who reported carried out 
both simplified (all but one using automated tools) and in-depth monitoring. Only 2 MS 
have not reported to the Commission 90. 

Some MS committed to test more than is required, according to analysis of findings from 
the Member States going beyond the scope of the WAD in the monitoring exercise 91. For 
example,  Germany  also  tested  three  requirements,  on  sign  language,  easy-to-read 
language,  and  the  ‘UE’  standard  for  PDF,  not  covered  by  the  HEN.  Both  Portugal  and 
Finland checked WCAG 2.1 AAA 92 criteria in their monitoring exercise, although not an 

87  65% of industry respondents to the targeted survey (34 out of 52) reported that the provisions included 
in  the  WAD  had  fostered  the  development  of  new  accessibility  tools  and  products,  at  least  to  some 
extent, according to Study, cited above (note 17), p. 55. 

88  Entering into practical effect in 2025 on a range of digital hardware and services. 
89  See Annex VI of this report for details on sample sizes, and information on monitoring and reporting 

exercise. 

90  See section 3.2, and national monitoring reports, cited above (notes 42, 43). 
91  Study, cited above (note 17), p. 46. 
92  According to the WCAG 2.1 guidelines, AAA success criteria relate to the highest level of accessibility. 

The HEN refers to A and AA level success criteria from the WCAG 2.1 guidelines. 

20 

 
                                                 
obligation under the WAD. The results were not shared in the monitoring report. On the 
procedural side, in Greece it is obligatory to publish self-assessment reports. 

From analysis of the reports, the following points emerge: 

 

It also emerged in the interviews that many MS are unaware of the accessibility criteria 
set out in the Harmonised European Standard EN 301 549 (HEN), and so they only 
focused on WCAG 2.1 criteria for monitoring. 16 MS monitored only WCAG success 
criteria, omitting the clauses 5, 6, 7 and 12 of the EN 301 549 (HEN). Only four MS 
monitored all the relevant requirements of the HEN. This is a significant problem, as 
while Annex A of the HEN sets the criteria for the presumption of compliance, some 
MS may use different baselines to judge conformity. Several MS mentioned that the 
requirements of WCAG 2.1 and HEN for mobile applications were hard to interpret, 
making it difficult to assess compliance. 

  MS  conducted  the  monitoring  in  different  periods  within  the  set  timeframe. 
Implementing  Decision  (EU)  2018/1524  asked  for  the  first  monitoring  report  to 
perform the monitoring ‘within’ the period between 1 January 2020 and 22 December 
2021 for websites, and 23 June 2021 and 22 December 2021 for mobile applications. 
In five countries 93, the monitoring exercise was conducted only in 2021. While the 
legal text is open to different interpretations, it should be noted that all MS conducted 
monitoring “within” the period defined in the WAD. 

  The  scope  of  the  monitoring  exercise  varied.  Under  the  WAD  and  Implementing 
Decision  (EU)  2018/1524,  MS  should  select  a  sample  of  websites  and  mobile 
applications 94  that  (i)  is  as  representative  as  possible  (considering  the  different 
administrative  levels,  the  type  of  services  provided  by  the  public  sector,  and  the 
geographical distribution); (ii) is proportionate to the population of the country, and 
(iii)  takes  into  account  the  views  of  stakeholders,  in  particular  organisations 
representing persons with disabilities and older people 95. 

  The differing interpretations of standards, accessibility conformance rules and testing 
methods can lead to inconsistent or even conflicting results. This makes it difficult or 
even impossible to reliably compare monitoring results produced by different tools. To 
address  uniform  interpretation,  for  example,  of  the  WCAG  and  provide  consistent 
results,  the  W3C  has  produced  a  recommendation  –  Accessibility  Conformance 
Testing (ACT) rules, which are constantly evolving 96. 

93  Estonia, Finland, Greece, the Netherlands and Spain. 
94  The minimum sample size is defined in the Decision (EU) 2018/1524, reproduced in Annex IX of this 

report. 

95  According  to  Implementing  Decision  (EU)  2018/1524  and  in  line  with  recommendations  of  UN 
Committee General comment No. 7 (2018) on the participation of persons with disabilities, including 
children  with  disabilities,  through  their  representative  organizations,  in  the  implementation  and 
monitoring of the Convention. 

96  W3C, Accessibility Conformance Testing (ACT). For example, according to a blog post ‘Harmonized 
Accessibility  Testing’  (30.07.2019)  by  Shadi  Abou-Zahra,  ‘a  method  may  mistakenly  fail  text 
alternatives that are too long when that is not actually defined by WCAG. Another method may correctly 
mark that as a warning because it is advisory good practice.’ 

21 

 
                                                 
It  should be noted that  Bulgaria  reported  that iOS-based applications  are not  used by a 
significant number of people with disabilities, because no adequate speech synthesiser is 
currently available in Bulgarian for that platform. 

During the interviews, several MS stated that for the next monitoring period they would 
welcome clearer guidance on how to report results of the monitoring activities and a 
standard  template  at  the  EU  level.  This  could  substantially  improve  the  outcomes’ 
reporting  comparability  and  results’  analysis.  Another  concern  that  emerged  in  the 
interviews is the high burden that the current monitoring process imposes on monitoring 
bodies, which often do not have enough expert staff to carry out these activities. 

When  asked  about  their  reasons  for  not  monitoring  all  requirements,  some  monitoring 
agencies stated they were not to be aware of the current requirements, and others said that 
the  external  experts  hired  to  carry  out  the  audits  did  not  have  the  skills  to  do  so.  One 
possible reason for the difference in scope may be that the automated tools used to support 
the testing cater to WCAG and do not contain all the HEN requirements. The Commission 
had  flagged  this  as  a  topic  for  discussion  in  the  WADEX  meetings  and  has  published 
detailed information about this topic online 97. 

Some  MS  reported  that  their  current  list  of  digital  public  services  is  incomplete  and/or 
unreliable as mapping and counting all public sector websites and mobile applications 
is  impractical,  if  not  impossible,  particularly  given  the  many  different  administrations 
involved at national, regional and local level. This limits the MS’ capacity to consider the 
entirety of websites and mobile applications in scope of the legislation 98. 

4.1.5  Member States have committed to training and awareness-raising 

Article  7(4)  of  the  WAD  states  that  MS  should  facilitate  training  programmes  and 
awareness-raising activities on the WAD and its main provisions, asking for the relevant 
information in a dedicated section of the monitoring reports. All MS which delivered the 
report  provided  this  information,  with  the  exception  of  Ireland  and  Portugal.  The 
monitoring report from Lithuania only shared the country’s plans for future training in this 
area. The remaining 22 countries stated that they had implemented the provisions of Article 
7(4) mainly through three channels: (i) setting up dedicated events and workshops at the 
national  and  sub-national  level99;  (ii)  organising  training  programmes  for  different 
stakeholder categories 100; and (iii) drafting technical guidelines and materials 101. 

97  European Commission (2022),  Web Accessibility Directive – Standards and harmonisation (HTML): 

EU legislation, technical standards and W3C international best practice on web accessibility. 

98  The legislator was aware of this challenge, therefore website and mobile application sample sizes are 
based on population of a Member State; Implementing Decision (EU) 2018/1524, cited above (note 30), 
Annex I, point 2. 

99  Confirmed by 11 MS: Czechia, Denmark, Germany, Greece, Spain, Italy, Latvia, Malta, Netherlands, 

Austria, Romania, and Sweden. 

100  Confirmed  by  12  MS:  Germany,  Greece,  Spain,  Croatia,  Latvia,  Lithuania,  Luxembourg,  Hungary, 

Poland, Slovakia, Finland, Sweden. 

101  Confirmed  by  12  MS:  Denmark,  Germany,  Estonia,  Greece,  Croatia,  Italy,  Luxembourg,  Hungary, 

Malta, Slovakia, Finland, Sweden. 

22 

 
                                                 
The analysis of the literature reveals several other factors influencing the adoption and 
implementation of web accessibility standards in PSBs 102. These include: (i) assigning 
clear responsibilities; while stakeholders claim to be in full control over their part of the 
process, once confronted with non-conformance of the end result, they pointed to others 
for  responsibility;  (ii)  management  decisions;  in  municipalities  where  managerial 
commitment  to  the  website  and  online  services  is  high,  stakeholders  perceive  fewer 
problems;  (iii)  perceived  benefits;  if  management  does  not  perceive  sufficient  benefits 
from  better  accessibility,  priorities  move  away  from  accessibility;  and  (iv)  legislation; 
although  it  does  not  directly  guarantee  the  proper  implementation  of  accessibility 
standards, it has a positive influence on the process. 

Opinions differed over the involvement of people with disabilities in the implementation 
of the WAD obligations. Several engagement activities and procedures were launched 
by national  authorities to  consult  relevant  stakeholders about  the implementation  of the 
WAD.  Some  MS  even  set  up  permanent  committees  with  representatives  from  DPOs, 
industry, academia and public authorities. Furthermore, many of them involved end-user 
groups  and  relevant  organisations  in  the  process  of  selecting  websites  and  mobile 
applications  to  be  tested  for  the  first  monitoring  activities,  an  obligation  under  the 
Implementing  Decision 103.  The  involvement  of  people  with  disabilities  ranged  from 
consulting on strategy and advising on the mechanisms implemented by the WAD to being 
involved in user testing, providing the views and knowledge of the group benefitting most 
from  the  proper  implementation  of  measures  and  monitoring,  following  the  principle 
“nothing about us without us”. 

The facilitation of training and awareness raising in MS can be considered a success 
factor in the implementation of the WAD. Since the beginning of the monitoring period, a 
variety of training initiatives have been promoted by most of the MS. Several countries 
introduced  training  on  accessibility  topics,  mostly  targeting  employees  in  PSBs.  For 
instance,  Czechia  is  launching  a  new  course  for  PSBs  on  web  accessibility,  which  will 
provide a certificate to participants who complete the course. Some MS (e.g. Slovakia and 
Spain)  reported  that  new  training  courses  were  devised  based  on  the  results  of  the 
monitoring activities. 

However,  a  2019  survey  conducted  by  the  European  Disability  Forum 104  showed 
significant  dissatisfaction  with  the  level  of  engagement  from  relevant  stakeholders. 
Respondents criticised the lack of consultation by MS before the adoption of the national 

102  Velleman,  E.  M.,  Nahuis,  I.,  &  VanDerGeest,  T.  (2017).  Factors  explaining  adoption  and 
implementation  processes  for  web  accessibility  standards  within  eGovernment  systems  and 
organizations. Universal Access in the Information Society, pp. 173-190. 

103  Implementing  Decision  (EU)  2018/1524,  cited  above  (note  30),  Annex  I,  points  2.2.4  and  2.3.5: 
‘Member States shall consult national stakeholders, in particular organisations representing persons with 
disabilities, on the composition of the sample of the websites to be monitored, and give due consideration 
to the stakeholders’ opinion regarding specific websites [and mobile applications] to be monitored’. 
104  European  Disability  Forum  (2019),  Survey  on  Web  Accessibility  Directive  transposition  and 
implementation, presented at the 6th WADEX meeting, Brussels, 28 November 2019, (download Survey 
PPTX or see meeting documents on WADEX page). 

23 

 
                                                 
laws and denounced the poor guidance and lack of support for stakeholder organisations. 
In addition, the results of the WAD review survey targeting DPOs show that 60% of DPOs 
(29 out of 48) reported little or no involvement in either the implementation of the WAD 
or in the selection of websites and mobile applications to be monitored. 

4.1.6  Feedback and enforcement procedures under-reported and under-used 
The Implementing Decision (EU) 2018/1524 105 requires MS to describe and detail their 
use  of  the  enforcement  procedure  set  up  at  the  national  level,  to  be  used  when  initial 
feedback  has  failed.  MS  may  also  (but  are  not  obliged  to)  give  data  on  the  feedback 
received  by  the  PSBs  through  the  first-level  feedback  mechanism  laid  down  in  Article 
7(1)(b) of the WAD. On the enforcement procedure, 18 monitoring reports describe the 
procedure in place, while only two countries (Germany and Finland) reported quantitative 
information on the complaints they had received. The enforcement procedures of various 
MS include the possibility of using administrative fines for PSBs that do not comply with 
the WAD 106. 

In  general,  countries  did  not  give  any  quantitative  information  about  the  feedback 
mechanism, apart from Finland, which shared results from a survey it had conducted on 
the mechanism. On the issue of feedback, several countries filled in the section devoted to 
feedback mechanisms  with information  that seemed unrelated. Some countries  reported 
only on complaints or on feedback given to the monitoring body by PSBs or experts. Such 
shortcomings in the reports suggest that the optional reporting requirements on feedback 
mechanism were either unclear or left room for interpretation to the MS. 

In addition,  from the public consultation  it appears that both  these procedures are little 
used. 

While the lack of complaints 107 could be seen as very positive (e.g. no accessibility issues 
found),  among  the  reasons  reported  for  not  complaining  were  the  inaccessibility  of 
feedback forms due to use of the online ‘captcha’ tools and a perception that a complaint 
would  not  change  anything.  Some  individuals  who  had  complained  reported  negative 
experiences, as enforcement bodies did not always reply satisfactorily to their complaint, 
nor seem to be aware of the mechanism. This lack of complaints, however, should not be 
fully interpreted as an indication of a malfunctioning instrument. It may also be the case 
that the answers given or actions taken in the feedback mechanism, are satisfactory in not 
giving rise to further formal complaints. 

The awareness of these mechanisms should continue to be monitored, as further research 
on the needs and barriers for end-user possible feedback and complaint may be needed, 
given the current low level of use. 

105  Commission Implementing Decision (EU) 2018/1524, cited above (note 30). 
106  France, Croatia, Malta, Poland, Romania, Slovenia and Slovakia; fines range from EUR 200 to EUR 

25 000, depending on the Member State. 

107  88% of respondents to the easy-to-read survey have never complained and 61% did not know how to 

complaint (Annex V, section 3.1.3 Feedback mechanism and complaints). 

24 

 
                                                 
The  box  below  presents  the  findings  from  the  monitoring  reports  on  the  enforcement 
procedures. 

Box 2. Analysis of findings of the monitoring reports on the enforcement procedures 
The  enforcement  procedures  of  most  of  the  MS  were  described  in  the  monitoring 
reports. These vary country by country depending on their constitutional structures. For 
instance, in  Austria, enforcement is  performed at  the federal  level,  while in  Germany 
each state has its own enforcement body, where the Conciliation Body acts at the federal 
level.  In  most  of  the  other  MS,  both  the  examination  of  the  complaints  and  the 
enforcement is carried out at the central level by the agency or the ministerial department 
which works on digital accessibility. 

Despite some differences, there are some similarities in the overall procedure, as most 
MS  follow  a  three-staged  approach  to  enforcement.  Firstly,  the  non-compliance 
complaints are collected and examined by the responsible body, which decides whether 
to start a procedure. Secondly, the body may issue binding instructions if a breach of the 
accessibility requirements is found, specifying a deadline for compliance. The deadlines 
are  usually  between  1  and  3  months.  Thirdly,  if  the  organisation  fails  or  refuses  to 
comply, the responsible body will proceed with enforcing the regulations. 

While in some countries (e.g. Germany and Luxembourg) there are mediation procedures 
in place, in others (e.g. Estonia), disciplinary proceedings can be conducted against an 
official if there are infringements. In most cases, the enforcement body has the power to 
decide on the infringement and issue an order to the PSB. However, in some other cases, 
when complaints are handled through mediation, the complainer may need to turn to the 
courts if the parties cannot reach an agreement. Finally, in a few cases (mainly in some 
states in Germany), the right to complain is limited to people with disabilities, people 
whose rights were infringed, or their representative organisations, limiting the personal 
scope of the complaint mechanism laid down in the WAD. Some MS made it possible 
to impose administrative fines 108, but reports did not provide data about the PSBs fined 
or about the intention of using fines in practice. 

To conclude, enforcement mechanisms are in place, and while differences still exist in 
terms  of  detail,  the  MS  appear  to  be  taking  this  seriously  within  the  framework  of 
national legislation. 

4.1.7  Accessibility experts are needed, as job opportunities exist 

MS highlighted the lack of accessibility expertise both internally in the public sector and 
available to hire. This was a particular problem when MS needed to hire external auditors 
(testers)  for  the  monitoring  activities.  The  public  consultation  also  identified  the  most 
relevant challenges as: (i) PSB web managers often lack skills in web accessibility; (ii) 

108  See above (note 106). 

25 

 
                                                 
technical difficulties in making existing websites/mobile applications compliant; and (iii) 
a lack of resources (both staff and money). 

The  WAD  has  shown  beneficial  effects  in  stimulating  the  job  market  for  web 
accessibility services 109 and capacity building. This has created new job opportunities and 
profiles (e.g. web accessibility specialist, web accessibility coordinator, web accessibility 
software engineer, web accessibility analyst) 110, and changed existing ones (e.g. the work 
of front-end engineers, user-interface developers, etc.). These findings are also confirmed 
by the available literature on the topic 111, as testing for accessibility requires distinct skills, 
as well as the use of specific tools. 

However, despite the increased number of private service providers in  the EU after the 
implementation of the WAD, MS monitoring bodies reported a shortage and/or high cost 
of experts to conduct accessibility audits. A Wall Street Journal article 112 from September 
2021 cited LinkedIn data that the number of job listings with “accessibility” in the title had 
grown  78%  in  the  year  August  2020  –  July  2021.  In  September  2022,  the  EURES  job 
database  listed  over  1,000  job  vacancies  with  accessibility  in  the  title 113.  As  manual 
accessibility checks must be done locally in the local language(s) it is important to ensure 
there is sufficient accessibility expertise in all regions in all the MS.  

The accessibility industry representatives interviewed expect that in the coming years the 
job market will develop even more, driven by higher demand from PSBs and by the new 
legal obligations for the private sector set out in the European Accessibility Act 114. 

Digital  accessibility  therefore  needs  to  feature  far  more  prominently  in  information 
technology courses, as well as usability design training. For more than a decade, the Digital 
Competence  Framework  for  Citizens  (DigComp) 115,  the  EU-wide  framework  for 
developing  and  measuring  digital  competence,  has  provided  a  common  understanding, 
across the EU and beyond, of what constitutes “digital competence”.  In 2022, the Digital 
Competence Framework – DigComp 2.2 116 – was updated to explicitly mention ‘digital 
accessibility’  across  several  knowledge,  skill  and  attitude  competencies,  including  the 
awareness  of  legal  requirements  such  as  accessibility  statements.  This  should  ensure 

109  Rajšp,  A.,  Kous,  K.,  Kuhar,  S.,  Sumak,  B.,  &  Sorgo,  A.  (2019),  cited  above  (note  79).  The  authors 
highlight that training new professionals to the adequate level of competencies in the sector is extremely 
beneficial towards achieving the goals of the Directive, improving both the internal market of jobs, and 
the quality of life of end-users with disabilities. 

110  For example, EU co-funded Erasmus+ Certified Digital Accessibility Training project offers specialised 
courses for: Digital Accessibility Manager, Digital Accessibility Tester, Web Developer with expertise 
in Digital Accessibility, and Web Designer with expertise in Digital Accessibility. 

111  Rubano, V., & Vitali, F. (2021). Making accessibility accessible: strategy and tools. 2021 IEEE 18th 

Annual Consumer Communications & Networking Conference (CCNC). Las Vegas: IEEE. 

112  Alcántara, Ann-Marie, WSJ, More Companies Are Looking to Hire Accessibility Specialists, 1.9.2021. 
113  https://ec.europa.eu/eures/portal/jv-se (search on 25 September 2022), for all European countries, with 

‘accessibility’ keyword. 

114  Study, cited above (note 17), p. 63. 
115  Joint Research Centre EU Science Hub (2022), The Digital Competence Framework. 
116  Ibid., DigComp 2.2 - The Digital Competence Framework for Citizens (PDF). 

26 

 
                                                 
accessibility is recognized as a basic part of digital competence and is taught throughout 
the MS. 

This fits also the targets and policy of the Digital Decade which is about making digital 
technology  work  for  people  and  businesses 117  and  with  the  target  of  20  million  ICT 
specialists in the EU by 2030 118, with significant funding under the DIGITAL programme 
to bridge the digital skills gap 119. There is clearly a high need for digital skills when it 
comes  to  integrating  digital  technologies  in  the  work  processes,  including  digital 
accessibility skills for both civil servants, and for contractors. Skills and competences of 
civil servants could be systematically boosted through specialised training 120. 

One should emphasize the opportunities for people with disabilities in this growing job 
market, particularly given the surge in remote work 121, which may help some.  In most EU 
countries, the percentage of remote jobs has increased nearly 5 times compared with the 
pre-COVID  period,  especially  in  the  Information  &  Communication  Technology 
sector 122. People with disabilities could be involved as professional accessibility auditors 
(testers) of websites and mobile applications, which can be stepping-stone jobs into the 
workplace. This is an important issue, as the unemployment rate for persons with disability 
is significantly and persistently higher than that of the general population 123. 

MS  also  identified  a  lack  of  official  skill  certification  systems 124.  Currently,  the  IAAP 
(International  Association  of  Accessibility  Professionals) 125  offers 
the  main 
internationally recognised professional qualification, with four types of certifications for 
online accessibility. In practice, auditors and external experts are generally hired based on 

117  According  to  press  statement  by  the  executive  Vice-President  for  A  Europe  Fit  for  the  Digital  Age, 
Margrethe  Vestager:  ‘The  Digital  Decade  is  about  making  digital  technology  work  for  people  and 
businesses. It is about enabling everyone to have the skills to participate in the digital society. It is about 
bringing government services closer to citizens. Europe's digital transformation will give opportunities 
for everyone.’ 14.7.2022. 

118  More than double compared to 2021 when almost 9 million people in the EU worked as ICT specialists, 

representing 4.5% of the total EU workforce. 

119  Regulation (EU) 2021/694 of the European Parliament and of the Council of 29 April 2021 establishing 
the  Digital  Europe  Programme  and  repealing  Decision  (EU)  2015/2240  (OJ  L  166,  11.5.2021).  The 
programme aims, among others, to bridge the digital divide, in particular under specific objective 4: 
advanced digital skills, with indicative envelope of EUR 577 million. 

120  Supporting Public Administrations in EU Member States to Deliver Reforms and Prepare for the Future, 

SWD(2021)101 final, cited above (note 7), p. 11. 

121  European Commission (2021), Joint Research Centre, Science Policy Briefs: Telework in the EU before 

and after the COVID-19: where we were, where we head to (PDF). 

122  Textkernel.nl, Remote Work, The current remote work trends in EMEA: An analysis across 8 European 

countries – Pre and during pandemic, 2018 – 2021. 

123  European  Parliament  (2020),  European  Parliamentary  Research  Service  (EPRS),  Employment  and 
disability  in  the  European  Union,  Briefing  (PDF),  p.  2,  “the  unemployment  rate  of  persons  with 
disabilities in the EU, aged 20-64, is 17.1% compared to 10.2% of persons without disabilities”. 

124  Study, cited above (note 17), pp. 62, 65. 
125  International 

Association 
https://www.accessibilityassociation.org/s/certification. 

of 

Accessibility 

Professionals 

(IAAP), 

27 

 
                                                 
their professional experience and references they can provide, rather than on professional 
qualifications. 

4.1.8  Accessibility is increasing, but more is needed, especially at regional and local 

levels 

As the public sector continues to digitise its delivery of public services, the expectation 
increasingly is for the public to use online services by default, also in line with the Digital 
Decade 126  target  to  have  100%  of  key  public  services  online  by  2030.  According  to 
targeted consultations organisations representing persons with disability, consumers and 
older people, the general opinion is that the WAD has been more effective at improving 
the  accessibility  of  PSB  websites  at  the  national  level,  and,  to  a  lesser  extent,  at  the 
regional and local level in all MS 127. 51% of these respondents believed that, since the 
application of the WAD, there has been an improvement in online information available 
on  websites  at  national  level.  The  statistics  suggest  a  slightly  lower  impact  on  the 
accessibility of websites  at the regional (45% reported an improvement) and local level 
(38%). 

This may be related to the size of public administrations, as larger PSBs tend to have more 
budget, staff and experience to enforce web accessibility. It is hoped and expected that as 
PSBs progress to the digital decade targets, accessibility will be embedded by design from 
the  start  –  e.g.,  as  observed  in  national  recovery  and  resilience  plans  that  feature 
prominently  measures  for  digitalising  public  services  and  introducing  or  improving  e-
government  solutions  to modernise public administration. To close the digital  divide in 
rural  and  remote  areas,  the  EU  supports  investments  in  an  unprecedented  manner.  For 
example,  around  EUR  16  billion  Recovery  and  Resilience  Facility  (RRF)  reforms  and 
investments have already been approved to roll out digital connectivity networks in the 
next four years, especially in rural regions 128. 

On the adoption of similar standards in the private sector, as encouraged by Recital 34 
of the WAD, web accessibility has been – and still is – a low priority in the private sector 129 
(see below Box 3). Some of the examples mentioned at Recital 34 will be covered by the 
EAA. Industry players reported that the WAD, together with the EAA, should help change 

126  2030 digital compass, COM(2021) 118 final, cited above (note 45). 
127  Study, cited above (note 17) and Annex V Synopsis report. According to targeted survey for persons 
with  disabilities  or  those  taking  care  of  persons  with  disabilities  (n=40):  respondents  noticed 
improvements in accessibility of online information at national level (51%), regional level (45%) and 
local level (38%). According to another survey for organisations representing persons with disabilities, 
older people or consumers (n=48), respondents rated overall accessibility levels of public sector online 
services less insufficient on national level (31%) than on regional (42%) or local level (46%). 

128  The Recovery and Resilience Facility is the key instrument at the heart of NextGenerationEU to help 
the EU emerge stronger and more resilient from the current crisis. Recovery and Resilience Scoreboard 
(2022): Digital transformation. Report from the Commission to the European Parliament and the Council 
– Review report on the implementation of the Recovery and Resilience Facility, COM(2022) 383 final 
(PDF), 29.7.2022. For thematic analysis and project examples, see  Thematic analysis –Digital public 
services (PDF), 2021. 

129  Sik-Lanyi,  C.,  &  Orban-Mihalyko,  E.  (2019).  Accessibility  Testing  of  European  Health-Related 
Websites.  Arabian  Journal  for  Science  and  Engineering,  9171-9190;  Teixeira,  L.,  Eusébio,  C.,  & 
Silveiro,  A.  (2019,  June).  Website  accessibility  of  Portuguese  travel  agents.  In  2019  14th  Iberian 
Conference on Information Systems and Technologies (CISTI) (pp. 1-6). IEEE. 

28 

 
                                                 
this  situation,  with  the  scope  of  accessibility  obligations  expanded  under  the  EAA,  so 
introducing  incentives  for  private  actors  to  develop  accessibility  solutions  (see  section 
4.1.11). 

In this context, many MS went beyond the scope of the WAD in their national laws, as 
detailed below. 

Box 3. Analysis of findings from the Member States going beyond the scope of the WAD 130 
Examples of national legislation that goes beyond the scope of the WAD. Legislation 
that partially covers the private sector in addition to PSBs: 

In  France,  the  relevant  legislation  also  covers  private  enterprises  with  a  commercial 
turnover of more than EUR 250 million. 

In Finland, various specific private companies also fall under the scope of the national 
legislation when their services are intended to provide public services (e.g. companies 
providing public services within the water, energy, transport or postal service sectors; 
insurance  companies 131).  Finland  did  not  monitor  these  private  sector  websites  and 
mobile applications in the first round of monitoring, but plans to include them in the 
next one. 

Many MS 132 opted to included schools, kindergartens and nurseries in their legislation 

Other MS limited the technical exemptions: 

Finland  and  the  city  of  Bremen  (Germany)  set  accessibility  requirements  for  public 
procurement. Czechia and Latvia do not exempt file formats from the Microsoft Office 
suite  of  software  products.  Latvia  also  does  not  exempt  time-based  media  (both  pre-
recorded and live) and third-party content. Slovenia generally does not allow exemptions 
if the content is necessary for administrative services, while Slovakia does not mention 
any  exclusions  in  the  national  legislation.  Poland  created  an  obligation  to  publish  an 
accessible  version  of  any  PSB  content  published  on  a  third-party  website  if  it  is  not 
accessible, and in Sweden content published by a PSB via a third party needs to be made 
as accessible as possible. 

4.1.9  Technological advancements could make accessibility easier for content types 

not covered by WAD 

Article 1 of the WAD allows for the temporary or permanent exclusion from its scope of 
some  types  of  content,  some  websites,  and  some  mobile  applications.  Technological 
advancements  in  digital  accessibility  could  make  accessibility  easier  for  content  types 
currently excluded from the scope of the WAD due to lack of automated or efficient and 

130  Study, cited above (note 17), p. 46. 
131  Ibid.,  other  examples  in  Finland  are:  suppliers  of  secure  electronic  identification  services;  credit 

institutions; payment institutions; investment firms. 

132  Spain,  France,  Italy,  the  Netherlands,  Poland,  Romania,  Slovakia,  and  various  states  and  regions  in 

Belgium, Germany and Austria. 

29 

 
                                                 
easy-to-implement  means  at  the  time  of  adoption  of  the  WAD 133.  Analysis  on  the 
technologies  that  have  significantly  evolved  in  the  last  years  shows  that  automation 
through  artificial  intelligence  and  machine  learning  is  increasingly  used  in  tools 
relevant for web accessibility 134. 

Respondents  to  the  public  consultation 135  identified  the  following  six  technological 
advances as being the most relevant for the WAD: 

  assistive technologies driven by artificial intelligence (AI); 
  accessible authoring tools; 
  AI and machine learning applied to monitoring; 
  accessibility monitoring and testing tools; 
 
  biometrics for identification and security aspects. 

the ‘internet of things’ for accessible products and services; and 

The experts interviewed agreed that automation is increasingly being used in different tools 
relevant for web accessibility. Based on the level of maturity, relevance for the WAD and 
size of the affected user groups, the study prioritized 13 technologies for deeper analysis 
(e.g., speech or sign language recognition, automated accessibility testing, web authoring 
tools)  in  scientific  fields  such  as  machine  learning,  natural  language  processing,  deep 
learning, computer vision, etc 136. 

To highlight some of the most relevant, the market offers automatic captioning for videos 
or online videoconferencing tools, built on speech recognition technologies, but only in 
major languages. While the quality of captions (subtitles) is incrementally improving, so 
is  language coverage. This  is  a welcome development  for WAD, as accurate automatic 
captions in official EU languages (and others) will make it cheaper and easier to subtitle 
live time-based media, currently excluded from the WAD. This will help PSBs to comply 
with the requirement to subtitle pre-recorded time-based media, which are in scope of the 
WAD.  There  are  promising  developments  in  automated  text  simplification  and 
summarisation 137 to support people with cognitive disabilities or reading difficulties 138. 

133  The WAD does not apply to content of websites and mobile applications listed in its Article 1(4), e.g. 

live time-based media. 

134  Study, cited above (note 17), ‘2.3 Technological advances in web accessibility’, pp. 27-29. 
135  See Annex V, point 3.2. 
136  Study, cited above (note 17), Annex F ‘Technological advancements in web accessibility’, p. 221-241. 
The  13  technologies  reviewed  are:  speech  recognition,  biometrics  and  face  recognition,  automated 
accessibility  testing,  automated  text  simplification,  brain-computer  interface,  virtual  and  augmented 
reality devices, sign language recognition, web authoring tools, automated summarisation, automated 
image  annotation,  automated  lip  reading,  speaker  recognition,  wearable  devices.  Scientific  fields 
covered  are  Natural  Language  Processing,  Cloud  computing,  Neural  networks  and  deep  learning, 
Computer vision, Optical Character Recognition, Gesture recognition. 

137  https://www.easyreading.eu/. 
138  Study, cited above (note 17), pp. 83-85. See also section 5.1.3 of this report for further analysis regarding 

relevance of these technological advancements. 

30 

 
                                                 
Research continues into digital access to sign languages 139, and speech / text multimodal 
conversion 140. 

While all promising, there is not sufficient maturity in all languages to modify the current 
exemptions. Except for English and, to a certain extent, German, Spanish and French, at 
present all European languages are massively under-resourced in terms of data, tools and 
technologies 141. 

It should be noted that the context is evolving in favour of accessibility outside the WAD 
obligations.  Social  media  platforms,  for  instance,  make  some  such  tools  available  by 
default. This evolving context is changing expectations and raising awareness, which may 
have a positive impact also on PSB websites and applications even before any potential 
revision  of  the  WAD.  Given  the  likely  spill-over  effect  of  the  increased  accessibility 
requirements of the EAA from 2025 onwards, there appears to be no pressing need to revise 
the WAD exceptions in the near term. 

4.1.10  Efficiency, adequacy and proportionality of the resources 

This section presents evidence-based analysis on (1) cost-effectiveness of the WAD and 
(2) adequacy and proportionality of the resources. 

Overall,  the  supporting  study  suggests  that  the  WAD  was  designed  and  implemented 
efficiently, and the use of EU resources was adequate and proportionate 142. 

However, a detailed assessment of economic, social, and environmental impacts is limited, 
mainly due to the difficulties in quantifying and monetising web accessibility, inclusion 
and  non-discrimination,  as  well  as  due  to  the  limited  availability  of  relevant  data.  The 
social impact of the activities for persons with disability in the digital sphere cannot be 
measured in the absence of comprehensive inclusion and equality data. 

The main costs imposed by the WAD are those for PSB to comply with the accessibility 
standards and those for the monitoring activities borne by MS. However, it is difficult to 
measure such costs, due to a lack of quantitative data. 

The evidence collected shows that accessibility related costs have not constituted a major 
barrier to implementing the WAD, as the national monitoring reports show that there was 
limited use of the disproportionate burden exemption clause. 

The  benefits  can  therefore  be  mainly  measured  in  qualitative  terms.  According  to  the 
supporting  study  the  main  economic  benefits  are  an  increase  in  the  demand  for  web 

139  E.g.  ‘SignOn  project’,  https://signon-project.eu/,  and  ‘EASIER’  https://www.project-easier.eu/  (EU’s 

Horizon 2020 programme). 

140  For example, the Commission has made its language tools, including automated eTranslation and speech 
transcription  services  available  to  all  European  public  sector  bodies,  SMEs,  academia,  and  NGOs  at 
https://language-tools.ec.europa.eu/. 

141  European  Language  Grid  (EU’s  Horizon  2020  programme),  https://live.european-language-

grid.eu/catalogue/dashboard (select all EU languages, and technological factors). 

142  Study, cited above (note 17), p. 79. 

31 

 
                                                 
accessibility products and services in the national markets, driven by PSBs, and an increase 
in opportunities for accessibility service providers to penetrate foreign markets. 

Costs and benefits of accessibility

0%

25%

50%

75%

100%

The objectives of the Directive have been achieved at a
reasonable cost?

15

42

81

101

45

27

The benefits arising from accessible websites and apps
outweigh the costs of implementing the Directive?

107

77

49

45

20

13

Strongly agree

Agree Neither agree nor disagree Not sure Disagree

Strongly disagree

Source: Public consultation (standard survey, n=311) 

With  little  information  publicly  available  about  the  costs,  it  is  not  surprising  that  the 
majority of respondents (58%) were neutral on whether the objectives of the Directive had 
been  achieved  at  a  reasonable  cost  (out  of  311  respondents,  81  neither  agreed  nor 
disagreed, 101 were not sure). However, a majority of the respondents (184 or 59%) agreed 
with the statement that the benefits arising from the availability of accessible websites 
and mobile applications outweigh the costs of implementing the WAD. 

According to  50% of the PSBs consulted (31 out  of 62) in  a targeted survey, the main 
benefit of the WAD has been the possibility of providing services to a larger pool of people 
because of improved accessibility. Moreover, consulted PSBs appreciated the role of the 
WAD  in  encouraging  organisations  to  provide  clearer  accessibility  requirements  in 
procurement  procedures.  Some  of  the  stakeholders  interviewed  identified  as  benefits 
increased usability and fewer complaints about services. 

68% of the PSBs consulted believe that the WAD introduced additional economic costs 
for  their  organisations.  The  main  sources  of  recurring  costs  were:  (i)  improving  the 
accessibility  of  digital  services;  (ii)  training  staff;  and  (iii)  drawing  up  accessibility 
statements.  Despite  this,  only  a  few  MS  reported  that  the  WAD  imposes  high  costs  on 
PSBs to implement its requirements. In fact, MS reported only very few cases of using 
the disproportionate burden exemption clause. 

The monitoring bodies provided information on the costs related to monitoring activities. 
However, the information provided and the figures varied significantly, which makes it 
difficult to provide a high-level overview across all MS. The supporting study contains 
some of these figures. Costs were reported only by few MS and range from a minimum 
cost of €65,000 up to a maximum cost of €650,000. The costs of additional activities, such 
as awareness-raising and training, were not reported by the MS 143. 

According to the industry players replying to the targeted survey, the WAD introduced 
new costs, associated mainly with: (i) hiring new specialists; (ii) updating the products and 
services to meet the new accessibility requirements; and (iii) expanding internal capacities 
to  meet  the  new,  higher  volume  of  demand.  However,  according  to  60%  of  industry 

143  The Directive and Implementing Decision (EU) 2018/1524, cited above (note 30), does not explicitly 

require reporting on costs. 

32 

 
                                                 
players, the common accessibility  requirements introduced by the WAD had a  positive 
impact  on  their  business.  The  main  benefits  highlighted  by  the  respondents  were  an 
increase in the demand for web accessibility products and services in the national markets 
and an increase in opportunities to penetrate foreign markets. The stakeholders interviewed 
reported  that  there  is  still  a  lack  of  industry  players  offering  accessibility  services. 
However, they  expect a significant increase in service  companies and organisations 
working in this area in the coming years. This represents an opportunity for the technology 
and educational sectors as new digital skills will be required by the market. The Digital 
Skills and Jobs Platform 144 may be an important EU-wide instrument to support companies 
looking for specialists. 

Finally, the supporting study also suggests that EU funds invested in the implementation 
of  the  WAD  were  adequate  in  promoting  research  on  web  accessibility  and  accessible 
technology, and in the deployment of new solutions, as well as in supporting accessibility 
and social inclusion for persons with disabilities 145. 

The EU supports MS through a wide range of instruments, from strategic and operational 
analysis  to  the  design  and  the  financing  of  reforms  and  investments.  As  mentioned  in 
section 4.1.8, the EU supports investments in digital transformation of the public sector in 
an unprecedented manner 146. 

According to a recent Commission report to the European Parliament and the Council 147, 
measures supporting the digitalisation of public services and e-government represent more 
than a third of the digital expenditure financed by the Recovery and Resilience Facility 
(RRF). The share of ‘E-government, digital public services and local digital ecosystems’ 
in RRF expenditure supporting the digital transformation is 36% (EUR 47 billion) 148. A 
third of that amount (around EUR 16 billion) has already been approved to roll out digital 
connectivity networks in the next four years, especially in rural regions. Moreover, the EU 
aims  to  also  leverage  connectivity  investments  through  the  new  Cohesion  Funds,  the 
EAFRD, InvestEU and EIB loans, and, last but not least, through CEF Digital 149. Also the 
Digital Europe Programme recognizes that digitalisation can facilitate and improve barrier-
free  accessibility  to  all,  including  older  people,  people  with  reduced  mobility  or  with 
disabilities, and people in remote or rural areas 150. 

144  Digital Skills and Jobs Platform, https://digital-skills-jobs.europa.eu/en. 
145  Study, cited above (note 17), pp. 74, 80. 
146  Here are only some examples and highlights provided. For a detailed overview of EU support measures, 
see Supporting Public Administrations in EU Member States to Deliver Reforms and Prepare for the 
Future, SWD(2021)101 final, Section 4, cited above (note 7). 

147  Report from the Commission to the European Parliament and the Council on the implementation of the 

Recovery and Resilience Facility, COM(2022) 75 final, 1.3.2022. 

148  Recovery  and  Resilience  Scoreboard  (2022):  Digital  transformation,  COM(2022)  383  final  (PDF); 

Thematic analysis –Digital public services (PDF), cited above (note 128). 

149  DESI: Digital infrastructures 2022 (PDF), p.9. 
150  Regulation (EU) 2021/694 of the European Parliament and of the Council of 29 April 2021 establishing 
the  Digital  Europe  Programme  and  repealing  Decision  (EU)  2015/2240  (OJ  L  166,  11.5.2021).  The 
programme  aims,  among  others,  to  bridge  the  geographical  digital  divide,  including  the  outermost 

33 

 
                                                 
It is worth noting that public administrations at central, regional and local level implement 
directly about 35% of the EU budget. In some MS, more than 50% of public investments 
were funded under the EU cohesion policy 151. 

These  investments  in  e-government,  and  the  digital  transformation  in  general,  should 
directly improve web accessibility. The Common provisions regulation 152 explicitly spells 
out  that  operations  must  comply  with  applicable  Union  law  and  national  law,  which 
includes  WAD  and  its  transposed  rules  in  national  law.  The  EU  has  co-financed  many 
national  projects  through  its  structural  funds,  which  have  produced  several  relevant 
outcomes, both in terms of new tools/solutions and in knowledge sharing 153. These can be 
considered a return on investment for the achievement of the two main objectives of the 
WAD. 

The Quality of Public Administration Toolbox 154 is a summary of Commission knowledge 
on  improving  public  administration,  which  has  a  dedicated  knowledge  base  on 
digitalisation and accessibility. The Technical Support Instrument (TSI) 155 aims to provide 
tailor-made expertise (technical support) on the ground to help MS design, develop and 
implement growth-enhancing reforms in a wide range of policy areas, including for digital 
public administration 156. 

The Commission also provides knowledge services like the Knowledge4Policy platform 
that  brings  together  policymakers  and  scientists  across  the  EU  for  evidence-informed 
policymaking 157.  These  centres  support  policymaking  by  monitoring  the  uptake  and 
impact of artificial intelligence in the public sector 158. 

To  illustrate  some  more  detailed  examples  related  to  WAD,  the  EU  has  allocated 
resources to implement the WAD in two further ways. Firstly, the WADEX group, using 

regions,  in  particular  under  specific  objective  4:  advanced  digital  skills  and  specific  objective  5: 
deployment and best use of digital capacity and interoperability. 

151  European  Commission,  European  Structural  and  Investment  Funds.  Percentage  of  cohesion  policy 

funding in public investments per Member State 2015-2017. 

152  Regulation (EU) 2021/1060 of the European Parliament and of the Council of 24 June 2021 laying down 
common provisions on the European Regional Development Fund, the European Social Fund Plus, the 
Cohesion Fund, the Just Transition Fund and the European Maritime, Fisheries and Aquaculture Fund 
and financial rules for those and for the Asylum, Migration and Integration Fund, the Internal Security 
Fund  and  the  Instrument  for  Financial  Support  for  Border Management  and  Visa  Policy  (OJ  L  231, 
30.6.2021), in particular Articles 2(3) and 74(1)(a). 

153  For example, according to Commission’s  Kohesio database, digital accessibility of local government 
sites project in Poland, with a budget of 1,1 million euro and co-funded from European Social Fund, 
aims to train public administration officials to create digitally accessible resources and to develop IT 
tools  to  support  the  process  of  monitoring  the  implementation  of  digital  accessibility  in  public 
administrations. 

154  Quality  of  Public  Administration.  A  Toolbox  for  Practitioners,  2017,  in  particular Theme  5:  Service 

delivery and digitalisation (PDF). 
155  Technical Support Instrument (TSI). 
156  European  Commission,  Directorate-General  for  Structural  Reform  Support,  Governance  and  public 

administration (PDF), 2020. 

157  https://knowledge4policy.ec.europa.eu/home_en. 
158  https://ai-watch.ec.europa.eu/topics/public-sector_en. 

34 

 
                                                 
it  to  facilitate  the  exchange  of  best  practices  between  the  MS,  as  well  as  with  relevant 
stakeholders and experts where appropriate. Secondly, the European Commission funded, 
through the Horizon 2020 programme, pilot projects and preparatory actions, such as (e.g. 
WAI-Tools 159,  WADcher 160,  WAI-Guide 161,  and  We4Authors 162)  in  the  areas  of  web 
accessibility,  accessible  technology,  and  the  deployment  of  solutions.  Most  of  the  MS 
considered that their engagement in the implementation of the WAD had contributed to 
achieving its objectives. The WADEX group has been very useful for these stakeholders, 
especially by making possible to obtain and exchange technical information and to meet 
and engage with other people working on web accessibility. Many member appreciated the 
meetings as a forum to exchange views on the standards, the monitoring methodologies 
and other practical examples. 

4.1.11  Coherence with other policies and instruments 

The supporting study confirmed that the WAD is coherent internally, and with relevant EU 
and international policies and actions, as well as with the UNCRPD, the EU Charter of 
Fundamental Rights and the European Pillar of Social Rights. This is based on feedback 
provided by stakeholders during consultation activities and the review of relevant EU and 
international policies. 

The supporting study shows that the WAD is coherent internally with the overarching 
Commission strategies on disability, the European disability strategy for 2010-2020 163 and 
the  Strategy  for  the  rights  of  persons  with  disabilities  for  2021-2030 164.  The  current 
strategy  itself  refers  to  this  evaluation  and  commits  to  include  accessibility  and 
inclusiveness in the reinforced EU digital government strategy, focusing on human-centric 
and  user-friendly  digital  public  services  across  Europe  that  respond  to  the  needs  and 
preferences of European citizens, including the needs of persons with disabilities. 

Effective  use  of  digital  technologies  requires  the  removal  of  accessibility  barriers  for 
persons with disabilities and investing in their digital skills. As announced in the Digital 
Education  Action  Plan  2021-2027 165,  MS  will  be  supported  in  securing  assistive 
technologies and in providing an accessible digital learning environment and content. To 
promote  disability-inclusive  education,  MS  can  use  the  opportunities  offered  by  EU 

159  https://cordis.europa.eu/project/id/780057  (EU  contribution  EUR  1  999  812),  project  website 

https://www.w3.org/WAI/about/projects/wai-tools/. 

160  https://cordis.europa.eu/project/id/780206  (EU  contribution  EUR  1  957  011,25),  project  website 

https://wadcher.eu/. 

161  https://cordis.europa.eu/project/id/822245  (EU  contribution  EUR  1  499  742),  project  website 

https://www.w3.org/WAI/about/projects/wai-guide/. 

162  Pilot on web accessibility for web authoring tools producers and communities (budget EUR 150 000), 

https://www.funka.com/en/projekt/we4authors/what-is-we4authors. 

163  European  Disability  Strategy  2010-2020:  A  Renewed  Commitment  to  a  Barrier-Free  Europe, 

COM(2010) 636 final (PDF), 15.11.2010. 

164  Strategy for the Rights of Persons with Disabilities 2021-2030, cited above (note 5). 
165  Digital  Education  Action  Plan  2021-2027,  Resetting  education  and  training  for  the  digital  age, 
COM(2020) 624 final (HTML), 30.09.2020. Related content on  https://education.ec.europa.eu/focus-
topics/digital-education/action-plan. 

35 

 
                                                 
funding, including the Erasmus+ and the European Solidarity Corps programmes, setting 
out dedicated inclusion measures. Furthermore, for schools and educational buildings, MS 
can also address accessibility through the Renovation Wave 166 aiming to make liveable 
environments accessible to everyone. 

The  European  Accessibility  Act  (EAA) 167  complements  the  WAD  in  both  scope  and 
coverage.  It  has  a  strong  digital  relevance  that  aims  to  improve  the  functioning  of  the 
internal  market  for  accessible  products  and  services,  by  removing  barriers  created  by 
divergent rules in MS. As the WAD, the EAA establishes accessibility requirements for a 
range of products and services that have been identified as rather important for persons 
with  disabilities while having diverging accessibility requirements across EU countries. 
This  includes,  among  others,  e-commerce,  banking,  and  transport  services,  concerning 
both public and private sectors. 

The MS had to transpose the EAA by 28 June 2022 and apply in practice as of 28 June 
2025,  so  businesses  across  the  EU  need  to  ensure  that  the  services  they  provide  and 
products  they  deal  with  comply  with  the  EAA’s  accessibility  requirements.  The  scope 
EAA is complementary to WAD, covering also websites and applications of the private 
sector.  Presumption  of  conformity  with  the  EAA  will  be  based  on  the  same  (updated) 
European  standard  EN 301 549  harmonised  under  the  WAD  for  websites  and  mobile 
applications. In addition, two other updated standards (EN17161 and EN17210) and three 
new ones will be developed to support specific elements of the EAA. 

The WAD is also coherent with other relevant EU legislation in the field of digital policies. 
Most recently, the updated set of rules for all digital services within the Digital Services 
Act 168  also  contains  an  obligation  for  the  Commission  to  encourage  and  facilitate  the 
drawing  up  of  Codes  of  Conduct.  Such  a  code  will  ensure  that  services  provided  are 
designed in  an understandable and  robust way in order to  ensure they are accessible to 
persons with disabilities, and it is publicly explained how accessibility requirements are 
met.  Providers  of  services  as  well  as  civil  society  representatives  and  recipients  of  the 
services should be involved in their drafting. The Audiovisual Media Services Directive 
(AVMSD) 169 obliges MS to adopt rules to impose accessibility requirements on providers 
of  audiovisual  media  services.  The  European  Electronic  Communications  Code 
(EECC) 170,  which  aims  to  modernise  EU  telecoms  rules,  covers  service  providers  and 

166  European Commission (2020), Renovation Wave: doubling the renovation rate to cut emissions, boost 

recovery and reduce energy poverty (HTML). 

167  Cited above (note 84). 
168  Regulation (EU) 2022/2065 of the European Parliament and of the Council of 19 October  2022 on a 
Single Market For Digital Services and amending Directive 2000/31/EC (Digital Services Act) (OJ L 
277,  27.10.2022);  Press  release  16.11.2022,  ‘Digital  Services  Act:  EU’s  landmark  rules  for  online 
platforms enter into force’. 

169  Directive  2010/13/EU  of  the  European  Parliament  and  of  the  Council  of  10  March  2010  on  the 
coordination  of  certain  provisions  laid  down  by  law,  regulation  or  administrative  action  in  Member 
States concerning the provision of audiovisual media services (Audiovisual Media Services Directive) 
(OJ L 95, 15.4.2010). 

170  Directive  (EU)  2018/1972  of  the  European  Parliament  and  of  the  Council  of  11  December  2018 

establishing the European Electronic Communications Code (Recast) (OJ L 321, 17.12.2018). 

36 

 
                                                 
requires that the services they provide be accessible. EECC does not cover content and 
does not lay down accessibility requirements for what is placed on the internet, but aims 
for equivalent access for end-users with disabilities, addressing the need for available and 
affordable equipment and services and the access to emergency services. The Directive 171 
and the Regulation implementing the Marrakesh Treaty 172 share the same objectives of 
accessibility as the WAD but cover different types of products. The Marrakesh Treaty sets 
the legal framework for changes needed in national copyright regimes to implement legal 
copyright  exemptions  for  the  benefit  of  blind,  visually  impaired,  and  otherwise  print 
disabled  persons  that  would  enable  the  beneficiaries  or  authorised  entities  to  make 
accessible format copies of books and other written material. The Treaty also facilitates 
the  exchange  of  accessible  format  copies  across  national  borders  between  authorised 
entities or with the beneficiaries. Finally, the eIDAS Regulation 173, which ensures that 
people and businesses can use their own national electronic identification schemes (eIDs) 
to access public services available online in other EU countries, is also coherent with the 
objectives of the WAD, by requiring that trust services and end-user products used in the 
provision of those services shall be made accessible for persons with disabilities, where 
feasible. Accessible eID and trust services are highly relevant for completing processes on 
the web, important in public services covered by the WAD, so accessible eID and trust 
services should lead to increased use of such online PSB services. 

The  requirements  in  the  WAD  together  with  the  measures  above  establish  a  coherent 
regulatory system that facilitates digital accessibility both in the public and private sector. 

Almost  all monitoring bodies interviewed during the consultation  activities reported  no 
overlaps or inconsistencies between the WAD and other relevant national measures 
related to web accessibility. German national law also sets another standard for checking 
the accessibility of PDF  files (the ‘PDF UE’ standard), but the two standards, although 
having different requirements, do not contradict each other 174. 

The WAD fits into the aims and objectives of major treaties and instruments addressing 
issues of digital and social inclusion for people with disabilities, to which the Commission 
is  party,  both  at  European  and  international  level,  i.e.  the  EU  Charter  of  Fundamental 

171  Directive  (EU)  2017/1564 of  the  European  Parliament  and  of  the  Council  of  13  September  2017  on 
certain permitted uses of certain works and other subject matter protected by copyright and related rights 
for the benefit of persons who are blind, visually impaired or otherwise print-disabled and amending 
Directive  2001/29/EC  on  the  harmonisation  of  certain  aspects  of  copyright  and  related  rights  in  the 
information society (OJ L 242, 20.9.2017). 

172  Regulation (EU) 2017/1563 of the European Parliament and of the Council of 13 September 2017 on 
the cross-border exchange between the Union and third countries of accessible format copies of certain 
works and other subject matter protected by copyright and related rights for the benefit of persons who 
are blind, visually impaired or otherwise print-disabled (OJ L 242, 20.9.2017). 

173  Regulation  (EU)  No  910/2014  of  the  European  Parliament  and  of  the  Council  of  23  July  2014  on 
electronic identification and trust services for electronic transactions in the internal market and repealing 
Directive 1999/93/EC (OJ L 257, 28.8.2014). 

174  The WAD does not apply to office file formats published before 23 September 2018, whereas in EAA 

that date is 28 June 2025. 

37 

 
                                                 
Rights 175 and the European Pillar of Social Rights (EPSR) 176, the UNCRPD 177 and the 
UN Sustainable Development Goals (SDGs) 178. 

The  WAD  is  coherent  with  the  relevant  articles  of  the  EU  Charter  of  Fundamental 
Rights, including: (i) on increased social participation for older people and people with 
disabilities (Articles 25 and 26); (ii) on the right of access to documents (Article 42); (iii) 
on  removing  any  form  of  discrimination,  including  disability  (Article  21);  and  (iv)  on 
freedom of expression and information (Article 11). 

Regarding the EPSR, coherence with the objectives of the WAD is evident especially: (i) 
on  inclusion  of  people  with  disabilities,  including  through  services  that  enable  them  to 
participate in the labour market and in society (Principle 17); and (ii) on access to essential 
services  which  states  that  everyone  has  the  right  to  access  essential  services  of  good 
quality, including digital communications (Principle 20). 

The WAD is also consistent with and supports the proposed digital principle that everyone 
should have access to technology and to all key public services online across the Union 179. 

The WAD has significant synergies with the UNCRPD, currently the most relevant piece 
of international law in the area of disability and is coherent with its objectives. Its two most 
relevant articles are: Article 9, which focuses specifically on accessibility, and Article 21, 
which focuses on freedom of expression and opinion, as well as on the freedom to retrieve 
and  access  information.  As  society  becomes  more  digitalised  and  most  administrative 
services are available online, a number of other UNCRPD articles show synergies with the 
WAD 180.  As  a  precondition  of  human  rights,  accessibility  indirectly  links  also  to  the 
exercise of rights of persons with disabilities under the UNCRPD provisions 181. 

Finally,  all  of  the  UN  Sustainable  Development  Goals  (SDGs)  are  coherent  with  the 
achievement of more inclusion of people with disabilities in society 182. The provisions of 

175  Charter of Fundamental Rights of the European Union (OJ C 326, 26.10.2012). 
176  European  Pillar  of  Social  Rights,  https://ec.europa.eu/info/strategy/priorities-2019-2024/economy-

works-people/jobs-growth-and-investment/european-pillar-social-rights_en. 

177  UN 

Convention 

on 

the 

Rights 

of 

Persons 

with  Disabilities 

(UNCRPD), 

https://www.un.org/development/desa/disabilities/convention-on-the-rights-of-persons-with-
disabilities.html. 

178  UN Sustainable Development Goals (SDGs), https://sdgs.un.org/goals. 
179  European Declaration on Digital Rights and Principles  for the Digital  Decade,  COM(2022) 28 final, 

26.1.2022. See more at https://digital-strategy.ec.europa.eu/en/policies/digital-principles. 

180  E.g.,  the  definitions  for  communication,  reasonable  accommodation,  universal  design  (Article  2); 
general  principles  of  accessibility  (Article  3(f));  general  obligations  (Article  4(a,  b,  c,  h))  of  the 
UNCRPD. 

181  E.g., on Living independently and being included in the community (Article 19(a, b, c)) requiring that 
community services and facilities for the general population are available on an equal basis to persons 
with  disabilities  and  are  responsive  to  their  needs;  Work  and  employment  (Article  27(a,  c,  e,  h)); 
Adequate standard of living and social protection (Article 28(b, c, d, e); Participation in political and 
public life (Article 29); and on ensuring that international cooperation is inclusive of and accessible to 
persons with disabilities (Article 32(a)) of the UNCRPD. 

182  https://www.un.org/development/desa/disabilities/publication-disability-sdgs.html. 

38 

 
                                                 
the WAD and its implementing acts are, in particular, relevant for achieving the objectives 
of SDGs 1, 3, 4, 8, 9, 10, 16, and 17 as flagged by the UN Flagship Report on Disability 
and Sustainable Development Goals 183. Achieving such goals requires the accessibility of 
PSBs’ digital administrative procedures, including services of general interest. The list of 
all relevant SDGs is provided in Annex VII with a description of how the key provisions 
of WAD and its implementing acts contribute to these 184. 

4.2  How did the EU intervention make a difference and to whom? 

As described above, especially at sections 4.1.7 and 4.1.8, the WAD has directly impacted 
persons with disabilities, accessibility solution providers, as well as the public sector itself. 

The analysis considered to what extent the value of the EU intervention can be considered 
additional to what could have been achieved by MS acting alone. In particular, it sought to 
assess to what extent the WAD made a difference in removing internal market barriers and 
in  improving  digital  access  to  persons  with  disabilities  and  older  people.  Finally,  the 
analysis  evaluated  to  what  extent  the  WAD  generated  additional  value  compared  to 
national measures implementing the UNCRPD, the EU Charter of Fundamental Rights and 
the EPSR. 

To what extent do you agree with the statement that, because of the 
Directive, Member States have...
0%

25%

50%

made public services
more accessible online

54

153

75%

47

100%

32

17

8

increased digital inclusion of users,
people with disabilities

30

helped exercise rights under
the Charter of Fundamental Rights

implemented the accessibility
provisions of the UNCRPD

16

15

achieved objectives of the
European Pillar of Social Rights

12

135

63

42

23

18

118

66

72

23

16

87

71

77

68

44

20

85

112

22

9

Strongly agree Agree Neither agree nor disagree Not sure Disagree

Strongly disagree

Source: Public consultation (standard survey, n=311) 

Most of the respondents to the public consultation agreed that there is an EU added value 
of the WAD in improving online accessibility for public services and in increasing the 
digital inclusion of users. Although information pre- and post-Directive on the accessibility 
of  PSBs’  websites  and  mobile  applications  is  limited,  interviewed  monitoring  bodies 
reported a significant increase in the awareness of web accessibility among PSBs since 
the  implementation  of  the  WAD.  In  particular,  the  monitoring  activities  have  been  an 
important  occasion  for awareness  raising. Many  PSBs addressed accessibility problems 
after the start of the monitoring period. 

183  https://social.un.org/publications/UN-Flagship-Report-Disability-Final.pdf. 
184  The mapping of SDGs in relation to WAD was partly done using the SDG Mapper tool, developed by 

the JRC and DG INTPAJRC, https://knowsdgs.jrc.ec.europa.eu/. 

39 

 
                                                 
The other added value reported by monitoring bodies is that, for 22 MS, the adoption of 
the WAD meant the introduction of national laws on web accessibility that did not 
previously exist. The general finding from the in-depth interviews was that the MS alone 
would  not  have  reached  this  level  of  awareness  and  web-accessibility  related  activity 
without the WAD. 

Concerning  the  EU  added  value  in  removing  barriers  across  the  internal  market,  most 
respondents across different stakeholder categories agreed that the main benefit has been 
the  harmonisation  of  national  legislation,  which  has  brought  tangible  results  in  the 
internal market, particularly the common standard which facilitates cross-border provision 
of web accessibility products and services. Industry players noted that several barriers to 
the  cross-border  provision  of  web  accessibility  services  had  been  removed.  Most  PSBs 
consulted (79%, 49 out of 62) believed that the WAD had contributed to making national 
laws  on  web  accessibility  requirements  more  harmonised.  Similarly,  72%  of  the 
organisations consulted, representing persons with disabilities, older people and consumers 
(35 out of 48 such organisations consulted) agreed that the WAD had contributed to the 
harmonisation of the legislative framework on web accessibility across EU countries. In 
addition,  54% of the organisations also  believed that the WAD enhanced  cross-border 
cooperation among different associations of people with disabilities. 

The  main  contributions  of  the  WAD  to  the  national  initiatives  of  MS  have  been:  (i) 
increasing the demand for web accessibility tools from public authorities (reported by 65% 
of the respondents); and (ii) spreading awareness about accessibility solutions among web 
professionals (reported by 62% of the respondents). 61% of industry players agreed with 
the statement that both the demand for – and supply of – web accessibility products 
and  services  increased  after  the  introduction  of  the  WAD.  In  addition,  44%  of  the 
respondents emphasised the impact of the WAD in incentivising private actors to develop 
accessibility solutions. 

The available literature agrees that the introduction of harmonised standards is a first 
essential step in removing barriers in the internal market. In addition, some authors 
have also shown the beneficial effects of the WAD in stimulating the job market for web 
accessibility services 185. 

4.3 

Is the intervention still relevant? 

The supporting study and the online surveys find that the WAD is still relevant, particularly 
considering the evolving European and international policy context 186. 

185  Rajšp, A., Kous, K., Kuhar, S., Sumak, B., & Sorgo, A. (2019), cited above (note 79). See also section 

4.1.7. 

186  Study, cited above (note 17), p. 106. 

40 

 
                                                 
To what extent is it still relevant... 

0%

20%

40%

60%

80%

100%

for PSBs to provide accessible
online content and services?

for PSBs to provide accessible
mobile content and services?

for WAD to contribute to digital
inclusion of people with disabilities?

reduce regulatory differences
among MS in web accessibility?

269

256

260

34

5
3

40

5

8

2

42

3
4
2

195

75

4

33

4

Very relevant

Relevant

Slightly relevant

Not sure

Not relevant

Source: Public consultation (standard survey, n=311) 

As can be seen above, the overwhelming majority of respondents to the public consultation 
consider it still relevant for PSBs to provide accessible content and services both online 
(97%) and on mobile devices, including apps (95%). It is still highly relevant for WAD to 
contribute to digital inclusion (97%) and to reduce regulatory differences among MS in the 
field of web accessibility (87%). 

The  WAD  remains  relevant  for  all  main  stakeholders,  as  the  increasing  use  of  PSB 
websites and online services shows the continuing need for more accessible websites and 
mobile  applications.  The  amount  of  online  information  provided  by  public  sector 
organisations has increased in the last three years according to 85% of respondents (265 
out of 311). Similarly, most respondents reported an increase of interactive services (84%), 
mobile applications (78%) and online documents (72%) 187. 

PSBs overall have not reacted to the new accessibility obligations by limiting their online 
presence.  The  surveys  and  interviews  noted  only  a  few  examples  of  limiting  online 
presence, such as reducing the number of published pdf documents, removing originally 
live-streamed  videos  after  2  weeks,  or  publishing  videos  on  third-party/social  media 
platforms.  Overall,  MS  have  not  widely  reduced  online  content  in  order  to  avoid  the 
WAD’s requirements. 

Indeed  many  MS  have  gone  beyond  the  scope  of  the  WAD  in  their  national  laws,  as 
detailed  at  section  4.1.8,  which  implies  a  genuine  commitment  to  digital  inclusion  for 
persons with disabilities. 

Regarding  the  policy  context,  the  WAD  objectives  are  directly  relevant  to  the 
implementation of the UNCRPD. On the UN policy framework, the WAD objectives are 
also still relevant to several  global SDGs 188, including SDG 10 on reducing inequality. 
The  WAD  objectives  are  still  pertinent  to  the  Charter  of  Fundamental  Rights  of  the 
European Union. 

187  See Annex V, section 3.3.1. 
188  In particular SDGs 1, 3, 4, 8, 9, 10, 15, 16 and 17 (see section 4.1.1 and Annex VII). 

41 

 
                                                 
The  fact  that  the  WAD  does  directly  not  apply  to  EU  institutions  was  considered  a 
limitation  by  some  of  the  stakeholders  interviewed.  The  Commission  has  already 
announced  measures  to  address  this  in  the  new  Strategy  for  the  rights  of  persons  with 
disabilities 2021-2030 189, and has adopted an action plan on web accessibility to ensure 
its web presence is compliant with the provisions of the WAD 190. 

Such intervention is also relevant in the aftermath of COVID-19 which highlighted how 
important  it  is  that  people  with  disabilities  can  access  public  sector  online  crisis 
information on equal terms 191. 

Tackling the digital divide is even more important, given the historic shift to using digital 
technologies  in  work,  education,  communication  and  public  services 192.  During  the 
COVID-19 pandemic, digital interaction has been the public administration’s main (and 
often  the  only)  communication  channel  with  citizens,  enabling  it  to  ensure  business 
continuity. 

The responses to the public consultation show that almost all respondents considered the 
objectives of the WAD to still be relevant today 193. Up to 68% of people with disabilities 
or older people reported using online public services of national, regional, and local PSBs 
at  least  monthly.  Looking  at  changes  since  the  adoption  of  the  WAD,  50%  of  the 
respondents (142 out of 311) to the public consultation affirm that their use of government 
websites  increased  slightly  (23%)  or  significantly  (27%)  in  the  last  3  years.  Regarding 
mobile  applications,  78%  agreed  that  the  number  of  applications  provided  by  PSB  has 
increased. On the kind of public services used online, health services were identified as the 
most  accessed  service  (12%),  followed  by  money  and  tax  (8%),  utilities (8%),  benefits 
(7%), employment (7%), disability (6%), and citizenship services (6%). 

In terms of scope, almost half of the respondents (49%) to the public consultation (151 out 
of 311) agreed that the WAD adequately covers online public services to ensure the full 
participation of people with disabilities in digital society. On subject and content types 
not  covered  by  the  WAD,  most  of  the  respondents  favoured  removing  the  current 
exemptions  and/or  expanding  the  scope  of  the  WAD  to  other  areas.  For  example,  full 

189  Strategy for the Rights of Persons with Disabilities 2021-2030, cited above (note 5), point 8.2, where 
the Commission commits to adopt ‘an action plan on web accessibility, to be shared and promoted in all 
EU  institutions,  bodies  and  agencies  in  view  of  ensuring  compliance  of  EU  websites,  documents 
published on these websites and online platforms, with European accessibility standards’. 

190  European Commission (2022), Web Accessibility Action Plan 2022–2025 (HTML);which sets out the 
steps the Commission must take if it is to achieve its stated ambitions and, in particular, ensure that its 
web presence is fully compliant with the provisions of the WAD. 

191  This  is  in  particular  recognized  in  the  new  European  Care  Strategy  and  Commission  Proposal  for  a 
Council Recommendation on access to affordable high-quality long-term care, COM(2022) 441 final 
(PDF). It calls for rolling-out accessible innovative technology and digital solutions in the provision of 
care services, including to support independent living for persons with disability (point 5(c)). 

192  Digital technologies affect the way in which every government function is carried out, from decision-
making to delivering services, collecting and managing tax, communicating, managing projects, etc. See 
European  Commission.  Joint  Research  Centre  (2021).  Digitranscope:  The  governance  of  digitally-
transformed society. 

193  The two general objectives are to ensure websites and mobile applications of PSBs are more accessible 

and to harmonise the internal market for web accessibility products and services. 

42 

 
                                                 
inclusion  of  universities  in  the  requirements  of  the  WAD  was  supported  by  81%  of 
respondents, while full inclusion of schools was supported by 70%. There was also strong 
support  for  full  inclusion  of  NGOs  (56%),  online  mapping  services  (57%),  live  videos 
(53%),  extranets/intranets  (51%)  and  third-party  content  (47%)  given  technological 
advances. 

Finally, on the Harmonised European Standard EN 301 549 (v2.1.2), many respondents 
(63%)  to  the  public  consultation  consider  that  the  standard  has  been  useful  for  making 
public sector websites accessible. Around one third of the respondents (32%) agree that 
the standard still covers all relevant end-user groups, with half of respondents (51%) not 
sure  or  neutral  on  the  matter.  In  the  interviews,  a  few  MS  mentioned  that  services  for 
people with cognitive impairments should be better covered. 

5  WHAT ARE THE CONCLUSIONS AND LESSONS LEARNED? 

5.1  Conclusions 

The  adoption  of  the  WAD  in  2016  aimed  to  address  two  major  problems  in  web 
accessibility: the need to improve (i) the functioning of the internal market for the provision 
of  web  accessibility;  and  (ii)  the  poor  level  of  accessibility  of  websites  and  mobile 
applications providing public services. 

All MS integrated the requirements of the WAD into national legislation and embarked on 
activities: (i) to make PSB websites and mobile applications more accessible; (ii) to ensure 
monitoring, reporting and enforcement of measures; (iii) to promote training programmes 
for  PSBs  on  the  accessibility  of  websites  and  mobile  applications;  and  (iv)  to  raise 
awareness  of  both  the  accessibility  requirements  introduced  by  the  WAD  and  the 
possibility for users to give feedback if they encounter cases of non-compliance with the 
accessibility requirements. 

5.1.1  Effectiveness 

Since  the  adoption  of  the  WAD  in  2016,  progress  has  been  made  concerning  both  the 
accessibility  of  PSB  websites  and  mobile  applications  The  WAD  is  going  in  the  right 
direction  towards  the  achievement  of  its  objectives,  as  both  cross-border  demand  and 
supply of web accessibility products and services increased after the adoption of the WAD. 
Since PSBs  still face challenges  in  complying with the requirements  established by the 
WAD, there is still room for improvement. 

The  Directive  has  helped  to  improve  the  internal  market  for  providing  web 
accessibility  services.  Most  importantly,  the  harmonisation  of  national  legislation 
throughout the EU has substantially improved the opportunities for private actors offering 
web accessibility products and services. Given this harmonisation, the WAD has lowered 
barriers to the supply of web accessibility services and products in other countries within 
the European market and facilitated the increase of cross-border provision of products and 
services for web accessibility. 

There are no official skill-certification systems in most MS, which makes it more difficult 
to identify appropriately trained experts and to assess the specific expertise of the auditors. 

43 

 
Monitoring bodies therefore generally assess the expert’s professional expertise based on 
former experience, rather than formal qualifications. 

The Directive is also working to improve the level of web accessibility of public sector 
websites and mobile applications across the EU. The results from the monitoring reports 
show there is much progress to be made in making all public sector websites and mobile 
applications fully accessible to persons with disabilities. This is partly due to the wording 
of  the  WAD,  calling  for  ‘more  accessible’  websites  or  mobile  applications,  which  can 
comply with the WAD, even if they do not fully meet all the accessibility criteria of the 
Harmonised  European  Standard.  This  should  be  read  since  the  definition  of  “partial 
compliance”  is  extremely  broad.  Regardless,  many  positive  considerations  should  be 
made. 

Monitoring  has  been  considered  an  important  success  factor  in  achieving  the  first 
objective of the WAD, as many PSBs addressed accessibility problems identified during 
the monitoring activities, so showing the value of monitoring and reporting exercises. The 
publication of the monitoring results is important, since PSBs can learn from each other’s 
reports. Moreover, comparing results can be very useful to help allocating resources, as 
they provide a benchmark and raise awareness of the need to “catch up” on accessibility. 

Transparency  of  monitoring  results  can  encourage  healthy  competition  between,  for 
example, municipalities and/or regions, where no one wants to have the lowest score. Good 
and improving scores can be used for various communication purposes. Suppliers of ICT 
services  (including  web  developers,  producers  of  authoring  tools,  and  other  service 
providers)  can  feature  satisfied  clients  in  their  marketing  activities.  Equally,  suppliers 
whose clients have bad results should quickly try to improve. Companies with expertise in 
accessibility issues and developers of accessibility tools can use the reports to focus their 
efforts  on  solving  specific  issues.  PSBs,  and  organisations  representing  people  with 
disabilities (DPOs) can use the reports in their awareness-raising campaigns, and academic 
research can explore and support accessibility efforts. 

The interviews with MS authorities highlighted that the implementation of the WAD at 
national level, together with the training and awareness-raising activities promoted by MS 
had helped to significantly raise awareness among PSBs at different administrative levels. 
Additionally,  the  testing  activities  during  the  monitoring  stage  have  been  an  additional 
important  occasion  for  awareness  raising.  In  some  cases,  the  number  of  websites  with 
accessibility  statements  doubled  from  the  first  to  the  second  year  of  the  WAD’s 
implementation. 

Similar conclusions cannot be drawn yet on mobile applications, given that the monitoring 
period for applications was very short in the first reporting period, and did not allow time 
for re-checking findings. 

The trends noted are likely to continue based on the infrastructure now in place. However, 
most of the monitoring bodies agree that it is still too early to notice an increase in the level 
of accessibility as a specific result of the monitoring activities. More comparable evidence 
will become available starting the second monitoring period, which requires a recurring 

44 

 
sample of websites and mobile applications monitored in the previous monitoring period. 
Given this context, the level of web accessibility is expected to continue increasing in the 
coming years: accessibility is a process, not a one-off exercise. 

5.1.2  Efficiency 

The  results  from  the  public  consultation,  and  the  lack  of  complaints  about  the  cost  of 
implementing the WAD, allow positive conclusions to be drawn as to the efficiency of the 
WAD. 

Both  from  the  perspective  of  PSBs  and  of  monitoring  bodies,  the  WAD  introduced 
additional  economic  costs  to  their  organisations.  However,  the  overall  costs  of  the 
monitoring activities conducted were reported only by a few MS and are all very different 
in nature. The supporting study contains some of these figures. The overall costs for the 
monitoring activities range from  a minimum cost  of €65,000 up to  a maximum  cost  of 
€650,000.  There  are  no  available  figures  on  the  costs  of  making  websites  and  mobile 
applications compliant with the WAD’s requirements. 

The main benefit of the WAD, according to the results of the consultations, has been to 
provide services to more people thanks to improved accessibility. In addition, PSBs can 
achieve direct cost savings by improving web accessibility. The most common example of 
cost  savings  for  PSB  are  related  to  the  use  of  digital  services,  compared  to  printed 
documents, phone services or face to face consultations (note: no data was available on 
cost savings, which could be requested on a voluntary basis for the next reporting period). 

Most of the consulted stakeholders believed that the WAD had not (yet) been effective in 
reducing the prices of these accessibility services. Despite an improvement in the internal 
market, many MS highlighted both: (i) a continued lack of experts in the market; and (ii) 
experiencing  problems  when  they  needed  to  hire  external  auditors  for  the  monitoring 
activities.  The  current  lack  of  relevant  technical  expertise  means  there  is  a  lack  of 
competition, keeping prices relatively high, meaning that the single market does not yet 
function as expected. However, a significant increase in the number of service companies 
and organisations working in this area is expected in the coming years, due to the impact 
of the EAA. Cross-border provision of tools is available and can be expected to increase. 

5.1.3  Coherence 

On the coherence of the WAD with other legislative interventions in the area of web 
accessibility and inclusion, at the international, EU and national level, the findings have 
not identified any relevant inconsistencies or overlaps. On the contrary, there are several 
areas of complementary interaction between the EAA and the WAD, both in scope and 
coverage.  For  example,  the  EAA  covers:  general-purpose  computer  hardware  and 
operating  systems;  automatic  teller  machines  (ATMs);  ticketing  machines;  check-in 
machines;  various  types  of  consumer  terminal;  audiovisual  media  services;  transport 
services  for  air,  bus,  rail  and  waterborne  passengers;  banking  services;  e-books;  and  e-
commerce. The web accessibility requirements  related to  this complementary scope are 
harmonised with the requirements of the WAD. 

45 

 
5.1.4  EU added value 

The EU added value of the WAD has been proven  from  the evidence  collected. The 
main added value identified has been the harmonisation of national legislation, which has 
brought tangible results in the internal market. 

For most countries, the adoption of the WAD also meant the introduction of national laws 
on  web  accessibility  that  were  not  present  before,  and  reference  to  the  same  European 
standard. As a result, industry players interviewed have noted that several barriers have 
been removed and the internal market has substantially improved. The interviews carried 
out with MS, show that since the implementation of the WAD there has been a significant 
overall increase in the awareness of web accessibility among PSBs. 

5.1.5  Relevance 

The  WAD  is  still  considered  as  relevant  for  the  current  policy  context,  at  all  levels: 
national, EU and international. 

The  WAD  remains  highly  relevant  for  its  main  stakeholders,  as  the  amount  of  online 
information provided by public sector organisations has increased in the last three years, a 
trend  expected  to  continue  in  line  with  the  Digital  Decade  target  to  have  100%  of  key 
public services online by 2030. The EU’s commitment to the digital transformation and 
tackling the digital divide is even more important, given the historic shift to using digital 
technologies  in  work,  education,  communication  and  public  services  since  the  current 
Commission took office. It was further highlighted during the COVID-19 pandemic, when 
digital  interaction  has  been  the  public  administration’s  main  (and  often  the  only) 
communication channel with citizens, enabling it to ensure business continuity. 

The context is evolving in favour of accessibility beyond the public sector (and scope of 
the WAD), as the Digital Services Act introduces the possibility of a dedicated code of 
conduct  addressing  the  specific  needs  of  people  with  disabilities  and  the  practical 
application of the European Accessibility Act from 2025 onwards. 

5.2  Lessons learned 

From the evaluation of the Directive, these are the main lessons learned. 

  Insufficient  accessibility  expertise  and  shortage  of  professionals.  Many  MS 
reported a lack of basic skills in web accessibility, within the PSBs as well as in the 
market.  The  problem  is  not  limited  to  the  availability  of  technical  experts  to  test 
accessibility; website design and graphic design sectors also need awareness raising 
and  remain  hard  to  reach  on  current  channels.  This  means  that  PSBs:  (i)  have 
difficulties in ensuring they have the necessary expertise to make their websites and 
mobile applications accessible; (ii) find it hard to procure accessible solutions; (iii) are 
unsure about how to formulate their requirements to the market and (iv) find it hard to 
procure  auditing  and  testing  services.  This  leads  to  two  further  problems:  (i)  it  is 
difficult for market players to deliver suitable websites or mobile applications; and (ii) 
many PSBs  lack the skills  to  check that the resulting services they have  bought  are 
compliant with the requirements of the WAD. 

46 

 
  Furthermore, there is a lack of accessibility experts who can support PSBs in the area 
of  testing  and  auditing.  This  gap  is  expected  to  be  partly  filled  by  an  increase  in 
knowledgeable market players in response to increasing demand. However, the pace 
of market growth has been slow so far. Therefore, the efforts already made by MS to 
launch training and capacity-building activities in the area of web accessibility should 
be strengthened (see below under ‘Training and support’). It would also be beneficial 
to  ensure  that  professional  certification  is  used  when  hiring  or  recruiting  web 
professionals. Forward-looking, increased offer of basic accessibility training for ICT 
(information  and  communication  technology)  curricula  in  universities  would  help 
further  address  the  shortage.  Moreover,  as  end  user  testing  is  encouraged  in  the 
monitoring methodology, there may be more opportunities for employment for persons 
with disabilities. Finally, the EU’s ambitious ‘Digital Decade’ target of increasing the 
number of ICT professionals from 8 to 20 million by 2030 offers a unique opportunity 
for web accessibility to be made a common component in all relevant digital courses. 

  Accessibility-related costs do not constitute a main barrier to implementing the 
WAD. Despite most PSBs stating that the WAD introduced additional economic costs, 
the evidence collected shows that accessibility related costs do not constitute a main 
barrier to implementing it. 

  Little  use  made  of  the  feedback  mechanism.  In  the  next  years  it  is  important  to 
continue to raise awareness publicly and ensure that this mechanism is enforced at the 
national level, giving all end-users the possibility to provide feedback. 

  Enforcement is not yet reaching its potential. Even though all MS have appointed 
their enforcement bodies and set up formal enforcement procedures, the effect of these 
bodies and procedures  seems  to  be limited so far. On other aspects  of enforcement, 
little  evidence  was  found  about:  (i)  starting  a  procedure  on  the  initiative  of  the 
enforcement body itself; (ii) reviewing disproportionate-burden exemption claims; or 
(iii) connecting enforcement measures to the monitoring activities. 

  Training  and  support  –  share  best  practice.  Most  MS  launched  training  and 
awareness-raising activities after the introduction of the WAD. However, the activities 
reported in monitoring reports are often described in a very broad and generic way, 
making  it  difficult  to  identify  and  exchange  best  practices  that  could  be  replicated. 
Given  that  these  activities  are  considered  one  of  the  main  success  factors  in  the 
implementation of the WAD, more information and more exchanges between MS on 
best practices may improve these instruments. 

The analysis of the monitoring reports reveals some challenges related to the monitoring 
exercise  itself.  Overall,  the  monitoring  and  reporting  mechanisms  under  the  WAD 
significantly limit the comparability of the results achieved in the MS. More specifically, 
the most relevant challenges could be identified in the bullet points below. 

  The  flexibility  of  the  monitoring  methodology  provided  in  the  Implementing 
Decision has been beneficial for MS with previous knowledge of web accessibility. 
However, in some cases it has paved the way to unintended consequences. 

47 

 
  Some MS do not involve end-users in the testing phase, whereas others rely solely on 
end-users to carry out the testing. In the former case, there is a high risk of missing 
important aspects of web accessibility (especially since the accessibility expertise of 
users is often lacking). In the latter case, results can be biased due to the abilities of 
testers (for example, blind users cannot check design-related aspects). 

  Some MS asked PSBs to conduct the in-depth monitoring on their own websites and 

applications without ensuring a clear process to validate the results. 

  Reporting on some aspects that this evaluation finds as key elements is optional under 
the  legislation  and  so  rarely  included  (e.g.  accessibility  statements,  feedback 
mechanisms, and use of the disproportionate-burden clause). 

 

In this context, the lack of an explicit obligation on the MS to consider the accessibility 
statement when monitoring a website or mobile application is not ideal. 

  There are different interpretations of the reporting and sampling requirements. 
Some MS have used the sample of websites initially chosen for in-depth monitoring in 
their simplified monitoring as well. They have usually done this as a quality-assurance 
process for the findings of the automatic tools used. This approach is consistent with 
the WAD, but as a result, the total number of websites monitored could be lower than 
expected,  and  therefore  fewer  PSBs  may  be  confronted  with  accessibility  issues 
detected.  Furthermore,  conformance  testing  tools  and  methods  do  not  necessarily 
produce comparable results due to differing interpretations of requirements. Some MS 
have reported the results from the different monitoring methods in an aggregated way, 
not  distinguishing  between  detecting  non-compliance  and  verifying  compliance 
(simplified and in-depth testing respectively). This obscures the results of the different 
methods and makes the exercise less useful. 

  As requested by several monitoring bodies consulted, one possible way to address these 
practical issues could be to launch joint work of the European Commission and MS on 
(i) a clear reporting structure to increase the coherence of the reporting among MS; 
and (ii) voluntary guidelines to orient MS in reporting on the accessibility statements, 
feedback mechanisms and use of the disproportionate-burden clause. This would make 
the monitoring  exercise  more efficient  for reporting  authorities and more consistent 
across countries. In addition, it would make it easier to compare results and measure 
progress over time. 

  Minimum requirements established by the WAD are still unclear at national level. 
Monitoring reports reveal a clear lack of awareness of the minimum requirements in 
some  cases.  In  the  interviews,  some  MS  claimed  to  be  unaware  of  the  minimum 
requirements  listed  in  Annex  A  of  the  Harmonised  European  Standard  EN 301 549 
(HEN). Other MS revealed that they focus only on WCAG 2.1 criteria and do so for 
three main reasons: (i) these are the ones covered by the chosen automatic tools; (ii) 
these criteria are the ones known by the experts; or (iii) these criteria are considered 
easier to handle. As a result, Clauses 5, 6, 7, and 12 of the HEN have sometimes been 
omitted from monitoring. The lack of knowledge persisted although the topic has been 
covered in WADEX meetings and that relevant information has been published on the 

48 

 
Europa.eu  portal.  More  clear  communication  at  EU  and  national  level  on  the 
monitoring requirements (which may further evolve) is required for future monitoring 
and reporting exercises. 

  MS  need  support  and  guidance  in  addressing  the  challenges,  gaps  and 
inconsistencies  identified  in  the  monitoring  exercise  and  in  the  supporting  study 
accompanying this evaluation. The guidance could take the form of written guidelines, 
workshops or other initiatives. This guidance could aim to increase the awareness and 
capabilities  of  MS  in  complying  with  their  obligations  under  the  WAD,  and 
strengthening correct and complete application of the WAD at all administrative levels. 
As ensuring accessibility is a process, strengthening correct and complete application 
of the WAD at national level will help achieve its objectives. Embedding accessibility 
in digitalisation and public communication strategies in public sector helps not only 
reach higher accessibility compliance rates, but also maintain it over time. 

  There are diverging reporting practices and differences in the interpretation of 
compliance  status.  Some  MS  have  not  always  followed  the  compliance-status 
categories  laid  down  in  the  Implementing  Decision,  whereas  others  drew  up 
compliance rules according to their own methodology. It is, therefore, not possible to 
compare  results  between  MS.  Additionally,  even  when  the  compliance  status  is 
calculated in line with the Implementation Decision, the reporting of ‘pass’ or ‘fail’ is 
not coherent among MS (as explained in section 4.1.1 above). So the results do not 
reflect the level of accessibility in a common and consistent way. 

  MS  would  need  support  and  guidance  on  uniform  definitions  of  the  various 
compliance statuses if the reporting process is to produce more useful and comparable 
results. 

  Finally, it should be noted that the MS that used assistive technology to complement 
their  testing  activities  used  almost  exclusively  screen  readers.  This  makes  the 
monitoring biased towards one of the nine user categories covered by the WAD and its 
implementing acts 194, namely users without vision. 

194  See, in particular, Implementing Decision (EU) 2018/1524, cited above (note 30), Annex I, point 1.3.2. 

49 

 
                                                 
ANNEX I. PROCEDURAL INFORMATION 

1.  Lead DG, Decide Planning references 

This evaluation was led by Directorate General Communication Networks, Content and 
Technology (DG CNECT), Unit G3 ‘Accessibility, Multilingualism and Safer Internet’. 

The DECIDE reference is: PLAN/2020/8486. 

2.  Organisation and timing 

On 2 September 2020, an Interservice Steering Group (ISG) 195 involving services with an 
interest in the measure under evaluation, was created to steer the evaluation in line with 
the ‘better regulation’ guidelines. 

The  ISG  was  consulted  over  the  course  of  the  evaluation,  in  conjunction  with  key 
milestones and deliverables, using various channels (virtual meetings, written exchanges, 
bilateral meetings, collaborative workspace). 

In October 2020, DG CNECT issued a request for services and terms of reference for a 
tender on external study supporting the review of the application of the Web Accessibility 
Directive. 

In November 2020, DG CNECT published the Evaluation Roadmap 196 for the initiative 
on  the  ‘Have  your  say’  portal,  describing  purpose  and  scope  of  the  evaluation.  The 
roadmap was open to public feedback for two months. 

The study contract was  signed on 25 January 2021, with task duration of 16 months. It 
began with a kick-off meeting on 12 February 2021 and ended in June 2022 (see Annex 
II). 

In June 2022, the ISG, as well as disability coordinators from additional services 197 were 
consulted on this draft staff working document. 

3.  Exceptions from the better regulation guidelines 

No exceptions from the usual procedural requirements of the ‘better regulation’ guidelines 
were required or applied to this evaluation. 

4.  Consultation of the Regulatory Scrutiny Board 

Not applicable. 

195  Ares(2020)4553282.  ISG  members:  SG,  SJ,  COMM,  DIGIT,  GROW,  EMPL,  JRC,  SANTE,  JUST, 

ESTAT, CNECT. 

196  Have your say portal, Published initiatives: Accessible web & digital content for people with disabilities 

– review of EU rules. 

197  Disability  coordinators  representing  COMP,  ECFIN,  EAC,  HOME,  REGIO,  TAXUD.  Disability 
coordinators have been nominated in all Commission DGs under the Strategy for the Rights of Persons 
with  Disabilities  2021-2030,  COM(2021)  101  final,  cited  above  (note  5).  See  also  its  Monitoring 
framework. 

50 

 
                                                 
5.  External expertise 

The evaluation was supported by an external consortium composed of PwC EU Services 
EESV (PwC), Open Evidence (OE) and, as a sub-contractor, Funka Nu AB (Funka). Due 
to corporate restructuring, one of PwC members, PricewaterhouseCoopers Public Sector 
Srl, changed its name to Intellera Consulting Srl (Intellera) in July 2021 and continued to 
service the contract as the lead contractor under the new name. 

The  consortium  collected  the  evidence,  performed  the  study  tasks,  and  submitted 
deliverables in accordance with the terms of reference. The final validation workshop on 
the findings and key results of the study was held on 20 May 2022 as part of the Global 
Accessibility Awareness Day and the European Month of Diversity 198. 

The draft Final report of the ‘Study supporting the review of the application of the Web 
Accessibility Directive (WAD)’ was submitted by the consortium on 15 May 2022. 

In addition, DG CNECT regularly consulted the Web Accessibility Directive Expert Group 
(WADEX) 199  composed  of  Member  State  experts  and  invited  relevant  stakeholders.  A 
total of 18 online meetings have been held in the period from January 2021 to June 2022 
covering  topics  relevant  for  the  review  of  the  WAD.  The  latest  meeting  discussing  the 
review findings was held on 28 June 2022. 

6.  Evidence, sources and quality 

Most of the evidence was collected with the support of an external contractor (see ‘Study 
supporting the review of the application of the Web Accessibility Directive (WAD),’ Final 
Report, PwC, Intellera Consulting, Open Evidence, Funka, September 2022). The study 
used  a  mixed-method  approach,  combining  both  primary  and  secondary  data-collection 
activities. 

The evaluation also considers evidence from Member State reports 200 on the outcome of 
monitoring and enforcement procedures under the WAD 201. 

The  evidence  is  based  on  external  expertise  and  knowledge  from  Member  States’ 
competent authorities, PSBs, international and umbrella organisations, independent bodies 
dealing with accessibility, NGOs and organisations representing persons with disabilities 
and older people, ICT (Information and Communications Technologies) industry players 
and  providers  of  accessibility  products  and  services,  academic  and  certification  bodies, 
accessibility experts, and citizens, especially persons with disabilities, and older people, 
through the numerous consultation activities held by the Commission and the contractor 
(see Annexes II and V). 

198  See Annex II, section 6. 
199  Register of Commission expert groups and other similar entities (E03475), and see above (note 32). See 
also its activities and meeting minutes on Web Accessibility Directive Expert Group (WADEX). 
200  Member State  monitoring reports are published on  national  websites and  made available also on the 

Commission’s website Web Accessibility Directive - Monitoring reports (4.01.2022). 

201  See above section 3.2 and Annex II, section 4. 

51 

 
                                                 
The quantification of the costs and benefits of the intervention was in some cases difficult, 
due to  unavailability of  data in  Member States (see section 4.1.10 and Annex  IV). The 
analysis of the effectiveness and efficiency criteria was thus limited, as described in Annex 
II, section 7. 

Nonetheless, it was possible to draft conclusions based on alternative proxy data and the 
triangulation  of  findings  from  the  literature  review  and  the  consultation  activities  with 
stakeholders (section 5). 

The overall evaluation is considered robust and thorough. 

52 

 
ANNEX II. METHODOLOGY AND ANALYTICAL MODELS USED 

The  evaluation  has  been  mainly  based  on  a  supporting  study  provided  by  an  external 
contractor.  The  study  adopted  a  mixed-method  approach,  combining  both  primary  and 
secondary data-collection activities. 

The  external  contractor  reviewed  and  analysed  information  available  in  the  literature, 
including academic papers and official reports (see section 1) as well as the first national 
monitoring  reports  shared  with  the  European  Commission  by  23  December  2021.  In 
addition,  the  contractor  set  up  a  consultation  strategy  to  involve  the  most  relevant 
stakeholder groups affected by the application of the Directive at the European and national 
level. The strategy planned the following activities: 

  an open consultation – namely the public consultation launched by the European 

Commission – held from July to October 2021 (section 2); 

 

 

four targeted surveys open between October and November 2021 (section 3); 

in-depth interviews conducted between January and February 2021 (section 4). 

1.  Literature review 

The literature review was based on a consolidated theoretical approach, also defined as 
scoping review. This kind of review attempts to provide an initial indication of the potential 
size  and  nature  of  the  available  literature  and  tends  to  privilege  breadth  over  depth  of 
coverage for a clearly defined specific focus. 

On geographical scope, the external contractor focused on Europe, although it also took 
into consideration as potential benchmark models papers from other countries that were 
deemed relevant. 

The selection of documents to be analysed was based on several criteria:  (i) number of 
references, (ii) type of publication, (iii) year of publication (within the last two decades) 
(iii) language (English-language was prioritised). Once a first group of relevant literature 
was defined, a snowball approach was used to extract further relevant documents. 

Overall, the literature review included not only academic papers, but also ‘grey’ literature 
identified online, such as such as policy documents, official reports, industry publications 
and statistics, working papers, published policy briefs or conference papers. As a result, 
the contractor revised over 60 documents. The Commission analysed further documents 
for the purposes of this staff working document (see bibliography in Annex VIII). 

2.  Public consultation 

The public consultation was open from 19 July to 25 October 2021. The aim of the public 
consultation  was  to  collect  views  and  experience  on  the  impact  of  the  Directive  on  the 
accessibility  of  public  sector  body  websites  and  mobile  applications  from  the  widest 
possible audience. It was based on one questionnaire with between 40 and 66 questions 
(depending  on  the  category  of  stakeholder  responding  to  the  questionnaire),  and  the 
questions were divided into two different sections: (i) general questions, meant for all the 

53 

 
respondents; and (ii) specific questions, meant for stakeholders with a knowledge of the 
Directive.  In  both  sections,  additional  questions  were  available  only  for  specific 
stakeholder groups. 

To complement the traditional format of EU survey, respondents also could reply to an 
easy-to-read version of the questionnaire. This easy-to-read questionnaire was comprised 
of 10 questions to give the public, especially those who may have found it difficult to reply 
to the long version, the opportunity to provide feedback on key aspects of the Directive 
(e.g. on their level of satisfaction when using public sector websites or mobile applications 
and on their use of the feedback mechanism). 

Before publishing the official questionnaire of the public consultation, 11 pilot interviews 
were conducted with different categories of key stakeholders to test the questions and see 
how accessible and easy to understand they were. 

Particular  attention  was  given  to  ensure  a  sufficient  geographical  coverage,  in  order  to 
avoid that the overall analysis resulted biased towards the opinion of respondents from few 
countries. This potential risk was firstly mitigated by involving key umbrella organisations 
at the European level. Moreover, targeted follow-up email or phone calls were made by 
the external contractor with relevant stakeholders in the least represented countries. Further 
information on the results of the public consultation is provided in Annex V. 

3.  Targeted surveys 

Targeted  surveys  were  conducted  on  four  key  stakeholder  groups  to  complement  the 
evidence gathered through the public consultation, by asking more specific questions to: 
(i) public sector bodies; (ii) people with disabilities, older people, and those taking care of 
people with disabilities and/or older people (formal and informal carer); (iii) Organisations 
representing people with disabilities (DPOs) and older people; and (iv) industry players 
and technology providers. 

These  stakeholder  groups  were  selected  in  order  to  cover  different  perspectives  on  the 
effects of the Directive: 

  people  with  disabilities,  older  people,  and  those  taking  care  of  people  with 
disabilities and/or older people (formal and informal carer), as well as organisations 
representing  people  with  disabilities  (DPOs)  and  older  people  were  asked  to 
provide  their  opinion  on  the  extent  to  which  the  Directive  (i)  has  affected  web 
accessibility of public information and services in EU Member States, and (ii) has 
helped  to  increase social and digital  inclusion of persons with  disabilities in  the 
EU; 

  Public sector bodies were asked to provide their opinion on the main costs and 
benefits brought by the Directive at different governance levels (national, regional 
and  local)  and  relevant  evidence  on  the  current  state  of  implementation  of  the 
Directive in the EU Member States; 

  Industry players and technology providers were asked to provide their opinion 
on the extent to which the Directive had an impact on the web accessibility market 

54 

 
in  the  EU  and  to  identify  technological  advancements  that  had  emerged  in  the 
European market since the Directive was applied 

These questionnaires comprised both open-ended and closed-ended questions. The surveys 
were developed in English only and remained open according to different timespans as set 
out in the table below. 

Table 1. Targeted survey audience and dates 

and 

technology 

Targeted survey audience 
Public sector bodies 
Industry 
players 
providers 
Organisations  representing  people  with 
disabilities, older people or consumers 
People with disabilities, older people, and 
taking  care  of  people  with 
people 
disabilities  and/or  older  people  (formal 
and informal carer) 

Start date 
18 October 2021 

End date 
30 November 2021 

18 October 2021 

30 November 2021 

18 October 2021 

3 December 2021 

10 November 2021  6 December 2021 

For  the  public  consultation,  particular  attention  was  given  to  ensure  a  sufficient 
geographical coverage, to avoid that the overall analysis results being biased towards the 
opinion of respondents from a few countries. This potential risk was firstly mitigated by 
involving key umbrella organisations at European level. In addition, the external contractor 
contacted  relevant  stakeholders  in  the  least  represented  countries  to  encourage 
participation.  Further  information  on  the  results  of  the  targeted  surveys  is  provided  in 
Annex V. 

4.  National Monitoring reports 

The external contractor also conducted a systematic analysis of national monitoring reports 
delivered to the Commission by the appointed reporting authorities of each Member State, 
in line with Article 8 of the WAD. Many Member States submitted the monitoring report 
to the Commission after the deadline set out in the Directive (i.e. 23 December 2021), and 
two have not  yet reported. Accordingly, 25 national reports were analysed, covering all 
EU Member States excluding France and Cyprus. 

All monitoring reports are publicly available in a dedicated webpage 202 and were provided 
in national languages. When an English version was not provided by the Member State, 
the  European  Commission  has  provided  a  courtesy  translation  into  English  using  its 
automated translation services. 

5.  In-depth interviews 

To  fill  data  gaps,  in-depth  interviews  were  conducted  to  complement  the  information 
collected  in  the  literature  review,  the  public  consultation,  the  targeted  surveys,  and  the 
national  monitoring  reports.  These  in-depth  interviews  covered  the  effectiveness, 
efficiency,  relevance,  coherence  and  EU  added  value  of  the  Directive.  Interviews  were 

202  Web Accessibility Directive - Monitoring reports (4.01.2022). 

55 

 
                                                 
carried out by videoconference and were based on a semi-structured questionnaire shared 
in advance with interviewees. 

Table 2. Number of interviews, by stakeholder group 
Stakeholder group 

Organisations of persons with disabilities (DPOs)  

Representatives of older people 

National monitoring, reporting or enforcement bodies 

Technology providers  

Industry players 

Standardisation bodies 

European Commission policy experts 

Total 

# 

1 

2 

28 

2 

1 

2 

2 

39 

Further information on the results of the in-depth interviews is provided in Annex V. 

6.  Final validation workshop 

The findings and recommendations that emerged from the supporting study were discussed 
in  a  final  workshop.  The  workshop  aimed  to  validate  the  key  results  presented  in  the 
supporting  study.  It  was  held  on  20  May  2022  as  part  of  the  11th  Global  Accessibility 
Awareness Day and the European Month of Diversity. 

Over  450  people  registered  for  the  event  with  participants  from  more  than  7  different 
stakeholders  groups  including  public  sector  bodies,  organisations  of  people  with 
disabilities,  accessibility  experts,  representatives  from  universities  or  research,  ICT 
generalists, private sector and non-governmental organisations (NGOs). There was also a 
good balance between participants from both EU (from 26 different Member States) and 
countries from outside of the EU (from 24 non-EU countries). The most represented non-
EU countries were: Norway, USA, U.K., Canada and Australia. 

7.  Methodological Limitations 

The interpretation of the evaluation findings presented in section 4 above should consider 
the following methodological challenges: 

  Limited  evidence  on  the  actual  impacts  of  the  Directive.  The  transposition 
process of the Directive at the national level was planned to close by 23 December 
2018. This means that the implementation of the Directive in Member States started 
in 2019, namely only two years ago. In addition, many Member States were not 
able to meet such deadline, and this means that the timeframe of implementation in 
some EU countries turned out to be even shorter. As a result, in some cases, limited 
evidence is available to assess the impact of the Directive at national level. 

56 

 
  Limited availability of data.  This  evaluation  comes at  a very early stage, with 
Member States sharing for the first time the monitoring reports on 23 December 
2021. Thus, the amount of data available is limited and difficult to compare. For 
example,  quantitative  figures  available  on  costs  experienced  by  stakeholders  is 
extremely limited. 

  Limited  comparability  of  monitoring  results.  National  Monitoring  Reports 
provided  by  Member  States  to  the  Commission  resulted  difficult  to  compare  in 
many  sections  as  sometimes  monitoring  results  were  reported  in  different  ways 
(depending  on  the  tools  used)  or  some  sections  were  not  filled  in,  leading  to 
incoherent outcomes among Member States. This may be due both to translations 
mistakes (Reports provided in non-English language were translated with English 
machine translation) or to different interpretations of the contents to be provided in 
some sections. 

  Geographical  distribution.  Overall,  the  amount  of  evidence  collected  in  some 
countries  is  considerably  higher  compared  to  other  countries.  Despite  the  great 
effort made by external supporting study team, the response rates for the different 
stakeholder consultation activities are uneven across countries. 

  Stakeholder  representativeness.  The  results  of  the  public  consultation  and  of 
targeted  surveys  shall  not  be  considered  as  statistically  representative  of  each 
stakeholder  group,  since  (i)  the  consultation  strategy  was  not  based  on  sample 
design techniques and (ii) some stakeholders engaged through dissemination did 
not reply to questionnaires or surveys. Therefore, the interpretation of the findings 
should consider that: 

o  In the long version of the public consultation, a limited number of replies 
was  received  from  human  rights  networks,  ombudspersons,  and  older 
from 
people  organisations,  while 
standardisation and certification bodies. 

replies  were  mainly 

received 

o  The  targeted  survey  for  individuals  with  disabilities,  older  people,  and 
people taking care of persons with disabilities and/or older people (formal 
and informal carers) gathered a few replies from carers (both in the case of 
older persons and in the case of persons with disabilities). 

o  The targeted survey for organisations representing persons with disabilities, 
older  people  or  consumers  gathered  evidence  mainly  from  organisations 
representing  persons  with  disabilities.  Consumer  organisations  and  older 
people provided a limited number of answers. 

o  The  targeted  survey  for  public  sector  bodies  covered  22  monitoring, 
reporting or enforcement bodies coming from 15 different countries. 

The issues listed above may limit the analysis especially in relation to the effectiveness 
and efficiency evaluation criteria. However, it was possible to draft conclusions based on 
the triangulation of findings from the literature review and the consultation activities with 
stakeholders. 

57 

 
ANNEX III. EVALUATION MATRIX 

The  table  below  summarises  the  evaluation  matrix,  which  is  structured  around  the  five 
evaluation  criteria  and  respective  evaluation  questions.  Indicators  used  to  address  each 
question were largely based on those developed in the SMART 2017/0068 study. Some 
relevant indicators were also added in order to address all the five evaluation criteria.

58 

 
Q1. To what extent 
have the objectives 
set out in the 
Directive been 
achieved, overall and 
regarding the 
specific obligations?  

Evaluation question 

Judgement criteria 

Indicators & descriptors 

Data sources 

EFFECTIVENESS 

Level of compliance with the minimum 
requirements of the Directive measured by: 

Main source:  
Member States monitoring reports 
(Task 4). 

Q1.1. To what extent has 
the Directive helped to 
increase the accessibility 
of public sector body 
websites and mobile 
applications? 

The Directive achieved its objective of 
making websites and mobile applications 
of public sector bodies more accessible 
(i.e. websites and mobile applications 
conform to the European EN 301 549 
standard). 

 

 

 

 

type of content (web, app, document, 
service etc.) 
level of government (national, 
regional, local) 
the POUR principles (perceivable, 
operable, understandable, robust) 
user situations (according to 
implementation acts). 

Q1.2. To what extent has 
the Directive contributed 
to putting in place 
procedures that ensure 
web accessibility? 

The Directive was transposed into 
national legislation, creating both a 
harmonised legislative framework and 
harmonised monitoring/enforcing 
procedures.  

  Level of transposition of the Directive 
  Set-up of monitoring and enforcement 

bodies in MS 

  Level of harmonisation in the 
monitoring methodology  

Q1.3. To what extent has 
the Directive contributed 
to improving the internal 
market? 

The Directive reduced the fragmentation 
of the European internal market for 
products and services related to web 
accessibility (i.e. products and services are 
available across different MS). 

  Use of the European EN 301 549 

standards in procurement in each MS 

  Number of companies offering web 
accessibility products and services 
selling across EU Member States 

59 

Other sources: 

  Targeted surveys 
 

In-depth interviews with public 
sector bodies, industry players, 
technology providers, and 
independent bodies  

  Desk research and policy 

review 

  Public consultation 
 

Interviews with competent 
authorities 

  Member States’ monitoring 

reports (Task 4)  

  Desk research 
  Public consultation 
  Targeted surveys 
 

In-depth interviews with public 
sector bodies, industry players, 
technology providers, and 
independent bodies 

 
 
 
 
Q.1.4 To what extent has 
the Directive contributed 
to the adoption of similar 
standards in the private 
sector? 

The minimum requirements set by the 
Directive are also used by private 
companies across the EU.  

  MS going beyond the minimum 

requirements and scope with national 
measures 

  Private companies’ compliance with 

  Desk research 
 

In-depth interviews with 
industry players and technology 
providers 

the minimum requirements 

Q2.1 What have been the 
success factors in the 
application of the 
Directive at MS level 
(monitoring agencies)? 

Q2.2 What have been the 
success factors in the 
application of the 
Directive at public sector 
level? 

Q2. What have been 
the success factors in 
the application of the 
Directive? How and 
why? 

List of two success factors as preliminary 
judgement criteria: 

  End-user groups have been involved 
in the selection process in each MS 

  The Web Accessibility Directive 

Expert Group (WADEX) supported 
the MS in the transposition of the 
Directive 

List of four success factors as preliminary 
judgement criteria: 

  End-users with disabilities have been 
involved in the selection of websites 
and applications to monitor 

  The feedback mechanism has been 
widely used by end-users with 
disabilities in a constructive way 

  The dual-model monitoring 

methodology (combination of in-
depth and simplified) has provided 
both qualitative and quantitative data 

  The facilitation of training and 

awareness raising in MS has been 
successful 

60 

  How many different end-user groups 

have been involved in the selection 
process in each MS? 

  Engagement of the WADEX in the 
transposition of the Directive and in 
the monitoring of methodologies 
adopted by MS 

 

In-depth interviews with public 
sector bodies, industry players, 
technology providers, and 
independent bodies (including 
independent bodies such as 
ombudsman, as well as 
standardisation and registration 
bodies) 

  Targeted surveys 
  Public consultation  

  Feedback received by users with 

disabilities to flag potential problems 
  The results of the dual monitoring (in-
depth and simplified) support MS 
planning of resources and focus 

  Public sector bodies, independent 
bodies and market players use 
supporting information provided by 
the monitoring authorities 

 

In-depth interviews with public 
sector bodies, independent 
bodies and bodies for 
standardisation/registration 

  Targeted surveys 
  Public consultation 

 
 
Q2.3 What have been the 
success factors in the 
application of the 
Directive at market level? 

The skills of the economic actors 
improved as a result of increased public 
sector body purchasing power.  

 

Increased skills in improving 
accessibility among economic actors 

  Further investments in further 

technical developments as a result of 
EU-wide technical specifications  

 

In-depth interviews with public 
sector bodies, industry players, 
technology providers, 
independent bodies, bodies for 
standardisation/registration 

  Targeted surveys 
  Public consultation 

Q.3. What have been 
possible gaps or 
challenges that have 
hindered 
achievement of 
objectives? Why? 

3.1 What have been 
possible gaps or 
challenges at the MS 
(monitoring agency) 
level? 

  Member States invested few 
resources for monitoring 

  MS agencies did not have enough 
skills for the monitoring activities 

  Challenges in the monitoring 

activities (e.g. selection of websites, 
methodology not implemented 
properly, wrong interpretation of 
requirements or technical 
specifications) 

  Other gaps coming from the market 
environment (e.g. market for 
automatic testing tools, few market 
players providing monitoring 
expertise) 

  Number of staff at the monitoring 

agency with expertise in the subject 
matter (or only external experts) 
  Level of investments by MS in the 

monitoring process 

  Number, quality and balance of 
websites monitored (how did the 
selection process work?) 

  Description of the monitoring set-up 

 

and testing tools used 
Involvement of the end-users with 
disabilities for the preparation of the 
monitoring activities 

  Market experts in monitoring hired in 

each MS 

  Member States’ monitoring 

reports 

  Targeted surveys 
 

In-depth interviews with public 
sector bodies, industry players, 
technology providers, and 
independent bodies 
  Public consultation. 

61 

 
 
3.2 What have been 
possible gaps or 
challenges at public sector 
body (PSB) level? 

  Not enough skills for testing 
  Few resources available for 

remediation 

  Other gaps coming from the market 
environment (e.g. in the market for 
automatic testing tools for internal 
audit, there are few market players 
providing audits, training and 
support) 

  Possible misunderstanding of the 

Directive (for example 
misunderstanding of the concept of 
undue burden) 

  Possible misinterpretation of 

requirements (for example intranets) 

  Possible misinterpretation of 

technical specifications (for example 
dual tagging of PDFs) 

Q.4. 
To what extent have 
stakeholders been 
actively engaged in 
the application of the 
Directive and how 
have they been 
affected? 

Q.4.1 Have stakeholders 
been actively engaged in 
the application of the 
Directive? 

Relevant stakeholders (i.e. DPOs, industry 
organisations) were consulted and 
engaged on a regular basis during the 
adoption of the Directive. 

Q4.1(a) In particular were 
the following stakeholders 
engaged? 

Yes/No for Q4.2(a) 

  People with 
disabilities 
  Representative 

organisations 

Q4.1(b) What were the 
main channels of 

  List of channels 
 

Judgement criteria is number of 
stakeholders reached by channel 

62 

  Whether the testing activities are 

conducted internally or with external 
experts 

  The automatic testing tools used 
 

Involvement of end-users with 
disabilities in testing 
  Training for internal staff 
  Wrong application of some 

requirements or technical 
specifications  

  Consultations with relevant 
stakeholders (and which 
stakeholders?) 

  Stakeholders involved in the 

implementation activities (and which 
stakeholders?) 

If yes, how many stakeholders were 
engaged for each group? 

  Self-reported assessment of channels’ 

effectiveness: 
  Very effective 
  Somewhat effective 
  Somewhat ineffective 
  Very ineffective 

  Targeted surveys 
 

In-depth interviews with public 
sector bodies, industry players, 
technology providers, and 
independent bodies 
  Public consultation 
  Note that in all these activities, 

respondents: will be asked to 
rate: (i) whether the gap they 
identified was large, medium or 
small; and (ii) whether they 
encountered unexpected effects 
and factors 

  Public consultation. 
  Targeted surveys 
 

In-depth interviews with public 
sector bodies, industry players, 
technology providers, and 
independent bodies 

  Public consultation 
  Targeted surveys 
 

In-depth interviews with public 
sector bodies, industry players, 
technology providers, and 
independent bodies 

 
 
 
 
 
 
 
 
engagement used and 
what channels worked 
best? 

Q.4.2 How have 
stakeholders (people with 
disabilities; representative 
organisations; web 
developers) been affected 
by the Directive? 

Relevant stakeholders (people with 
disabilities; representative organisations; 
industry players) were impacted by the 
Directive 

  More websites accessible for people 

 

with disabilities 
Increased involvement of 
representative organisations in the 
process 

Targeted survey and interviews with 
people with disabilities; 
representative organisations and 
industry players 

EFFICIENCY 

Q.5. To what extent 
has the application 
of the Directive been 
cost-efficient? 

Q.5.1 What have been the 
costs borne by the 
Member States for the 
monitoring activities? 

There have been relevant costs borne by 
the MS for the monitoring activities 

Q.5.2 What have been the 
costs borne by the public 
sector bodies to 
implement the Directive? 

There have been relevant costs borne by 
the public sector body for the 
implementation of the Directive  

  Cost of information obligation (i.e. 
reporting schemes for common 
monitoring and implementation) 
  Cost of monitoring activities (in terms 
of staff costs, communications to 
public sector body, reporting) 
  Cost of managing complaints 
  Other administrative costs  

  Compliance costs for making websites 
and mobile applications accessible 

  Costs for keeping accessibility 

statement updated 

  Costs for taking care of feedback 
  Costs for guaranteeing appropriately 

skilled staff 

  Costs to train content-creation staff 

and procurement staff 

  Desk research 
 

In-depth interviews with public 
sector bodies (monitoring 
bodies and national authorities) 
  Targeted surveys to monitoring 

bodies 

  Desk research 
 

In-depth interviews with public 
sector bodies (monitoring 
bodies and national authorities) 

  Targeted surveys 

63 

 
Q.5.3 What have been the 
cost savings for the public 
sector bodies as a result of 
the implementation of the 
Directive? 

There have been relevant cost savings by 
public sector bodies as a result of the 
implementation of the Directive 

  Reduced expenditure due to the 

transition from individually adopted 
interactions with citizens with 
disabilities and the elderly to 
mainstream online services 

  Reduced expenditure as specialised 

 

support is reduced 
Jobs created for the development, 
maintenance and monitoring of 
websites and mobile applications 
  Lower prices as a result of increased 

competition 

  Better quality of life as a result of 

improved accessibility 

  Desk research 
 

In-depth interviews with public 
sector bodies (monitoring 
bodies and national authorities) 

  Targeted surveys 

Q.5.4 What have been the 
costs for industry players 
related to the application 
of the Directive? 

There have been relevant costs for 
industry players due to the 
implementation of the Directive  

  Compliance costs 
  Costs related to capacity building 
  Costs related to necessary new 

investments  

  Desk research 
 

In-depth interviews with 
industry players and technology 
providers 

  Targeted surveys  

Q.5.5 What have been the 
monetary benefits for 
industry players related to 
the application of the 
Directive? 

There have been relevant benefits for 
industry players due to the 
implementation of the Directive 

 

Increased demand for 
products/services 
 
Increased market opportunities 
  Decreased research and innovation 

costs 

  Desk research 
 

In-depth interviews with 
industry players and technology 
providers 

  Targeted surveys 

Q.6. To what extent 
were the resources 
(and especially EU 
funding) across the 

Q.6.1. What have been the 
administrative costs borne 
by the EC for the adoption 
of the Directive? 

The resources provided by the European 
Commission across the areas of action 
were adequate and proportionate. 

  Costs of setting up the Web 

Accessibility Directive Expert group 
(WADEX) 

Interviews with high-level EC 
policy experts  

64 

 
areas of action at the 
EU level adequate 
and proportionate? 

  Costs of organising meetings and other 

activities of the WADEX group 

Q.6.2. What has the EU 
funded across the areas of 
action at the EU level? 

The EU funded a number of research 
activities across the areas of action of the 
Directive.  

  Research projects funded (for example 
the 7 ones referred to in the ToR) 

Q.6.3. What are the 
potential returns of the 
investments supported by 
EU funding? 

How many projects (either research 
projects funded by the EU or other more 
applied projects funded by the EU or other 
sources) have been launched to support 
the application of the WAD? Do they 
address the most necessary areas/impacts? 

  How many projects have been funded 
to support the application of the 
WAD? 

  Do they address the most necessary 

areas?  

RELEVANCE 

 

 

Interviews with high-level EC 
policy experts 
Interviews with EC officials 
and research organisations 
involved 

 

Interviews with high-level EC 
policy experts 

Q.7.1. Have the objectives 
of the Directive been 
relevant considering the 
evolving policy context? 

The Directive has been complementary to 
other policy initiatives to achieve its 
broader objectives.  

Relation of the Directive to national, EU 
and international policy initiatives: 

  Procurement Directive and European 

Accessibility Act 

  The Electronic Communications Code 
 
the Audiovisual Media Services 
Directive, 
the Marrakesh Treaty 
the UNCRPD and the EU disability 
strategy 

 
 

Desk research: review and analysis 
of other digital-accessibility-related 
policy and legislation at national 
and EU level  

Q.7.2. Have the objectives 
of the Directive been 
relevant considering the 
new developments in key 
markets? 

The Directive is still relevant today 
considering the new market 
developments.  

  New technologies not included in the 

scope of the Directive 

  Desk research (Task 2) 
 

In-depth interviews with 
technology providers 

Q.7. To what extent 
have the Directive 
and its objectives 
been instrumental, 
and do they continue 
to be relevant, in 
addressing the needs 
of users, in particular 
persons with 
disabilities and other 
users with functional 
limitations 
(including older 
people), considering 
the evolving policy 
context (including 
other European 
legislation related to 
digital accessibility) 

65 

 
 
and new key market 
developments 
(including new 
digital interfaces)? 
Q. 8. To what degree 
is the Directive 
relevant for its 
different 
stakeholders, 
including the general 
population in 
Europe? How does 
the Directive 
contribute to digital 
and social inclusion 
and the participation 
of users, in particular 
persons with 
disabilities and older 
people? 

Q.8.1. Is the Directive 
relevant for different 
stakeholders? 

The Directive is relevant for its different 
stakeholders (end-users, public sector 
bodies, industry) as well as the general 
population 

  Level of awareness of the different 

objectives of the Directive among the 
different stakeholders 

  Level of approval of the main points of 

the Directive among different 
stakeholders 

  Public consultation 
 

In-depth interviews with public 
sector bodies, industry players, 
technology providers and 
independent bodies  

Q.8.2 Has the Directive 
improved the participation 
of end-users? 

The Directive helped to increase the 
participation of end-users with disabilities 

  End-users’ satisfaction 
  End-users’ participation/use of public 
sector body websites and mobile 
applications 

  Targeted surveys (as Q.7.1) 
 
  Public consultation  

In-depth interviews with DPOs 

COHERENCE 

  For the WAD objectives and specific 

Q. 9. To what extent is the Directive coherent 
internally? Are there any incoherent parts of the 
Directive in terms of its goals and provisions? 

The different obligations and mechanisms 
of the Directive work well together to 
achieve the main objectives  

o 

o 

66 

obligations: 
o 

they are consistent/are not 
consistent with each other (in 
identified ways); 
there is/is not overlap 
identified (in identified 
ways); 
there is/is not conflicting 
goals and MS public sector 
body obligations; concepts: 
A: perceivable; B: operable; 
C: understandable; D: robust. 

Desk research of the Directive, 
implementation acts and any other 
documents attached. 

Interviews with experts (high-level 
European policy experts and web 
accessibility experts). 

 
 
Q.10. To what extent is the Directive coherent 
with other relevant EU policies and/or actions, 
notably the Audiovisual Media Services Directive, 
the European Accessibility Act, the European 
Electronic Communications Code, and other 
relevant legislation? To what extent has digital 
accessibility been mainstreamed in those 
policies/actions? To what extent is the Directive 
coherent with other relevant EU policies and/or 
actions in related policy areas? 

The Directive is coherent with other 
relevant EU policies and actions. In 
addition, the issue of digital accessibility 
has been mainstreamed in those 
policies/actions. 

Q.11. To what extent is the Directive coherent 
with the UNCRPD, the EU Charter of 
Fundamental Rights, the European Pillar of Social 
Rights and measures addressing digital/social 
inclusion of older people? 

The Directive is coherent with the 
UNCRPD, the EU Charter of 
Fundamental Rights and the European 
Pillar of Social Rights, and it adopted the 
key definitions and concepts contained in 
those policies and measures  

Q.12. To what extent have the EU measures and 
initiatives been coherent with related policy 
measures in Member States? Are there any 
national policies that go beyond the Directive in 
improving access to digital public information and 
digital public services for users? 

The Directive is coherent with other 
related policy measures in Member States 

  Synergies between the Directive and 

wider EU policy 

  Other EU legislation where web 
accessibility is considered 
(Audiovisual Media Services 
Directive, the European Accessibility 
Act, the European Electronic 
Communications Code) 

  Overlaps between the Directive and 

other EU interventions 

  Convergence of the objectives 

between the Directive and the other 
international policies/measures 
  Synergies with the other policies in 

terms of definitions used, and concepts 
and principles adopted 

  Overlaps between the Directive and 
the provisions of policy measures 
already existing in Member States 
  Synergies with other actions taken by 
Member States to improve web 
accessibility 

  Desk research of the other key 
EU policies and actions listed. 
Interviews with experts (high-
level European policy experts 
and web accessibility experts) 

 

  Desk research of the other key 
international policies and 
measures listed. 

  Use of the DARE index made 
by the UN initiative G3ict and 
their DPO representatives in all 
EU MS.  

  Desk research of the policy 
measures in all the Member 
States, or previous EC reports 
on the matter. 
In-depth interviews with public 
sector bodies (monitoring 
bodies and national authorities) 

 

EU ADDED VALUE 

67 

 
Q.13. What is the added value of the Directive 
compared to what is likely to have been achieved 
both at the EU (including by institutions) and 
Member State levels in the absence of the 
Directive to remove barriers across the internal 
market of web accessibility? 

The Directive generates additional value 
compared to what could have resulted 
from national interventions to remove 
barriers across the internal market 

  Level of legislative harmonisation 

across the EU compared to baseline 
scenario (with no EU intervention) 

  Changes in national laws on 

accessibility requirements after the 
adoption of the Directive 

  Activities of the Web Accessibility 
Expert Directive Group (WADEX) 

  Training activities organised by 

Member States and other stakeholders 
  Market perspective: more companies 

to invest in accessibility innovation 
with a much bigger market 

Q.14. What is the added value of the Directive in 
improving access to digital public information and 
digital public services for users, in particular 
people with disabilities and older people? 

The Directive generates additional value 
compared to what could have resulted 
from national interventions to improve 
digital access for people with disabilities 
and older people 

  More websites compliant with 

 

requirements 
Increased levels of accessibility of 
websites and mobile applications, as a 
result of the Directive 

  Desk research 
 

Interviews with relevant 
stakeholders and expert 
organisations 
In-depth interviews with public 
sector bodies (monitoring 
bodies and national authorities) 

 

  Desk research of the policy 
measures taken in all the 
Member States, data on the new 
businesses operating cross-
country, and reports from the 
WADEX 

 

  Consultation with relevant 
stakeholders and expert 
organisations 
In-depth interviews with DPOs 
and representatives of older 
people 
In-depth interviews with web 
accessibility experts 

 

Q.15. What is the added value of the Directive in 
implementing the relevant provisions of the 
UNCRPD, the EU Charter of Fundamental Rights 
and the European Pillar of Social Rights? 

The Directive generates additional value 
compared to what could have resulted 
from national interventions to implement 
the relevant provisions of the UNCRPD, 
the EU Charter of Fundamental Rights 
and the European Pillar of Social Rights 

  Level of legislative harmonisation 

across the EU compared to baseline 
scenario (with no EU intervention) 
Integration of the UNCRPD provisions 
into national laws 

 

  Desk research of the policy 
measures taken in all the 
Member States, data on the new 
businesses operating cross-
country 

  Consultation with relevant 
stakeholders and expert 
organisations 

68 

 
 
  Progress in achieving the specific 
objectives of the EPSR related to 
disability (Principle 17).  

 

 

In-depth interviews with public 
sector bodies (monitoring 
bodies and national authorities) 
In-depth interviews with web 
accessibility experts 

69 

 
 
ANNEX IV. OVERVIEW OF BENEFITS AND COSTS 

The table below summarises the costs and benefits of the implementation of the Directive. 
A discussion of the efficiency of the Directive, as well as of the limitations of the data 
collection related to the costs, has been presented in section 4.1 of the evaluation report. 
Information in this annex is based on interviews and surveys, extracted from the supporting 
study, and only reflects the views of the authors of the supporting study.

70 

 
Overview of costs and benefits identified in the evaluation 

Citizens/Consumers  

Businesses 

Administrations 

Quantitative   Comment 

Quantitative  

Comment 

Quantitative 

Comment  

COST: Improving accessibility of public sector body websites and mobile applications 

N/A 

N/A 

Not available 

Making  websites  and  mobile 
applications 
compliant  with 
WAD  minimum  requirements. 
Main costs reported: 
a) cost for external services and 
suppliers 
improve 
to 
accessibility;  b)  direct  labour 
costs from training personnel and 
hiring  new  skilled  staff;  c) 
implementation  costs  for 
the 
creation 
accessibility 
of 
statements. 
Costs are lower for MS with pre-
existing 
accessibility 
measures. 

web 

Lower – but recurrent – costs are 
to be considered by public sector 
bodies  to  update  websites  and 
mobile applications.  

COST: Maintenance and update of websites and mobile applications 

N/A 

Not available 

COST: Monitoring and enforcement activities 

N/A 

Overall 
monitoring  costs 
each  MS 
for 
the 
depend  on 

for  monitoring 
Major  costs 
include:  automated 
activities 
testing 
assistive 
tools, 
technologies,  external  experts, 

71 

Direct 
compliance 
costs 

one-off  

Direct 
compliance 
costs 

recurrent 

Enforcement 
costs 

recurrent 

N/A 

N/A 

 
 
 
 
 
 
 
 
staff, 

internal 
user-testing, 
reporting  costs,  and  publication 
of information. 
Other  costs  for  enforcement 
activities: 
an 
setting 
enforcement  body,  handling 
complaints. 

up 

used, 

tools 
personnel 
and 
involved, 
experts 
external 
Costs 
hired. 
from 
ranged 
EUR 65 000 
to 
EUR 650 000  for 
first 
the 
monitoring  period 
by MS. 

COST: Administrative burden 

N/A 

Not available 

Indirect costs 

one-off 

Updating  products  and  services 
to  meet  the  new  accessibility 
requirements.  Also  hiring  new 
specialists  or  training  personnel 
to comply  with WAD  minimum 
requirements. 

Not available 

Training  and  awareness-raising 
activities,  mainly 
targeting 
public sector body personnel.  

BENEFIT: Increased economic and social participation of citizens 

N/A 

N/A 

Direct benefit 

recurrent 

Not available 

access 

Improved 
to 
essential  online  public 
services,  saving  costs  and 
time  by  accomplishing 
digitally 
tasks 
autonomously 
(without 
support personnel, carers). 
Overall,  better  quality  of 
for  persons  with 
life 
disabilities. 

Indirect 
benefit 

recurrent 

Not available  More job opportunities as a 
consequence  of  market 

Not available 

Increased  demand  for  products 
and services both in national and 

Not available 

More  competitive  offers  and 
lower prices on the market  

BENEFIT: Increased size of market and cost reduction 

72 

 
 
 
 
growth, greater demand for 
accessibility  products  and 
services.  

in  other  European  markets,  due 
to  lower  market  barriers  and 
harmonised standards  

BENEFIT: Reduced costs for public sector 

N/A 

N/A 

Not available 

Indirect 
benefit 

recurrent 

Decreasing 
providing 
alternatives 
services. 

staff 

costs 
for 
face-to-face 
to  online  public 

  Fewer  complaints  about 
inaccessible public services. 

 

reputation 

Improved 
of 
public  sector  bodies  and 
governments. 

73 

 
 
 
ANNEX V. STAKEHOLDER CONSULTATION – SYNOPSIS REPORT 

1.  Introduction 

This document summarises the consultation activities held by the European Commission 
for the review of the Directive (EU) 2016/2102 203 (Web Accessibility Directive or WAD). 
The consultations aimed at collecting evidence from different stakeholders to evaluate the 
implementation of the Directive. 

2.  Consultation activities 

The  consultation  activities  were  conducted  between  July  2021  and  February  2022, 
including: 

  A public consultation from July to October 2021 204; 

  Four targeted surveys between October and December 2021; 

  39 in-depth interviews conducted with relevant stakeholders between January and 

February 2022 205. 

The public  consultation  and the targeted surveys were conducted online 206, and widely 
disseminated among stakeholders. 

The  overall  consultation  strategy  was  constantly  updated  to  fill  gaps  emerging  both  in 
terms of geographical coverage and representativeness of stakeholder groups. The in-depth 
interviews were conducted using videoconference platforms (Microsoft Teams and Zoom). 

2.1 

Public consultation 

The aim of the public consultation was to collect evidence on the impact of the Directive 
and its implementing acts, especially to what extent these: 

1)  have  made  it  easier  for  people  with  disabilities  to  access  public  services  and 

information, strengthening social and digital inclusion; 

2)  are  still  relevant  and  fit  for  purpose,  considering  related  laws  and  changes  in 

technology; and 

3)  have harmonised the web accessibility market. 

The standard survey  included from  40 to  66 questions, depending  on stakeholder type, 
structured around the five formal evaluation criteria: effectiveness, efficiency, relevance, 
coherence, and EU added value of the Directive. For the first time, respondents also had 
the opportunity to  answer to  an easy-to-read version  of the questionnaire, not  requiring 
log-in  (hereinafter  also  referred  to  as  the  “easy-to-read  survey”).  The  questionnaire 

203  Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the 
accessibility of the websites and mobile applications of public sector bodies (OJ L 327, 2.12.2016). 
204  Summary report of the public consultation, with survey data is published online, Have your say portal, 
Published  initiatives:  Accessible  web  &  digital  content  for  people  with  disabilities  –  review  of  EU 
rules/public consultation. 

205  See list of interviewees in section 4 of this Annex below. 
206  EU Survey tool. 

74 

 
                                                 
included 10 questions and offered a simpler and more accessible way to give feedback on 
key  aspects  of  the  Directive  (e.g.,  on  the  level  of  satisfaction  when  using  public  sector 
websites). 

Overall, the two versions of the public consultation received 967 replies from stakeholders 
and end-users. The easy-to-read survey attracted more than twice as many responses (656) 
as the standard survey (311). 

Number of responses per survey

Easy-to-read survey
656 (68%)

Standard survey
311 (32%)

0

200

400

600

800

1000

Source: Public consultation 

Replies were received from 32 countries – all EU Member States, EFTA/EEA countries 
(Norway, Switzerland, Liechtenstein), as well as the United Kingdom and Albania. The 
largest number of responses were submitted from Sweden, Italy, Netherlands, Finland, and 
Germany. 

This distribution is important when analysing the results, as 60% of the responses come 
from five countries where web accessibility policies have been in place long before the 
adoption of the WAD. 18% of the responses come from Sweden 207. 

207  It is important to note that responses in the public consultation as well as the targeted surveys represent 
more  opinions  from  countries  where  web  accessibility  policies  were  in  place  before  the  adoption  of 
WAD. This imbalance of the statistical data must be taken into consideration when analysing the results. 

75 

 
                                                 
82

34 18

32

27

97

89

75

67 66

11

1

34 28

Number of replies per country

Easy-to-read survey (n=656)

Standard survey (n=311)

8

4

5
6

5
6

3
7

1

8

0

8

3
5

4
4

4
2

0
5

2
2

10

26

19 10 4

4

2

6
6
8 14 19 17 15 17 11 10 6

4

6

Source: Public consultation 

Considering the standard survey, the most represented stakeholder  groups were citizens 
(78) and public sector bodies (65), accounting for 46% of the total respondents, followed 
by  a  relatively  high  number  of  web  accessibility  experts  (55)  and  organisations 
representing persons with disabilities (40). 

Respondent categories (standard survey only)

Citizen
Public sector body
Web Accessibility Expert
Disabled persons organisation
Other
Provider of ICT systems
National monitoring and reporting body
Institution providing web accessibility training
Provider of solutions for accessibility services
Industry organisation
Human rights network
Ombudsperson
Older people organisation

Source: Public consultation 

25%

21%

18%

13%

7%

5%

3%

3%

3%

1%

1%

1%

0,3%

Standard survey (n=311)

Persons with disabilities were especially targeted in this consultation, as among citizens 
they are the main beneficiaries of the web accessibility directive. Nearly 40% (354 out of 
967) of all respondents  considered themselves as a person with a disability or having a 
recognised disability. Among them, about 80% responded to the easy-to-read survey (278 
out of 354), showing that the easy-to-read survey could capture views which the standard 
survey alone probably would have missed. 

76 

 
Respondents in relation to disability

Easy-to-read survey (n=656)

278

71

282

25

Standard survey (n=311)

76

24

161

14

36

0

100

200

300

400

500

600

700

I consider myself a person with disabilities or have a recognized disability

I'm responsible for caring for a person with disabilities

I do not consider myself a person with disabilities

I prefer not to say

Other

Source: Public consultation 

2.2 

Targeted surveys 

The additional, targeted surveys focused on four different categories of stakeholders: (i) 
Public  sector  bodies,  (ii)  Industry  players  and  technology  providers,  (iii)  Organisations 
representing  persons  with  disabilities,  older  people  or  consumers,  (iv)  Individuals  with 
disabilities, older people, and people taking care of persons with disabilities and/or older 
people. The questionnaires included a mix of open and closed questions, addressing all the 
evaluation criteria, and were conducted between October and December 2021. 

2.2.1  Public sector bodies 

The sample of public sector bodies consisted of 62 respondents, representing mostly public 
authorities  (50%)  and  monitoring,  reporting  or  enforcement  bodies  (35%).  The  most 
represented countries were Sweden, (22%) Germany (13%) and Poland (10%). 

2.2.2  Industry players and technology providers 

The survey targeting industries received 52 responses, with different types of expertise. In 
particular,  44%  of  the  respondents  worked  as  expert  consultants  in  topics  related  to 
accessibility, 11% were ICT system providers, 10% worked in institutions providing web 
accessibility  training.  In  terms  of  size  of  the  organizations,  the  sample  represented  a 
heterogeneous group of companies, from small to large enterprises. The most represented 
countries  in  this  area  were  the  Netherlands,  Sweden,  Denmark,  Finland  and  Belgium, 
representing roughly 50% of the sample. 

2.2.3  Organisations  representing  persons  with  disabilities,  older  people  or 

consumers 

The  sample  of  NGOs  consisted  of  48  respondents,  mostly  working  in  organizations 
representing people with disabilities (88%). In addition, a small share of the sample (8%) 
represented consumer organizations and other bodies (4%). The most common disabilities 
of the people represented by the organizations were with visual or hearing impairments, 
speech impairments,  mental/intellectual disabilities and neurological  impairments.  Most 
of  the  organizations  operated  at  the  national  level  (65%)  while  a  minority  worked  at 

77 

 
regional (6%) or local (13%) level. A small proportion of organisations (16%) reported 
working internationally. The most represented countries in this area were Sweden (20%), 
Belgium (10%) and Slovenia (8%). 

2.2.4  Persons with disabilities, older people, and people taking care of persons with 

disabilities and/or older people 

Overall, 40 people took part in the survey. Sweden and Italy were the most represented 
countries, with 38% and 15% of respondents respectively. 22 people reported having some 
form  of  disability.  11  people  reported  living  with  people  with  disabilities,  16  people 
worked on accessibility issues and 12 were part of organizations representing people with 
disabilities. 

2.3 

In-depth interviews 

39 interviews were conducted. The in-depth interviews included 28 authorities covering 
21 Member States, as not all accepted the interview or replied to the request of interview, 
and other relevant stakeholder groups. The Table below shows the number of interviewees 
per stakeholder group. List of interviewees is annexed in Tables 3 and 4 in this Annex V, 
section 4 below. 

Table 2. Number of interviews, by stakeholder group 
Stakeholder group 
Organisations representing persons with disabilities (DPOs)  
Representatives of organisations of older people 
Monitoring, reporting or enforcement bodies 
Technology providers 
Industry associations 
Standardisation bodies 
European Commission policy experts 
Total 

3.  Main results of the consultations 

3.1 

Use of online public services 

# 
1 
2 
28 
2 
1 
2 
2 
39 

The  initial  responses  to  the  standard  survey  of  the  public  consultation  aimed  to  gather 
views  on  the  impact  of  the  Directive  on  (i)  access  to  online  public  services,  (ii)  users’ 
experience when dealing with such services and on (iii) knowledge and use of the feedback 
and  complaints  mechanisms.  With  regard  to  these  three  aspects,  evidence  was  also 
collected from the easy-to-read survey, the results of which are explicitly mentioned when 
relevant. 

3.1.1  Access to public services 

The open public consultation investigated the kind of public services accessed online, and 
the  ways  used  to  access  them,  through  a  specific  sub-set  of  questions  targeted  only  to 
respondents  identifying  themselves  as  citizens,  members  of  organisations  representing 

78 

 
persons  with  disabilities  or  older  people,  web  accessibility  experts  or  in  the  residual 
category other. A total of 198 responses were submitted by respondents in these categories. 

The majority of respondents (187 out of 198) declared to make a frequent use of domestic 
online public services, ranging from 39% of weekly users, to a 33% of daily users and a 
22% of monthly users. Percentages are significantly lower when considering the use of 
public services of another EU country, given that 71% of respondents (142 out of 198) 
replied to use such services less than once a month (41%) or never (30%). With regard to 
the kind of public services used online, health services are the most accessed (150 out of 
198), followed by money and tax services (106), utilities services (99), benefits services 
(92) and employment services (86). 

The results of the open consultation reveal that when accessing public services online, the 
most  important  accessibility  issues  were  experienced  with  navigation  and  forms 
(reported by 50% of the 198 participants replying to this question) and issues related with 
image and text contents (40%). This is partially confirmed by the responses received to the 
easy-to-read survey, where navigation (41%) and forms (17%) are still the ones marked as 
most difficult to use. 

However,  individuals,  persons  with  disability  (40  respondents)  and  NGOs  (48 
respondents) in a targeted survey rated overall accessibility levels of online public services 
slightly more insufficient at the regional (30% individuals, 42% NGOs) and local level 
(27,5% individuals, 46% NGOs) than on national level (25% individuals, 31% NGOs). 

3.1.2  User experience 

User experience was investigated with the same sub-sample of 198 participants (citizens, 
members  of  organisations  representing  persons  with  disabilities  or  older  people,  web 
accessibility  experts  or  others)  in  the  public  consultation.  A  large  majority  affirms  that 
their use of online public service increased slightly (42%) or significantly (37%) in the last 
three years. 

Regarding  ease  of  use,  even  though  more  than  57,7%  of  individuals  that  replied  to  the 
easy-to-read  survey  (379  out  of  656)  reported  that  government  websites  have  become 
easier to use in the last three years; but when we look at the percentages of individuals 
finding government websites more difficult to use, it is almost three times higher among 
respondents  identifying  themselves  as  persons  with  a  disability  or  with  a  recognised 
disability or impairment (almost 12% of both categories) than among those declaring to be 
persons without a disability (only the 4% of this category). 

3.1.3  Feedback mechanism and complaints 

The same sub-sample of 198 participants of the public consultation (i.e., citizens, NGOs, 
accessibility  experts  and  others)  revealed  a  significant  level  of  awareness  about  the 
feedback mechanism (73%) and the possibility to move a complaint to an enforcement 
body (71%). 

79 

 
However,  a  significant  difference  emerges  among  the  different  stakeholder  groups. 
Awareness is very high among web accessibility experts (around 90% of them are aware 
of both feedback and complaints) and members of organisations representing persons with 
disabilities or older people (more than 70%) but is considerably lower among citizens 
(around 45% not aware). Similarly, in the easy-to-read survey, only 39% of the participants 
(255 out of 656) know how to complain to the government when an accessibility problem 
is not fixed. 

Moreover,  the  feedback  mechanism  is  still  not  widely  used  when  accessibility  issues 
arise. 44% of respondents (86 out of 198) have never given feedback to a public sector 
body. Among them, 46 respondents added a comment in the public consultation, which 
shows  a  high  engagement  rate.  The  three  more  common  reasons  were  (i)  users  believe 
feedback do not have an effect, (ii) users did not know how to give feedback, and (iii) users 
did not have time to do so. This evidence is reinforced by the results of the easy-to-read 
survey,  where  70%  of  individuals  (457  out  of  656)  have  never  used  the  feedback 
mechanism. These results are confirmed by the findings of the in-depth interviews with 
the monitoring bodies, stating that the feedback mechanism was considered useful, but still 
not much used. One of the reasons provided was the low level of awareness among end 
users, which may be linked to the availability of accessibility statements, as many PSBs 
failed to add the statement which contains the information on how to provide feedback. 
Furthermore, among those respondents who used the feedback, the response received by 
the PSB was regarded in most cases as only partially satisfying (64%), and few respondents 
claim to be satisfied from the reply (13%). Similarly, 60% of individuals from the easy-to-
read survey claim not to be happy with the answer to their feedback. 

Similarly, the right to launch complaints has not yet been widely used. In the standard 
survey,  73%  of  respondents  (145  out  of  198)  never  complained  to  an  ombudsman,  a 
monitoring body or a human-rights network in their country. Among the reasons reported 
were the perceived uselessness of the mechanism and the inaccessibility of the complaint 
forms due to captcha. The percentage of users who never complained gets to 88% (576 out 
of  656)  in  the  easy-to-read  survey.  Furthermore,  few  individuals  (11  out  of  40) 
participating  in  the  targeted  survey  have  complained  to  an  enforcement  body.  The 
experiences  reported  by  those  individuals  were  mostly  negative,  as  enforcement  bodies 
have  not  always  given  a  satisfactory  response  and  were  considered  not  aware  of  the 
mechanism. In case of complaints made, results differ between the standard version and 
the  easy-to-read  version.  In  the  former,  81%  of  respondents  claimed  to  be  satisfied  or 
partially satisfied with the response received. In the latter, 64% of individuals (51 out of 
80 who complained) were not happy of how the complaint was managed. 

3.2 

Scope and standards of the Directive 

Almost half of the respondents (49%) of the standard survey (151 out of 311) agreed that 
the  Directive  adequately  covers  online  public  services  to  ensure  full  participation  of 
people  with  disabilities  in  digital  society.  Regarding  subject  and  content  types  not 
covered by the Directive (multiple choice), the majority of respondents favoured removing 
the current exemptions and/or expanding the scope: full inclusion of universities (81%), 

80 

 
schools (70%), NGOs (56%), online maps (57%), live videos (53%), extranets/intranets 
(51%) and third-party content (47%) in view of technological advances. 

Respondents identified the following top five technological advancements as relevant for 
the  Directive:  ‘artificial  intelligence  driven  assistive  technologies’  (60%),  ‘accessible 
authoring tools’ (57%), ‘artificial intelligence and machine learning applied to monitoring, 
testing tools’ (43%), ‘Internet of Things (IoT) for accessible products and services’ (42%), 
‘biometrics for identification and security aspects’ (31%). 

Concerning  the  Harmonised  European  Standard  EN  301  549  (v2.1.2),  many 
respondents  (63%)  consider  the  standard  useful  for  making  public  sector  websites 
accessible.  This  view  drops  to  48%  for  mobile  applications,  and  25%  for  cross-border 
service offering. Around one third of respondents (32%) agree that the standard still covers 
all relevant end-user groups, with half (51%) not sure or neutral on the matter, and 17% 
disagreeing with the statement. 

3.3 

Evaluation of the Directive 

Each  consultation  was  structured  around  the  five  evaluation  criteria.  The  analysis 
presented here reflects main results from the public consultation, the targeted surveys and 
the in-depth interviews according to each criterion. 

3.3.1  Effectiveness 

Increasing web accessibility 

Regarding the first objective of the Directive (i.e., to increase accessibility of websites 
and mobile applications of public sector bodies), the consultations provided a composite 
picture.  66%  of  respondents  (207  out  of  311)  of  the  standard  survey  considered  the 
Directive  successful,  at  least  to  some  extent,  in  making  online  public  services  more 
accessible.  The  figures  are  slightly  different  when  considering  the  targeted  surveys  to 
specific stakeholder groups. According to 50% of NGOs (24 out of 48), the Directive has 
contributed,  at  least  to  some  extent,  to  increasing  the  accessibility.  Similarly,  50%  of 
individuals (20 out of 40) believed that the level of accessibility of digital public services 
improved since the implementation of the Directive. 

In addition, the consultations also provided evidence on the availability and the perceived 
accessibility  improvements  on  different  online  channels.  Despite  online  information 
provided by PSBs has increased in the last three year according to 85% (265 out of 311) 
of participants (84% for interactive services, 78% for mobile applications, and 72% for 
online documents), when asked whether such online channels were made more accessible, 
the percentages were substantially lower (61% for online information, 43% for interactive 
services, 33% for mobile applications and 39% for online documents). The results of the 
targeted surveys  with  specific stakeholder categories are  aligned with  the results of the 
public consultation,  with  some differences  in the magnitudes. Most of the  PSBs  (84%) 
reported a positive impact of the Directive on websites’ accessibility and, to a lower degree, 
on online documents (56%) and mobile applications (26%). Similarly, 71% of the NGOs 

81 

 
reported  a  positive  impact  on  websites’  accessibility  and,  to  a  lower  degree,  on  online 
information (64%), interactive services (58%) and mobile applications (51%). Individuals 
provided slightly more negative answers. 51% of the respondents reported that there has 
been  an  improvement  in  accessibility  of  online  information  on  websites,  while 
significantly  less  individuals  reported  the  same  for  interactive  services  (29%),  mobile 
applications (20%) and online documents (19%). In addition, the answers to the targeted 
survey for all stakeholder categories suggest that the Directive has been mostly effective 
at improving accessibility at the national level, and, to a lesser extent, at the regional and 
local level. 

The in-depth interviews conducted provided additional information on the achievement of 
the Directive’s first objective. Monitoring bodies reported that since the implementation 
of the directive they noticed a significant increase in the awareness of web accessibility 
among PSBs, which with time may lead to increased levels of accessibility in different 
online channels.  In particular, the testing activities during the monitoring have been an 
important occasion of awareness raising. 

Improving internal market 

Moving to the second objective of the Directive (i.e., to harmonise the internal market 
for the accessibility of websites and mobile applications of public sector bodies), the most 
interesting results came from the in-depth interviews and the survey targeting technology 
providers.  The  other  respondents  to  the  public  consultation  reported  high  levels  of 
uncertainty  and  the  results  from  the  survey  targeting  public  sector  bodies  were 
inconclusive. While 30% of PSBs (19 out of 62) believed that the directive has increased 
cross-border provision of products and services for web-accessibility, 47% did not have an 
opinion on the topic. Moreover, 42% of the PSBs did not believe that the Directive has 
been effective at reducing prices of accessibility services. In line with this, 42 out of 311 
participants (13.5%) of the public consultation considered reduced prices for accessibility 
tools and solution the least important key success factor for WAD. 

The  survey  targeting  industry  players  and  technology  providers  aimed  at  exploring 
more  in  detail  the  issues  related  to  the  second  objective  of  the  Directive.  61%  of 
participants (32 out of 52) believed that the Directive has increased cross-border demand 
of web accessibility  products  and services. This  was beneficial for SMEs, according to 
59% of them (31 out of 52). On the supply side, the impact on local companies has been 
reported to be slightly more significant than that on European companies in other countries 
(61%  versus  56%).  Moreover,  64%  believed  that  the  Directive  also  contributed  to  the 
development of new accessibility tools, and 59% reported an increase in  the number of 
industry actors specializing in accessibility tools. However, only 22% reported an impact 
on the prices of digital accessibility services. In the in-depth interviews, there was little 
evidence for market changes. Rather the monitoring agencies reported lack of expertise as 
a major barrier for the success of WAD. 

The  results  of  the  targeted  survey  were  confirmed  by  the  in-depth  interviews  with 
technology  providers,  who  reported  that  the  Directive  has  lowered  barriers  to  entry 
other  European  markets,  facilitating  cross-border  sales.  Technology  providers 

82 

 
highlighted  that  tech  companies  are  making  considerable  investments  to  provide  better 
accessibility  services  and  tools  and  that  the  Directive  has  created  several  new  business 
opportunities for companies of all sizes. 

Success factors and challenges 

According to the participants of the public consultation, the three key success factors in 
the implementation of the Directive were (i) the accessibility statements (121 out of 311), 
(ii) the harmonisation of minimum accessibility requirements (118 out of 311) and (iii) the 
motivation, awareness and expertise of managers working in PSBs (113 out of 311). 

Moreover, the 126 participants that replied that have been involved in the implementation 
of the Directive (including legislators, monitoring or enforcement bodies, a standardisation 
body, DPOs and PSBs), reported further opinions on the success factors. First, 76% of the 
participants of this sub-group (96 out of 126) believed that the Directive has contributed 
to the implementation of a formal monitoring procedure, while 60% believed the same 
for the formal enforcement procedure (76 out of 126). Finally, 57% believed that the 
minimum requirements introduced by the Directive have helped increase the availability 
of accessible websites and mobile applications (71 out of 126). 

From the perspective of the monitoring bodies consulted during the in-depth interviews, 
one  of  the  main  success  factors  mentioned  was  the  involvement  of  persons  with 
disabilities  and  other  stakeholders  in  the  implementation  of  the  Directive  and  the 
monitoring activities. However, the survey targeting NGOs provided a different picture. 
Most  of  the  respondents  reported  little  or  no  involvement  of  their  organizations  in  the 
implementation of the Directive (60% - 29 out of 48) and the selection of websites and 
mobile  applications  to  be  monitored  (64%  -  31  out  of  48).  Despite  the  low  degree  of 
reported  involvement,  67%  of  the  respondents  (33  out  of  48)  stated  that  the  Directive 
helped organizations representing people with disabilities or older people in the EU to be 
heard on issues related to web accessibility. 

Lastly, two other important success factors mentioned during the in-depth interviews with 
monitoring bodies were the facilitation of training and awareness raising in Member 
States and the frequent exchanges with the Web Accessibility Directive Expert Group 
(WADEX). However, according to only 35% of NGOs consulted in targeted surveys (17 
out of 48), such initiatives have raised awareness, at least to some extent. 

Moving to the challenges, the survey targeting PSBs (232 respondents) revealed that the 
main  ones  were  related  to  (i)  technical  difficulties  in  making  existing  websites/apps 
compliant  (38%),  (ii)  low  level  of  accessibility  knowledge  in  suppliers  offering  ICT 
products  and  services  (33%),  and  (iii)  lack  of  capacity  of  web  managers  working  with 
accessibility at public sector bodies (32%). These results were confirmed by the in-depth 
interviews: several monitoring bodies (16) highlighted a lack of experts in the market, 
although  an  increase  of  competent  market  players  is  expected  in  the  upcoming  years. 
Furthermore,  none  of  the  monitoring  bodies  asked  for  a  skill  certificate  when  hiring 
external  experts, mostly  because often there are no official  certification systems, which 
makes it more difficult to identify experts. Monitoring bodies would welcome for the next 

83 

 
monitoring  period  clear  and  uniform  instructions  from  the  Commission  to  conduct  the 
monitoring and to report the results. 

3.3.2  Efficiency 

The consultation activities covered the efficiency evaluation criteria, gathering views and 
information on the costs and benefits related to the implementation of the Directive. 

Costs 

The public consultation did not provide conclusive responses related to the costs, as 58% 
of participants were not sure about whether the Directive’s objectives have been achieved 
at a reasonable cost. Some participants in the comments lamented the lack of available 
data to reply to this question. Only 18% agreed with the statement and 23% disagreed or 
strongly disagreed. 

However, the consultations with specific stakeholders provided more information on the 
costs. 68%  of the PSBs  consulted (42 out  of 62) with  targeted survey believed that the 
Directive introduced additional economic burden to their organizations. Among them, 
(i) improving the accessibility of digital services, (ii) training of human resources and (iii) 
the  creation  of  accessibility  statements  were  the  main  sources  of  recurring  costs.  The 
Directive  introduced  some  additional  costs  also  to  industry  actors  and  technology 
providers, according to the responses to the targeted survey. In particular, 35% of them (18 
out  of  52)  reported  new  costs  associated  to  hiring  new  specialists,  while  33%  of  them 
reported new investments to update products and services to meet the new accessibility 
requirements  (17 out of 52). On the contrary, most of the (79%) reported no additional 
costs for their organisations (38 out of 48). 

In  addition,  the  Directive  introduced  new  costs  for  the  monitoring  and  enforcement 
bodies. According to the participants in the targeted survey, the main factor contributing 
to an increase in costs was represented by the burden of monitoring activities (30 out of 
62).  Monitoring  bodies  provided  some  information  on  this.  However,  the  figures  vary 
significantly  and  provide  only  a  high-level  overview  and  some  examples,  without  the 
possibility  to  draw  solid  conclusions.  The  overall  figures  were  reported  only  by  few 
Member States and they range from €65,000 to €650,000 a year. 

Benefits 

The benefits brought by the implementation of the Directive were different depending on 
the stakeholder group. Overall, 59% of participants of the standard survey (184 out of 311) 
agreed that the benefits arising from the availability of accessible websites and mobile 
applications  outweigh  the  costs  of  implementing  the  Directive.  Some  participants 
commented that accessibility is a human right, compared to which the costs are irrelevant. 
Looking at specific benefits, 56% of PSBs consulted appreciated the role of the Directive 
in inducing organizations to provide clearer accessibility requirements in procurement 
procedures. According to monitoring and enforcement bodies, the main benefits were (i) 
the harmonization of minimum requirements, (ii) the knowledge increase connected to the 

84 

 
trainings, and (iii) the feedback, compliant and enforcement mechanisms, even if they are 
not  yet frequently used. Finally, the industry players and technology providers reported 
during the interviews that the Directive brought benefits also for the private sector. The 
main benefits highlighted by the survey respondents were an increase in the demand of 
products  and  services  in  the  national  markets  (27  out  of  52)  and  an  increase  in  the 
opportunities to penetrate different national markets (18 out of 52). 

3.3.3  Relevance 

Most of the 311 respondents to the standard survey of the public consultation consider the 
specific objectives still relevant today, considering the evolving context of the last years. 
97%  confirm  the  relevance  of  providing  accessible  online  content  and  services;  95% 
confirm that for mobile content and services, including mobile applications; 98% confirm 
that for digital inclusion of persons with disabilities and people with functional limitations; 
and 98% confirm that for reducing regulatory  differences among Member States in  the 
field of web accessibility. 

While  interviews  with  representatives  of  older  people  organisations  confirmed  that  the 
objectives  of  the  WAD  are  highly  relevant  also  for  older  people,  two  specific  issues 
emerged. First, a wide share of older people in Europe do not access public services online. 
Second, those who use them are not much aware of the provisions of the Directive (e.g., 
the feedback mechanism). 

3.3.4  Coherence 

Most of the respondents to the standard survey of the public consultation believed that the 
Directive  is  coherent  with  existing  EU  legislation  (54%),  as  well  as  with  national 
legislations (56%). Similarly, 71% of the PSBs (44 out of 62) have never or rarely noticed 
any difference between the Directive and existing policy measures. 

Monitoring bodies in the interviews reported that the Directive is coherent internally and 
with other key EU legislations, such as the European Accessibility Act. Finally, almost all 
monitoring  bodies  interviewed  reported  that  there  are  no  overlaps  or  inconsistency 
between  the  Directive  and  other  relevant  national  measures  related  to  web 
accessibility. 

3.3.5  EU added value 

Most  of  the  participants  of  the  standard  survey  agreed  that  the  Directive  brought  an 
added value in improving online accessibility for public services in their country (66% - 
207 out of 311) and in increasing digital inclusion of users (53% - 165 out of 311). 

Looking at the different stakeholder categories that replied to the targeted surveys, most of 
the PSBs (79% - 49 out of 62) and NGOs (72% - 35 out of 48) believed that the Directive 
contributed  to  making  national  laws  on  web  accessibility  requirements  more 
harmonised. This  opinion was also  largely shared by monitoring bodies during the in-
depth interviews. Similarly, the majority of PSBs (61% - 38 out of 62) believed that the 

85 

 
Directive contributed to implementing the provisions of the UNCRPD and the EU Charter 
of  Fundamental  Rights  at  national  level.  According  to  industry  players,  the  Directive 
brought an added value also on the internal market, by increasing the demand from 
public  authorities  (65%  -  34  out  of  52),  as  well  as  by  increasing  awareness  about 
accessibility solutions among web professionals (62% - 32 out of 52). 

Finally,  the  monitoring  bodies  interviewed  reported  that  the  main  added  value  of  the 
Directive has been the significant increase in the awareness of web accessibility among 
PSBs. In particular, the communication around monitoring results have been an important 
occasion of awareness raising. 

4.  List of interviewees 

Table 3. List of interviewed stakeholders 
Stakeholder group 

Organisation / Institution 

Disabled persons organisations (DPOs) 

European Disability Forum 

Representatives of older people 

Industry players 

Technology providers  

Standardisation bodies 

EU policy experts 

AGE Platform 

SeniorNet 

IAAP 

Independent expert* 

Yahoo 

Google 

ETSI  

CEN / CENELEC 

European Commission – DG EMPL 

European Commission – DG COMM 

*The interviewee replied to the questionnaire in writing 

Table 4. List of interviewed national monitoring, reporting and enforcement bodies 
Country 

Name of the body 

Bulgaria 

State  e-Government  Agency,  E-government  Policies  Directorate 
(Bulgaria) 

Croatia 

The Office of the Information Commissioner of the Republic of Croatia 

86 

 
 
Country 

Name of the body 

Cyprus 

Department of Information Technology Services | Deputy Ministry of 
Research, Innovation and Digital Policy 

Czechia 

Ministry of the Interior 

Denmark 

Danish Agency of Digitisation 

German federal monitoring body, shortly BFIT-Bund 

Monitoring Body for Digital Accessibility of the Saarland * 

Monitoring Body Brandenurg* 

Zentralstelle 
Landesbehindertenbeauftragten Bremen 

für  barrierefreie 

Informationstechnik  –  Büro  des 

Germany 

Monitoring  Body  for  Accessibility  of  Information  Technology  in 
Saxony 

Monitoring Body Baden Wuttenberg* 

Monitoring Body North Rhine-Westphalia* 

Monitoring Body Niedersachsen* 

Greece 

Ministry  of  Digital  Governance,  Directorate  of  Digital  Strategy,  Web 
Accessibility and Social Affairs Dept 

Hungary 

Governmental Agency for IT Development (KIFÜ) 

Ireland 

National Disability Authroity 

Italy 

AgID – Agenzia per l’Italia Digitale 

Latvia 

Ministry of Environmental Protection and Regional Development 

Luxembourg  Service information et presse 

Malta 

Malta Communications Authority 

Netherlands 

Ministry  of  the  Interior,  the  Government's  expert  centre  for  web 
accessibility 

Poland 

Chancellery of the Prime Minister 

Portugal 

AMA,  I.P.  –  Agência  para  a  Modernização  Administrativa,  Instituto 
Público. 

87 

 
Country 

Name of the body 

Slovakia 

Slovenia 

Spain 

Ministry of Investments, Regional Development and Informatization of 
the Slovak Republic 

Information Security Administration of the Republic of Slovenia 

Ministry of Public Administration 

Observatorio de Accesibilidad Web (Ministerio de Asuntos Económicos 
y Transformación Digital) 

Sweden 

The Agency for Digital Government (DIGG) 

Finland 

Regional State Administrative Agency for Southern Finland 

*The interviewee replied to the questionnaire in writing 

88 

 
ANNEX VI. ADDITIONAL CONSIDERATIONS FROM NATIONAL MONITORING REPORTS 

Member State monitoring reports are published on national websites and made available 
also  on the Commission’s website  Web Accessibility Directive  –  Monitoring reports  in 
original language and automated translations in English). According to these reports, total 
of 10 412 websites and 298 mobile applications have been tested for accessibility by MS 
during  the  first  monitoring  period,  in  what  has  been  described  as  the  ‘world’s  largest 
accessibility test’. 

The information provided in this Annex VI is complementary to findings and analysis in 
the main text of the evaluation report. 

With regards to the administrative levels, the first monitoring reports show that almost all 
countries ensured  a representative sample from local,  regional and national  websites of 
public  sector  bodies,  taking  as  reference  the  Nomenclature  of  Territorial  Units  for 
Statistics 208 (NUTS) and the Local Administrative Units (LAU) set out in the NUTS. Not 
all NUTS and LAU levels exist in Latvia, Luxembourg and Malta. Therefore, reports from 
Latvia and Luxembourg 209 referred only to national and local public sector websites and 
not to any “regional” website. Malta also reported that it was not feasible for the Authority 
to consider websites at a regional level. 

On  sample  size,  the  first  monitoring  reports  show  that  most  countries  respected  the 
minimum  target 210  set  in  the  Implementing  Decision  for  all  three  monitoring  methods: 
simplified  monitoring  of  websites;  in-depth  monitoring  of  websites,  and  in-depth 
monitoring of mobile applications. 

208  https://ec.europa.eu/eurostat/web/nuts/background. 
209  Luxembourg  stated  that  no  website  at  regional  level  was  found  in  the  country’s  inventory  of  public 

websites, source: https://data.public.lu/fr/datasets/inventaire-des-sites-publics/. 

210  Source: Study supporting the review of the application of the Web Accessibility Directive (WAD), Final 
Report, PwC, Intellera Consulting, Open Evidence, Funka, 2022, cited above (note 17). No information 
available yet for France and Cyprus. 

89 

 
                                                 
Table 5. Minimum sample size for monitoring 

Minimum sample size 211 

Monitoring  methods 
and scope  

First monitoring period 

Second 
period 

monitoring 

Third 
monitoring periods 

and 

following 

Simplified  monitoring 
of websites 

2 websites per 100 000 inhabitants + 75 websites 

3  websites  per  100  000 
inhabitants + 75 websites 

In-depth monitoring of 
websites 

5% of the minimum sample size for the simplified monitoring in the  first and 
second monitoring period + 10 websites 

In-depth monitoring of 
mobile applications 

of 

third 

One 
the 
minimum sample asked 
following 
the 
in 
monitoring period 

1 mobile application per 1 000 000 inhabitants +  6 
mobile applications 

On  the  simplified  monitoring  of  websites 212,  only  Ireland  and  Finland  declared  a 
significantly  lower-than-minimum  number  of  websites  monitored 213  (respectively  less 
than 30% and less  than  60%).  Belgium and Malta significantly  exceeded the minimum 
target (respectively monitoring 412 and 87 websites more than the minimum). 

On  the  in-depth  monitoring  of  websites 214,  eight  countries  did  not  reach  the  set 
minimum. Sweden monitored less than 80% of the minimum target sample, while Estonia 
monitored  less  than  75%,  Greece  less  than  65%,  and  Ireland  less  than  30%.  Some  EU 
countries (Belgium, Germany, Malta and Poland) monitored in-depth a sample that was 
above  the  minimum  target  (these  countries  respectively  monitored  6,  33,  14  and  75 
websites more than the minimum). Latvia, Hungary, Romania and Slovenia did not report 
any data. 

Finally, on mobile applications, two countries monitored a smaller than minimum sample 
(Ireland  and  Sweden),  while  Belgium,  Latvia  and  Slovenia  did  not  provide  any  data. 
Several  EU  Member  States  went  beyond  the  minimum  sample  size:  Bulgaria,  Czechia, 
Germany,  Estonia,  Greece,  Croatia,  Lithuania,  Hungary,  Malta,  Portugal,  Romania  and 
Slovakia. 

The Implementing Decision laying down a monitoring methodology and the arrangements 
for  reporting 215  specifies  that  the  sample  analysed  in  monitoring  reports  ‘shall  include 

211  The  Implementing  Decision  (EU)  2018/1524  provides  exceptions  for  Member  States  which  have  an 
overall number of websites or mobile applications lower than the size of the population-based sample. 
In such cases, MS shall monitor at least 75% of all websites available, and at least 50% of all mobile 
applications available. 

212  Simplified monitoring is a method applied to websites that detects instances of non-compliance with a 

sub-set of the requirements in the standards. 

213  It should be noted that these Member States have committed to update their national report in 2022. 
214  In-depth monitoring method thoroughly verifies whether a website or mobile application satisfies all the 
requirements  identified  in  the  standards.  Ideally  this  should  be  performed  by  users  with  a  range  of 
disabilities and on a variety of devices. 

215  Annex I, section 2.2.3. 

90 

 
                                                 
websites representing as much as possible the variety of services provided by the public 
sector  bodies’.  Three  Member  States  did  not  report  any  breakdown  of  the  sectors 
monitored. Most of the remaining 22 reports (from 16 countries) declared that the sample 
size covered all the sectors recommended in the Implementing Decision 216, namely: social 
protection;  health;  transport;  education;  employment  and  taxes;  environmental 
protection;  recreation  and  culture;  housing  and  community  amenities;  and  public 
order  and  safety.  In  11  countries,  most  websites  monitored  did  not  fall  within  the 
categories provided in the Implementing Decision and were included in a residual category 
of  ‘other’.  Among  the  categories  provided,  the  three  most  monitored  services  were:  (i) 
education; (ii) social protection; and (iii) recreation and culture. The two least monitored 
services were health and transport 217. 

For mobile applications, the Implementing Decision required different operating systems 
to be covered. 16 Member States reported data on how many Android-based or iOS-based 
applications were included in the sample. Four Member States monitored only one type of 
operating system application 218, while two Member States did not report the distribution, 
and 3 (Belgium, Latvia and Slovenia) did not monitor mobile applications at all. Bulgaria 
reported that iOS-based applications were not used by a significant number of people with 
disabilities, because no adequate speech synthesiser is currently available in Bulgarian for 
that platform. Two countries provided a breakdown of the applications according to the 
service provided by public sector bodies, though this is not required by the Implementing 
Decision. 

216  Ibid. 
217  This statement is based on the figures provided by Member States specifying the number of websites 

monitored for each sector, namely all the Member States except for Ireland and Bulgaria. 

218  Three Member States monitored only Android-based applications and 1 Member State monitored only 
iOS-based applications. Czechia stated that iOS-based applications could not be monitored due to the 
lack of available techniques. 

91 

 
                                                 
ANNEX VII. MAPPING OF UN SUSTAINABLE DEVELOPMENT GOALS (SDGS) 

The mapping of SDGs in relation to WAD and its implementing acts was partly done using 
the  SDG  Mapper 
INTPAJRC, 
https://knowsdgs.jrc.ec.europa.eu. 

tool,  developed  by 

and  DG 

JRC 

the 

All of the UN Sustainable Development Goals (SDGs) are coherent with the achievement 
of more inclusion of people with disabilities in society 219. The provisions of the WAD are, 
in particular, relevant for achieving the objectives of SDGs 1, 3, 4, 8, 9, 10, 16, and 17 as 
flagged by the UN Flagship Report on Disability and Sustainable Development Goals 220. 

Figure 2. SDG Goals and the percentage of corresponding keywords detected in the text of WAD and its 
implementing acts (calculated as the ratio of keywords in one Goal to the total number of keywords detected) 

Achieving  such  goals  requires  the  accessibility  of  public  sector  bodies’  digital 
administrative procedures, including services of general interest. The bullet points below 
describe how the key provisions of WAD and its implementing acts contribute to these 
SDGs 221. 

  The WAD aims at increasing the digital inclusion of people with disabilities and this 
is directly related to SDG 10 – ‘Reduce inequality within and among countries’. SDG 
10 seeks to promote social, economic and political inclusion of all people (including 

219  https://www.un.org/development/desa/disabilities/publication-disability-sdgs.html. 
220  https://social.un.org/publications/UN-Flagship-Report-Disability-Final.pdf. 
221  The mapping of SDGs in relation to WAD was partly done using the SDG Mapper tool, developed by 

the JRC and DG INTPAJRC, https://knowsdgs.jrc.ec.europa.eu/. 

92 

 
 
                                                 
people  with  disabilities)  ensuring  equal  opportunities  and  ending  discrimination 
(through Targets 10.2 and 10.3). 

  The aim of increasing digital inclusion is also connected to SDG 16 – ‘Promote just, 
peaceful  and  inclusive  societies’.  SDG  16  seeks  to  ensure  responsive,  inclusive, 
participatory and representative decision-making at all levels (Target 16.7) as well as 
to  promote  and  enforce  non-discriminatory  laws  and  policies  for  sustainable 
development (Target 16.b). 

  Ensuring accessible communication also supports SDG 17 – ‘Strengthen the means of 
implementation and revitalize the Global Partnership for Sustainable Development’. 
Target  17.8  of  this  goal  ‘aims  to  operationalize  the  Technology  Bank  and  Science, 
Technology  and  Innovation  capacity-building  mechanism  and  enhance  the  use  of 
enabling technology, in particular information and communications technology’. 

  By  aiming  to  increase  the  inclusion  of  online  public  services,  the  WAD  can  bring 

indirect positive impacts to the achievement of the following (sectoral) SDGs: 

o  SDG 1 – ‘Ending poverty in all its forms everywhere’; 

o  SDG 3 – ‘Ensure healthy lives and promote well-being for all at all ages’; 

o  SDG  4  –  ‘Ensure  inclusive  and  equitable  quality  education  and  promote 

lifelong learning opportunities for all’; 

o  SDG 8 – ‘Promote sustained, inclusive and sustainable economic growth, full 

and productive employment and decent work for all’; 

o  SDG  9  –  ‘Build  resilient  infrastructure,  promote  inclusive  and  sustainable 
industrialization  and  foster  innovation’,  because  application  of  the  WAD 
contributes for technical developments of digital products and services to make 
them  more  accessible  and  inclusive  (particularly  target  9.c  ‘significantly 
increase access to information and communications technology’). 

93 

 
ANNEX VIII. BIBLIOGRAPHY 

Angélico,  M.  J.,  Silva,  A.,  Teixeira,  S.  F.,  Maia,  T.,  &  Silva,  A.  M.  (2017).  Web 
Accessibility  and  Transparency  for  Accountability:  The  Portuguese  Official  Municipal 
Websites.  Research  Paradigms  and  Contemporary  Perspectives  on  Human-Technology 
Interaction, 140-166. 

Alcántara, Ann-Marie, The Wall Street Journal (WSJ), More Companies Are Looking to 
Hire  Accessibility  Specialists, 
https://www.wsj.com/articles/more-companies-are-
looking-to-hire-accessibility-specialists-11630501200, 1.9.2021 

Barricelli,  B.  R.,  Sciarelli,  P.,  Valtolina,  S.,  &  Rizzi,  A.  (2018).  Web  accessibility 
legislation  in  Italy:  a  survey  10  years  after  the  Stanca  Act.  .  Universal  Access  in  the 
Information Society, 211-222. 

Berlin Declaration on Digital Society and Value-based Digital Government (2020). 

Berlin  Declaration,  Report  on  the  monitoring  of  the  Berlin  Declaration,  Directorate 
General for Informatics, (2022). 

Berton, R., Gaggi, O., Kolasinska, A., Palazzi, C. E., & Quadrio, G. (2020). Are captchas 
preventing robotic intrusion or accessibility for impaired users. 2020 IEEE 17th Annual 
Consumer Communications & Networking Conference (CCNC), 1-6. 

CAPGEMINI & DIALOGIC. (2017). The evaluation of implementation costs and future 
effects of the Web Accessibility Directive. Bruxelles: European Commission. 

Easton,  C.  (2013).  Website  accessibility  and  the  European  Union:  Citizenship, 
procurement and the proposed Accessibility Act. Int Rev Law Comput Technol, 187-199. 

EDF. (2017). Toolkit - Directive on accessibility of the websites and mobile applications 
of public sector bodies. Bruxelles: Publication office of the EU. 

Empirica, Funka, WRC. (2014). Measures to improve accessibility of public websites in 
Europe. 

Eurofound. (2011). Measures for social inclusion of the elderly: The case of volunteering. 

European  Commission  (2007).  Measuring  progress  of  eAccessibility  in  Europe  – 
Assessment of the Status of eAccessibility in Europe, Main report. 

European  Commission  (2010).  Communication  from  the  Commission  to  the  European 
Parliament, the Council, the European Economic and Social Committee and the Committee 
of the Regions – European Disability Strategy 2010-2020: A Renewed Commitment to a 
Barrier-Free Europe, COM(2010) 636 final. 

European Commission (2011). Monitoring e-accessibility in Europe: 2010-2011, Annual 
report. 

European Commission (2012). Impact assessment accompanying the document ‘Proposal 
for a Directive of the European Parliament and of the Council on the accessibility of public 
sector bodies’ websites, SWD(2012) 401 final; Executive summary, SWD(2012) 402 final; 
Proposal for a Directive, COM(2012) 721 final. 

European Commission (2013). G3ict, Study on Assessing and Promoting e-Accessibility, 
Final report. 

94 

 
European  Commission  (2017).  Communication  from  the  Commission  to  the  European 
Parliament, the Council, the European Economic and Social Committee and the Committee 
of  the  Regions  –  European  Interoperability  Framework  –  Implementation  Strategy, 
COM(2017) 134 final. 

European  Commission.  (2017).  Quality  of  Public  Administration.  A  Toolbox  for 
Practitioners. 

European Commission. (2018). e-Govenrment in Spain. European Commission. 

European  Commission  (2018).  EU-funded  research  projects  into  technologies  for 
accessibility. Luxemburg: Publication office of the EU. 

European Commission (2019). Shaping Europe’s digital future, Accessibility: Essential for 
Some, Useful for All, Factsheet/Infographic. 

European Commission (2019). Peer Review on ‘Strategies for supporting social inclusion 
at older age’. 

European Commission (2019). Survey on Web Accessibility Directive transposition and 
implementation. Bruxelles: Publication office of the EU. 

European  Commission  (2020).  Digital  Education  Action  Plan  2021-2027,  Resetting 
education and training for the digital age, COM(2020) 624 final. 

European  Commission  (2020).  Governance  and  public  administration,  Directorate-
General for Structural Reform Support. 

European Commission (2020). Communication ‘Identifying and addressing barriers to the 
Single Market’, COM(2020) 93 final. 

European Commission (2021). Better regulation: guidelines and toolbox. 

European Commission (2021). Joint Research Centre, Science Policy Briefs: Telework in 
the EU before and after the COVID-19: where we were, where we head to. 

European Commission (2021), Joint Research Centre, Digitranscope: The governance of 
digitally-transformed society. 

European  Commission  (2021).  Staff  Working  Document  –  Supporting  Public 
Administrations  in  EU  Member  States  to  Deliver  Reforms  and  Prepare  for  the  Future, 
Commission Staff Working Document SWD(2021)101 final. 

European  Commission  (2021).  State-of-play  report  on  digital  public  administration  and 
interoperability, 2021, Directorate General for Informatics. 

European Commission (2021). Union of Equality: Strategy for the Rights of Persons with 
Disabilities 2021-2030, COM(2021) 101 final. 

European Commission (2022). Digital Skills and Jobs Platform. 

European  Commission  (2022).  eGovernment  Benchmark  2022  &  Background  Report. 
Synchronising Digital Governments. 

European  Commission  (2022).  Joint  Research  Centre  EU  Science  Hub,  The  Digital 
Competence Framework and DigComp 2.2. 

95 

 
European Commission (2022). Report from the Commission to the European Parliament 
and  the  Council  on  the  implementation  of  the  Recovery  and  Resilience  Facility, 
COM(2022) 75 final. 

European Commission (2022). Report from the Commission to the European Parliament 
and the Council – Review report on the implementation of the Recovery and Resilience 
Facility, COM(2022) 383 final. 

European Committee of  the Regions  (2019). Resolution  ‘the Green Deal  in  partnership 
with local and regional authorities’. 

European Disability Forum (2019), Survey on Web Accessibility Directive transposition 
and implementation, presented at the 6th WADEX meeting, Brussels, 28 November 2019. 

European  Parliament  (2020),  European  Parliamentary  Research  Service  (EPRS), 
Employment and disability in the European Union, Briefing. 

European Union (2012). Charter of Fundamental Rights of the European Union. Official 
Journal of the European Union. 

Fiers,  Wilco 
www.deque.com/blog/eu-runs-worlds-largest-accessibility-test. 

(2022).  EU  Runs  World’s  Largest  Accessibility  Test, 

Ferri,  D.,  &  Favalli,  S.  (2018).  Web  Accessibility  for  People  with  Disabilities  in  the 
European Union: Paving the Road to Social Inclusion. Societies. 

FRA.  (2020).  What  do  fundamental  rights  mean  for  people  in  the  EU?  Luxembourg: 
Publications Office of the European Union. 

Freire, A. P., Russo, C. M., & Fortes, R. P. (2008). The perception of accessibility in web 
development by academy, industry and government: A survey of the Brazilian scenario. 
New Review in Hypermedia and Multimedia, 149-175. 

Gambino, O., Pirrone, R., & DiGiorgio, F. (2016). Accessibility of the Italian institutional 
web pages: a survey on the compliance of the Italian public administration web pages to 
the Stanca Act and its 22 technical requirements for web accessibility. Univ Access Inf 
Soc, 305-312. 

Grammenos,  S.  (2018).  European  Comparative  Data  On  Europe  2020  &  People  With 
Disabilities. Centre de politique sociale et économique européenne Asbl (CESEP). 

Hermann,  M. 
https://medium.com/@marcusherrmann/what-isdirective-eu-2016-2102-a2ce364c77dd 

is  Directive  EU  2016/2102?  Retrieved 

(2018).  What 

from 

Inal, Y., & Ismailova, R. (2020). Effect of human development level of countries on the 
web  accessibility  and  quality  in  use  of  their  municipality  websites.  Journal  of  Ambient 
Intelligence and Humanized Computing. 

Loiacono, E. T., & McCoy, S. (2006). Website accessibility: a cross-sector comparison. 
Univ Access Inf Soc , 393-399. 

LoPresti,  E.  F.,  Bodine,  C.,  &  Lewis,  C.  (2008).  Assistive  Technology  for  Cognition  - 
Understanding the Needs of Persons with Disabilities. Quality of Life Technology. 

Medeiros-Eler, M. J. (2018). Automated accessibility testing of mobile apps. IEEE 11th 
International Conference on Software Testing, Verification and Validation (ICST). 

96 

 
Ministru  Kabineta.  (2020).  Kārtība,  kādā  iestādes  ievieto  informāciju  internetā.  Riga: 
Ministru Kabineta. 

Parmanto,  B.  (2005).  Metric  for  web  accessibility  evaluation.  Journal  of  the  American 
Society for Information Science and Technology. 

Pribeanu, C. (2019). Large-scale accessibility evaluation of Romanian municipal websites. 
International Journal of User-System Interaction , 83-98. 

Rajšp, A., Kous, K., Kuhar, S., Sumak, B., & Sorgo, A. (2019). Preliminary review of jobs, 
skills and competencies for implementation of Digital Accessibility. Proceedings of the 
Central European Conference on Information and Intelligent Systems. 

Ringlaben,  R.,  Bray,  M.,  &  Packard,  A.  (2014).  Accessibility  of  American  University 
Special Education Departments' Web sites. Universal Access in the Information Society. 

Rubacek,  F.,  Jindřichovská,  I.,  Horváthová,  Z.,  &  Abrhám,  J.  (2020).  Accessibility  of 
Websites of the European National Tourism Boards. International Journal of Economics 
and Business Administration, 114-125. 

Rubano, V., & Vitali, F. (2021). Making accessibility accessible: strategy and tools. 2021 
IEEE 18th Annual Consumer Communications & Networking Conference (CCNC). Las 
vegas: IEEE. 

Sabev,  N.,  Georgieva-Tsaneva,  G.,  &  Bogdanova,  G.  (2020).  Research,  analysis  and 
evaluation of web accesibility for a selected group of public websites in Bulgaria. Journal 
of Accessibility and Design for All. 

Sala, A., Arrue, M., Perez, J. E., & Valencia, X. (2019). Accessibility-in-use of public e-
services: An exploratory study including users with low vision. INTERACCION’19. 

Sanchez-Labella-Martin, I., Simelio, N., & Moreno-Sarda, A. (2017). Web accessibility 
for people with disabilities in Spanish city councils. Cuadernos, 155-172. 

Sik-Lanyi, C., & Orban-Mihalyko, E. (2019). Accessibility Testing of European Health-
Related Websites. Arabian Journal for Science and Engineering, 9171-9190. 

Teixeira, L., Eusébio, C., & Silveiro, A. (2019, June). Website accessibility of Portuguese 
travel agents. In 2019 14th Iberian Conference on Information Systems and Technologies 
(CISTI) (pp. 1-6). IEEE. 

Textkernel.nl,  Remote  Work,  The  current  remote  work  trends  in  EMEA:  An  analysis 
across 8 European countries – Pre and during pandemic, 2018 – 2021. 

Traxler, C. (2000). The Stanford Achievement Test, 9th Edition: National Norming and 
Perfirmance Standards for Deaf and Hard-of-Hearing students. Journal of Deaf Studies and 
Deaf Education 5. 

UNDP. (2016). Human Development Report 2016. UNDP. 

United  Nations.  (2006).  Convention  on  the  Rights  of  Persons  with  Disabilities.  United 
Nations. 

United  Nations.  (2014).  General  comment  no.  2  (2014),  Article  9,  Accessibility: 
Committee on the rights of persons with disabilities. Geneva. 

97 

 
United  Nations.  (2015).  Concluding  observations  on  the  initial  report  of  the  European 
Union, Committee on the Rights of Persons with Disabilities. Geneva. 

United Nations. (2017). General comment No. 5 (2017) on living independently and being 
included in the community: Committee on the rights of persons with disabilities. Geneva. 

United  Nations.  (2018).  General  comment  no.  6  (2018)  on  equality  and  non-
discrimination: Committee on the rights of persons with disabilities. Geneva. 

Vasquez-Rodriguez,  L.  M.  (2021).  Investigating  text  simplification  Evaluation.  Cornell 
University. 

Velleman, E. M., Nahuis, I., & VanDerGeest, T. (2017). Factors explaining adoption and 
implementation  processes  for  web  accessibility  standards  within  eGovernment  systems 
and organizations. Universal Access in the Information Society, 173-190. 

Vollenwyder, B., Iten, G. H., Bruhlmann, F., Opwis, K., & Mekler, E. D. (2019). Salient 
beliefs  influencing  the  intention  to  consider  Web  Accessibility.  Computers  in  Human 
Behavior, 352-360. 

W3C. (1999). Web Content Accessibility Guidelines 1.0. 

W3C. (2008). Web Content Accessibility Guidelines 2.0. 

The WebAIM Million.  The 2022  report on the  accessibility of the top  1,000,000 home 
pages. 

Waddington, L. (2019). Regulating e-Accessibility and Digital Equality in Europe from a 
Multilevel Perspective. Scientifica, 3-18. 

WHO. (2011). World Report on Disability. Geneva: WHO Press. 

Yesilada, Y., Brajnik, G., Vigo, M., & Harper, S. (2014). Exploring perceptions of web 
accessibility: A survey approach. Behaviour & Information Technology, 119-134. 

98