Position Paper on the Suitability to Task of Automated Utilities for
Testing Web Accessibility Compliance

Bill Killam, User-Centered Design, Inc., Ashburn, VA 20147

bkillam@user-centereddesign.com

Bill Holland, Olympus Group, Inc. Alexandria, VA 22314

bill@olympusgroup.com

# Position

In our opinion, the current discussions about accessibility have
included two very separate issues that we believe need to be addressed
separately. The first issue is the law and, specifically the
requirements of Section 508. As a law, these requirements must be
unambiguous or, as members of our industry are more likely to think of
it, provide for near perfect inter-rater reliability. We believe that
automated utilities that canvass individual web pages, web-based
applications, or other software applications can improve the
*efficiency* of accessibility-related heuristic evaluations.
Specifically, they are valuable at identifying *barriers* to
accessibility, but they currently do not and potentially cannot address
accessibility itself.

For an encompassing evaluation of a product accessibility, it is
ultimately a question of the end user's ability to use locate
information or exercise functionality that determines accessibility.
Given this distinction, we believe that the accessibility compliance
issue is two fold problem: (1) removing the barriers to accessibility,
and (2) making the site accessible. Human intervention is required to
supplement a utility's findings, as well as conduct individual reviews
of certain page elements and heuristics that automated tools are unable
to perform at this time.

##### 

##### 

##### 

##### 

##### 

##### 

##### 

##### 

##### 

##### Barriers to Accessibility

##### Potential Benefits and Risks of Automated Tools

1.  **Benefit:** Automated tools minimize the chances of missing
    accessibility problems, and therefore omission in the final report.
    For instance, a human evaluator may have time to review only a
    representative sample of images for ALT attribute inclusion, whereas
    an automated tool can review *all* images on a site.

2.  **Benefit:** Automated tools can operate much more quickly than a
    human evaluator, therefore conducting an evaluation in mere minutes,
    whereas a human evaluator may take several hours to review the same
    accessibility barrier criteria.

3.  **Risk:** Use of an automated tool may give a human evaluator a
    false sense of security when evaluating a user interface. Some tools
    are better than others in informing evaluators when further review
    is necessary. Human evaluators need to be aware of these
    shortcomings.

4.  **Risk:** Sole reliance on automated tools by web developers as
    indicators of barrier requirement compliance gives a false sense of
    security to the developers. In addition, it may inaccurately portray
    the site as being fully accessible to persons with disabilities,
    when in fact there may be glaring accessibility-related, and
    therefore, usability, problems.

##### Discussion

The accessibility of a web site is directly related to its usability.
And its usability is directly related to its accessibility. A site may
be in compliance with guidelines for persons with disabilities but still
be completely unusable to both able and disabled persons. Automated
tools that measure a site's "usability", despite the best intentions of
their developers, have not traditionally covered all aspects of a site
adequately. Correspondingly, no tool so far has demonstrated its ability
to get beyond the accessible *and* usable barrier.

Many recent tools available provide feedback to evaluators for
situations where the tool was unable to discern compliance with
accessibility standards. This can help identify where manual review by a
human evaluator is necessary. Some examples:

-   *Color:* Because of the combination of style sheets for text,
    deprecated FONT tags, and graphic images to render color, there are
    many combinations of web coding to consider. A change in headings
    from one page to another can be signified not only by non-color
    attributes, but color as well. In some cases however, color is the
    sole indicator of change in an item. Automated tools are not
    currently equipped to compare before-and-after views of web pages
    based on style sheets or graphics. Therefore, human evaluation must
    be done to assure web pages are accessible to color-blind persons.

-   *Images:* The mere presence of an ALT attribute for an image is not
    necessarily sufficient for true compliance with current guidelines.
    For example, an image in the upper left-hand corner of a page may be
    labeled as "logo" in its ALT attribute, and may even be a hyperlink.
    If no TITLE attribute is set, the ALT attribute is the sole
    indicator of what that link/image does. A visually impaired person
    would have to guess as to the target of this particular link. An
    automated tool would pass through this IMG tag, note the presence of
    an ALT attribute, and give that tag a passing mark, even though the
    attribute is not fully descriptive of the image or the target of the
    link. Here it may meet the accessibility barrier criteria, but still
    be inaccessible.

-   *Tables:* Most guidelines require column and row heading attributes
    be set for each table cell where there are two or more rows of table
    data. Because tables are used not only for numeric presentation, but
    also for page element positioning, the determination of each
    application is difficult to make without visually reviewing the
    table. For this reason, automated tools cannot adequately verify
    table-related guidelines.

##### 

##### 

##### Accessibility as usability for persons with disabilities

E.B. White, when reviewing a "Reading Ease calculator," an early version
of automated tools such as the FOG index for determining readability,
wrote:

"There is, of course, no such thing as the reading ease of written
matter. There is the ease with which matter can be read, but that is
function of the reader, not the matter."

We believe this applies to accessibility in the same manner as we have
always applied it to usability.

By our definition, once the barriers to accessibility have been removed,
we are now faced with the greater and more difficult task of addressing
actual accessibility. Since accessibility, in our definition is the
ability of persons with disabilities to locate and use information or
exercise the functionality of a web-based or other software application,
we are, in fact, talking about usability. As such, the tools,
techniques, and practices of usability now apply. Expert reviews,
heuristic evaluations, style guide compliance, and other non-user-based
techniques can be applied (assuming experts, heuristics, and style
guides exist). These techniques, though cost effective and valuable, are
not substitutes for user-based testing.

We are often compelled to perform user-based testing when the population
of users is a unique subset of the overall population. Some of our
clients and potential clients have rejected this notion since there is
such a wide range of users with disabilities. But as expert reviewers,
we cannot claim to be able to eliminate our backgrounds, experiences,
and cognitive processes from the reviews, so we recognize the necessity
of testing children, older adults, accountants, or other populations.
And we accept 5-8 participants, or a minimum of 8, or even 20 (depending
on to whose position you subscribe) can provide valuable insight into
the issues of a population from a practical (though not research)
perspective. So why not for handicapped populations?

Even if we learned to use the special adaptive tools of the handicapped
community (e.g., refreshable Braille, screen readers, a mouth stick, a
sip-and-puff interface), can we really believe that we can close their
eyes and be blind? Can a person with "normal" hearing ever really
emulate the difference in language skills between themselves and a
congenitally deaf person? Can our infrequent use of a mouth stick ever
compare with a person who is required to use one for their daily
activities? The unique nature of the handicapped population (which is,
in fact, multiple populations) in terms of human factors effecting
usability, should be more likely to compel us to push for user-based
testing.

# conclusion

If we use automated tools to triage the process of identifying the
barriers to accessibility, we have made our jobs significantly easier,
more thorough, and more cost effective. If we can develop fully
automated tools to assess the issue of removing the barriers to
accessibility, than we have accomplished no small feat in terms of both
technical accomplishment and value to the industry. But, if our
assertion is correct, then we have only performed the first necessary
step in addressing accessibility---removing the barriers. We must now
address the special condition of usability related to handicapped users
and accept that user-based evaluation is the only true test of success,
though there may be other tools that can serve as reasonable
substitutes. We must also recognize that we are attempting to address
two fundamentally different but interdependent problems and must apply
the right tool to the job and know when each job is truly done.