Position Paper for UPA 2001 Workshop #6:

Exploring Measurement and Evaluation Methods for Accessibility

Stephanie Rosenbaum,

Tec-Ed, Inc.

# Evaluation and Design Work I Have Done

As founding principal of a usability research firm, I have spent a large
portion of my professional activities over the past ten years developing
and adapting methodologies for usability evaluation. I participate in
many of the initial planning meetings with client organizations needing
usability research, and I work with these clients to identify the
research methods appropriate to achieve their goals. Often that process
involves modifying existing methods and evolving new ones (see also my
UPA 2000 paper, \"Not Just a Hammer: When and How to Apply Multiple
Methods in Usability Programs\").

Although most of Tec-Ed\'s usability research projects are not focused
primarily on accessibility, we are increasingly able to include an
accessibility component, most often by including one or two disabled
participants in usability tests. Occasionally an entire study addresses
a particular accessibility issue, for example, a study of a new laptop
screen where we recruited only participants who wore eyeglasses.
Tec-Ed's staff members are familiar with Section 508 and are comfortable
working with clients who want to test for accessibility.

**Need for Ethnographic and Longitudinal Data in Accessibility
Research**

The challenge for the usability community, as described by the workshop
organizers, is to develop techniques for designing and testing products
for accessibility\--in particular, low-cost test processes to measure
users\' performance with the user interface (UI). One important approach
for developing these techniques and processes is to consult ethnographic
and longitudinal data.

Just as in usability research with non-disabled users, in accessibility
research we often concentrate on measuring user efficiency,
effectiveness, and satisfaction under controlled circumstances such as
usability testing. A shortcoming of this approach is that, when we
design the tasks and their context, we can\'t know what circumstances of
actual use we are not observing. For this reason, a successful usability
research program should include a combination of techniques: usability
testing, inspection methods as appropriate, and ethnographic studies to
inform the other techniques about the real-world context of use.

For us to be successful in creating techniques that both measure Section
508 compliance and improve product design, we need to include
ethnographic studies to inform the development of standard tasks and
performance methods. Ethnographic data, especially longitudinal data,
will help us produce better evaluation methods both for user testing and
for user-free evaluations.

**Example of Ethnographic Accessibility Observations**

Although Tec-Ed has not yet been commissioned by clients to perform
longitudinal studies of accessibility, we work regularly with a
technical consultant (a network software specialist) who has retinopathy
and consequently extremely low vision. For the past two years, we\'ve
observed this person\'s computer use for several hours about twice a
month, and we learned about several accessibility issues we probably
would not have seen during usability testing.

Our subject sets all software to display text in a large type size and a
readable type face, which he then reads one word at a time with a
hand-held magnifier. He also occasionally uses Jaws screen-reading
software, but the issues described here involve attempts to read text
on-screen (an accessibility concern of increasing interest to the aging
U.S. population).

His major concern during software use is not the primary functionality
of an application, which usually displays in the large type size as
instructed (although it does not always allow him to change displays to
a more distinct and readable type face). Rather, whenever any problems
occur, the error messages display in a default type size and type face
that the user cannot adjust, even with the accessibility settings. This
behavior occurs both for error messages from most applications and those
from the Windows operating system.

Thus at unpredictable times, an otherwise competent computer user must
ask for help from a physically present person with better vision.
(Problems also occur when Jaws attempts to read error messages; I can
describe these during the workshop if they\'re of interest.) Can the
Help system provide assistance? Rarely, because most applications also
have their Help messages programmed to display a fixed text size.

Usability problems occur with software applications that would otherwise
be excellent accessibility tools, if a low-vision person could gain
access to the tool. Our subject attempted to use IBM\'s dictation
product, because dictation would be much faster than finding a letter at
a time on the keyboard. To \"train\" the software to recognize the
user\'s voice, one must read sample text out loud\--and the sample text
is displayed in a type face our subject can\'t read. (The product
apparently offers the ability to change type sizes and type faces, but
only after the start-up activities are completed.)

Similarly, a year or two ago I saw a demonstration of a new Adobe
product that would enable users to change the type size of a pdf file,
and the text would \"rewrap\" to create a readable page in the new type
size. Unfortunately, the menus and icons to specify the larger type were
themselves too small for a low-vision person to read\--and I was told
these could not be changed.

Some quite basic software bugs raise questions about QA of accessibility
options. Windows 98 had an accessibility feature that magnified the
whole screen by a specified percentage. About two years ago, our subject
tried setting this feature on his new laptop to 200%, at which point
only a quarter of the screen was visible (the top-left quarter). When he
wanted to turn the feature off, or try a different magnification, all
the controls he needed were on undisplayed parts of the screen. But the
screen displayed no scroll bars (perhaps because this was a \"setting,\"
not a central part of the operating system). After many unsuccessful
calls to the Microsoft Help desk, our subject returned the laptop to his
vendor, who also couldn\'t solve the problem. Their solution was to
replace the hard disk of the laptop with a new one!

Windows also offers a \"hand magnifier,\" which you can move around to
magnify selected portions of a screen. When our subject clicks on this
tool, it works for about 15 minutes and then fails (with another
unreadable error message). Probably most users of the magnifier don\'t
try to use it for hours at a time. Again, the Help desk hasn\'t been
able to help. Our subject pleads strongly for major software companies
having accessibility specialists in tech support, so that Help desk
questions about accessibility options can easily be escalated to someone
knowledgeable about these features.

**Issues Resulting from Ethnographic and Longitudinal Observations**

The example above involves just one subject and only one disability
condition (although we expect problems with low-vision users to become
increasingly prevalent), yet it provided many insights. Our observations
led to ideas, implications, and issues that we can usefully address as
we think about developing tasks and performance measures for
accessibility:

-   Even inspection methods can improve accessibility far more than they
    currently do, simply by extending accessibility standards to the
    entire product---including the installation or start-up sequences
    (or wizards), the error messages, and the help system. Perhaps I'm
    wrong in saying "simply"; the technical challenge is
    straightforward, but the organizational challenges in requiring many
    development teams to cooperate may be greater. Still, user-free
    methods can be developed to ensure that every part of a user
    interface can be displayed in large, distinct type.

-   The process of developing "standard" tasks for accessibility
    usability testing should be an iterative one, rather than performed
    once and never revisited. An ongoing ethnographic, longitudinal
    research program should be carried out with representatives of the
    major groups of people with disabilities. The information gleaned
    from this longitudinal research should be used to update any
    standardized tasks, at least yearly (probably more often).

-   Although an ideal user interface would not require tech support, we
    won't achieve such perfection in product design in our lifetimes
    (considering the diversity of user audiences and the trend toward
    more complex products). Given the requirement to provide user
    support for products, we can improve accessibility dramatically by
    providing specialized user support for accessibility features.

Note that, as Gregg Vanderheiden from TRACE R&D Center said during his
CHI 2001 talk, most of these approaches to improving accessibility also
result in more usable products for their wider audiences, providing an
additional justification for investing in the required resources. For
example, our low-vision subject commented that he could use Jaws more
effectively if its help index included more "ordinary-language" entries
rather than only the terminology used in the product; and usability
studies of indexes show that adding "see also" entries supplied by users
is an effective---and low-cost---way to improve product usability.