Testimony of Noel Runyan

1.  Are there emerging broad themes in the accessibility and usability
    community?

As the population demographic shifts to older voters, more voters will
begin to need features of accessibility. Because of this, the division
between those who have needs for accessibility and those who don\'t will
also become gray. Eventually, the concept of separate disability
accessibility may fade away.

2.  What are the overarching usability and accessibility concerns and
    their intersection?

Generally, the technology needed to make a computerized voting system
accessible is minor compared to the rest of the system, assuming that it
is built into the design from the beginning and not tacked on as an
accessibility Band-Aid, long after the rest of the voting system design
is finished.

It would usually make more sense to design a new voting system to have a
single model that serves for the general public and as an \"accessible
voting system.\" Making a single voting system available, instead of
two, would decrease manufacturing costs, reduce maintenance costs,
simplify voter and pollworker training, boost the reliability of
accessible systems, and have a host of other advantages.

Also, like curb cuts, features that improve accessibility often have
unexpected advantages in improved usability for the general public.

3.  Do the accessibility requirements in the Draft TGDC VVSG allow
    individuals with disabilities to vote independently?

Generally, no. Some clarification of \"independence\" is necessary.
Personal independence should more accurately be referred to as
\"autonomy\".

Many folks are carelessly insisting on privacy and independence
throughout the voting process. For many voters with disabilities, voting
with such perfect personal independence is likely to never be possible.
There are many portions of the whole voting experience that require
personal independence (more rightly autonomy) to assure the privacy of
the voter's ballot. However, there are other portions of the voting
experience, such as signing in on the poll book, that do not require
independence to assure the privacy of one\'s vote.

Personal independence in configuring the voting station to accommodate
one\'s accessible interface needs can be important for maintaining the
privacy of one\'s personal disabilities or language choices.

Personal independence in the process of casting a marked ballot may not
be necessary for ballot privacy, if that privacy is assured with privacy
sleeves and proper assistance procedures.

Personal independence throughout the voting experience is also valued
for improving voter self esteem, reduction of demeaning treatment, and
other factors such as reduction of frustrating waits for assistance from
busy pollworkers.

Although the verifying research has not been done, many voters with
disabilities assume that complete personal independence would increase
the efficiency and effectiveness of their voting experience. This is
perhaps true for certain portions of the voting experience. However,
from my own experiences trying to vote independently, it is
substantially more efficient for me to have an assistant vote for me, as
I did before electronic voting machines.

Hence, it is important to consider personal independence, personal
privacy, and ballot privacy as separate concerns that should not be
casually lumped together when referring to voting systems as \"private
and independent\" or \"independent\".

A separate sense of the word "independence" is now being used in the
voting field to refer to the independence of portions of voting
processes from the use of software or other high tech systems that
cannot be simply and transparently observed directly by the human senses
(primarily eyesight). Use of software independence has been proposed and
promoted to improve the transparency of the voter\'s verification of
marked ballots. With the exception of braille ballots, tactile sleeve
manual ballot markers with verification wands, and the MIT VAAT
proposal, there are currently not many technologies that can support
software independent ballot verification. Additionally, none of these
low tech, software independent technologies currently meet the
accommodation needs of the wide variety of voter disabilities that are
legally required to be met.

An additional sense in which \"independence\" is used in the voting
field, is with regards to isolation of modules or separate processes of
voting systems. The modules for ballot marking and for ballot
verification may need to be kept independent or isolated to improve
reliability and voter confidence.

As currently worded, the measures required in the draft VVSG for
assuring that voters with disabilities can have personal independence
and privacy in their verification of paper vote records would require
several quantum leaps in technology development. The draft VVSG goes
unreasonably far overboard in apparently requiring that paper record
verification for voters with disabilities and alternative language needs
must carry out advanced OCR, autonomous ballot parsing and format
extraction, and translation of languages other than English.

This seemingly desirable super-verification system for voters with
disabilities would require software and therefore not be software
independent.

Writing a requirement like this into the VVSG is somewhat like requiring
a similarly desirable goal of converting all of our energy generation to
fusion power plants within four years.

These requirements for complete personal independence in paper ballot
record verification are so technologically far into the future and
impractical in the near term that the effect of requiring them in the
VVSG would be to simply ban the use of paper ballot records systems.

4.  How do we obtain qualified usability and accessibility testers?

First we need to define the types of tests that are needed, before we
can start to define the skill sets needed to perform that testing. It
would appear that testers with different approaches will be needed. In
our California TTB review, we used a combination of experts to perform
heuristic testing, and subjects to help with separate qualitative and
quantitative testing. The four expert testers had varied backgrounds in
usability and accessibility, which turned out to add a valuable richness
to the results.

5.  Would component testing and certification assist in bringing new and
    better assistive technology to voting systems? What technologies
    have you seen that would be useful for voting?

Component Certification:

There could be a new class called \"Certified Component\".

A certified component should have no interaction with the voting system
software. For example, it should not be able to upload its own drivers
or other software into the voting system through a USB port.

Certified Components must have fully defined interface standards. Should
these interface standards be required to be public?

Some definition of what could be considered a Certified Component is
needed. Would it include COTS laser printers, scanners, wheelchair
accessible tables and stands, keypads, dual-switch input controls,
telephones, headphones, CCTV or digital video image magnifiers,
uninterruptible power sources, smart cards, E-pollbooks, etc.?

What kind of testing would be necessary before a vendor could claim that
their system is compatible with a specific Certified Component. For
example, standard dual-switch input controls might be expected to be
certified as components, yet at least one well known voting system will
not accept the standard right-angle miniature phone plugs that are used
on many of the commonly available dual-switch input control devices.
Another well known voting system is only compatible with dual-switch
input controls that have right-angle plugs.

Would the Certified Component classification be needed for permitting
the use of certain PADs (Personal Access Devices) that might otherwise
be considered a possible security risk, for example a refreshable
braille display unit using a USB interface?

6.  How can cognitive disabilities be better addressed in the standard?

There is a rich body of knowledge about computer user interfaces that
improve access for users with cognitive impairments. Perhaps it has been
mostly ignored in the voting system design community because of lack of
awareness among both designers and advocates of voting systems.

Here are some fairly well known examples of cognitive disability
oriented design concepts.

-   Visual displays should have a clear margin around text areas. When
    text runs directly up to the edge of a screen, for example, the
    human brain has to do a lot of extra processing to handle the
    possibly-hidden-text conflict. This causes most readers a little,
    usually unnoticed, trouble. However, many voters with cognitive
    impairments are strongly impacted by possibly hidden portions of a
    text image. Clear margins can help, even when it is just for a
    single word inside a screen button or control box.

-   Systems need more consistency of controls. The basic function of a
    control should not change when the system is in different modes or
    processes.

> For example: Some ATI systems use the left and right arrow keys to
> move race-by-race throughout the ballot during the vote selection mode
> or process, but switch to using the up and down arrow keys to move
> race-by-race in the review or verify modes. Control mode switching
> such as this presents a very heavy cognitive load that easily confuses
> most ATI voters.
>
> Such over-moding overloading is one of the most common flaws that adds
> complexity to and limits the usability of voting systems.

-   A commonly mistaken impression is that fewer keys makes a voting
    system less complicated. Too many keys is a cognitive load problem,
    but reducing a voting system for navigating an audio ballot to just
    three keys does not make it easier for voters, and especially not
    for voters with cognitive impairments. This is because too few keys
    results in the kinds of inconsistencies and over-moding mentioned
    above.

Fortunately, reducing cognitive loading will always improve the
usability for all voters, whether or not they have obvious disabilities.

7.  How can existing interfaces between vendors and the usability and
    accessibility communities be improved for proper design and testing
    of systems?

The EAC could encourage the establishment of a review and consulting
advisory group of usability and accessibility experts who would be
available to help supply vendors direction and feedback from a wide
perspective, not just limited to a single disability. There have been
too many stories of vendors in the voting accessibility field getting
design advice from single individuals or from single disability oriented
organizations. A variety of experts with broad backgrounds is essential
for avoiding misguiding and limiting the vendors. This organization of
UI experts would probably be most effective if they were not also used
to render official opinions on vendors\' final voting systems.

8.  What is your professional assessment of the usability benchmarks,
    their strengths, and needs for improvement?

In addition to the vendors\' subjective VPP testing and benchmarks,
there should be VSTL qualitative testing by usability and accessibility
experts. These tests should use heuristic testing, including persona
walk throughs. This kind of testing can detect many types of usability
issues that would not be observed or reported through the currently
defined vendor VPP or safety lab testing procedures.

The VVSG draft is too vague about who would perform the VPP testing for
accessibility and where and when in the process that would be done.

Benchmarks for accessible voting performance should also be defined.

In-the-field performance benchmarks should be defined and used to revoke
certification of systems that cannot accomplish or maintain performance
in actual elections. Performance should include measures that test how
well pollworkers can manage the set up and support of voting systems.

In my own experience voting on electronic voting machines in six
elections, the voting machines themselves failed 1/3 of the times.
Another 1/3 of the times the pollworkers could not figure out how to get
the systems working properly. I don\'t blame the poll workers. Rather, I
blame the poorly designed and overly complex pollworker interface of the
systems and the poor manufacture and support of the systems.

The new VVSG should set up a method for monitoring and rejecting such
unacceptably ineffective performance in the field.

The current VVSG draft specifies use of a usability VPP test, but it is
much too limited to catch the types of problems I\'ve witnessed in real
polling places. The VVSG draft is also far too vague about the testing
of the pollworker interface. A thorough pollworker VPP must be defined
and implemented to start to find and flag systems that are not going to
be practical in the field.

Another problem with the pollworker interface usability is that it gets
more complicated every time a new problem is discovered and \"fixed\" by
forcing further mitigation procedures on the pollworkers and elections
officials. If mitigation procedures are implemented subsequent to the
certification of a system, the usability of any impacted voter and
pollworker interfaces should be retested with a VPP, in order for the
system to maintain its federal certification status.

It does not appear clear that the VVSG requires that the VPP testing
should include mobility, hearing, or cognitive impairments.
Additionally, it is also not clear that testing of accommodation of
multiple disabilities should be part of the VPP. Examples of some
combinations of disabilities to be tested might be helpful.

Specific comments on VPP benchmarks:

\"3.2.1.1-B Perfect ballot performance

The system SHALL achieve a perfect ballot index of at least 2.33 as
measured by the VPP.\"

Comment: It seems that this index should possibly be more reasonably
expressed as a percentage of the inverse (as num of incorrect over num
of perfect ballots).

\"3.2.1.1-C Voter inclusion performance

The system SHALL achieve a voter inclusion index of at least 0.35 as
measured by the VPP.\"

Comment: A simple definition of how this benchmark is actually
calculated should be given in the requirement.

\"3.2.1.1-D.2 Voting session time

The test lab SHALL report the average voting session time, as measured
by the VPP.

Discussion: This requirement encourages systems to enable voters to vote
with reasonable speed. Note that this requirement does not apply to the
audio interface of a system, or to the use of special input devices for
voters with dexterity disabilities.\"

Comment: Is there to be any threshold value for voting time?

Why shouldn\'t there be separate time measures for voting with access
interfaces such as audio and non-manual inputs?

\"3.2.1.1-D.3 Average voter confidence

The test lab SHALL report the average voter confidence, as measured by
the VPP.\"

Comment: Is there to be any threshold value for average voter confidence
If not, why not??

A ballot design similar to the NIST neutral ballot should be accepted as
a standard for the VPP and VSTL usability/accessibility testing.

In our California TTB review testing, we\'ve found it hard to get
identical local ballots for test comparison of different types of
machines.

NIST has realistic, but fictitious, names and ballot questions, and uses
fictitious instead of real party names.

Some elections researchers from MIT have recently recommended using
ballots with real names from local elections, but this introduces many
biases and voter frustration and/or rebellion against having to vote for
a candidate or party the voter may strongly dislike.

MIT researchers have suggested that the VPP should include enhanced
pollworker training, but that could inadvertently hide and cover up some
of the most important factors that currently effect a voter\'s chances
for successful voting in the polling place.

For example, in a third of my own experiences voting on electronic
voting systems, pollworkers were never able to figure out how to get the
accessible voting system working so I could use it to vote with an audio
ballot.

Using unrealistically skilled and highly trained pollworkers in the VPP
testing could limit and seriously bias the testing, and problems similar
to mine might never be detected during the VPP.

Wasted in-booth time and thumb twiddling frustration time is very
irritating and relevant to voters, but it is discounted for the VPP
testing by some MIT researchers.

The Brennan Center Usability study shows that longer in-booth voting
times appears to be correlated with higher vote selection accuracy.
However, simplified voter satisfaction measures may be negatively
impacted by systems that have lower error rates, if those systems also
increase in-booth time and/or frustrating wait times.

This suggests that meaningful voter satisfaction may not be easy to
measure without the full context that includes voters\' awareness of and
confidence in their vote selection accuracy.

Vote selection accuracy testing needs to separate scoring of uncommon
types of voting behaviors, such as write-in voting, so their errors do
not unreasonably dominate and misrepresent errors in typical voting
behavior.

Accessibility testing measures should include how well a system can
accommodate a wider spectrum of voters with different disabilities and
multiple disabilities.

There should be a standard or canonical voting system required as a
second or control system in every VPP test. The vendor should not be
allowed to pick which second machine to test as the calibration control
machine.

Lab conditions can be controlled to test extremes found in real polling
places. Lab environments don\'t have to be held to \"ideal conditions\".
In fact, a properly controllable lab would be the best place to test the
impact of many environmental extremes of noise, lighting, temperature,
humidity, visual distractions, cramped spaces, etc.

A realistic voting environment should be used for the VPP, not a random
real polling place or one picked by the vendor.

If the VPP environment and staff are under the control of the vendor,
imagine the many ways that a vendor who is desperate to get passing
benchmarks might manipulate the testing\...including placing the control
standard machine in an area with worse lighting, more noise, more visual
distractions, uncomfortably hot or cold, etc. Even when trying to be
unbiased, it is very difficult for test monitors or \"pollworkers\" to
avoid communicating biases for or against a particular system with voice
tone, body language, and other forms of communication.