# Presentation: 519179

## Improving the Usability and Accessibility of Voting Systems and Products

- Dr. Sharon Laskowski
- _[sharon.laskowski@nist.gov](mailto:sharon.laskowski@nist.gov)_
- July 9, 2004
- TDGC Meeting

## Slide 2

## Human Factors/Usability Perspective on Voting Systems: Voters 

- Cognitive and physical nature of the voters
- Physical environment
- Psychological environment
- Voting product
- Usability is determined by the demands of the system and the voter’s ability to perform under those demands

**Notes:**
What do we mean by a human factors and the usability perspective on voting systems?

This takes into consideration ...

Psych—stress of having a long line of people behind you or asking for help. 

...and in this case voters successfully casting a vote the way they intended.

## Measuring Accessibility and Usability

- Accessibility
- The degree to which a system is available to and usable by individuals with disabilities
- Usability
- A measure of the effectiveness, efficiency, and satisfaction achieved by a specified set of users performing specified tasks with a given product
- Metrics: errors causing a vote cast not as intended or a vote not cast,  (errors prior to success), and time to cast vote
- Designing and measuring process
- User-centered design
- Diagnostic usability evaluation
- Testing performance—usability testing

**Notes:**
In order to show improvements in accessibility and usability, you must be able to measure these qualities.  These definitions are useful:

Accessibility...     note that the usability is part of good access.
Usability... 

(errors prior to success not necessary a problem)

There are different stages  in design and development where you can measure, for example:
User-centered design techniques can be applied.  As development progresses, various measurement techniques ranging from expert evaluation to testing with small numbers of users can be employed
An end product can be tested for performance against the metrics for eff., effic and satisfaction with usability testing with typical sets of users.  
This is sometimes called summative testing.  At NIST we worked with industry to develop a standard report format called the CIF which is now an ANSI/INCITS standard 354-2001 based on best practices in the industry.

## State of Usability of US Voting Systems

- In general, voting systems have not been measured for usability nor have they been developed using a user-centered design process
- We do not know the degree to which voters cast their vote NOT as they intended due to confusion with the user interface
- Note observations by , CalTech/MIT Voting Technology Project, Herrnson et.al., and others such as Doug Jones

## Design and Performance Standards

- Design Standards—how the product is designed
- For example,  font size, ballot instructions
- Performance Standards—how the product functions
- No overvoting, test by demonstration
- Time to cast vote, failures in casting vote as intended
- Requires: measuring with users against benchmarks,
- Sample ballots of different complexity, and
- Well-defined test protocols and user groups

**Notes:**
There are 2 types of standards:

Design, test by inspection, demonstration
Performance, test by demonstration, operation

...

These do not currently exist.

So, as you can see, we have a lot of issues to discuss about how to design and test for improved accessibility and usability.

## We currently cannot measure usability of voting systems
E.g., select/deselect
Need high degree of usability 
Following design guidelines does not necessarily insure usability
Usability engineering provides measurement methods, but not necessarily to the degree we need specifically for voting

We need standards and conformance tests that do measure degree of usability and accessibility, if systems are going to be qualified and certified for usability and accessibility

- E.g., select/deselect
- Need high degree of usability
- Following design guidelines does not necessarily insure usability
- Usability engineering provides measurement methods, but not necessarily to the degree we need specifically for voting
- We need standards and conformance tests that do measure degree of usability and accessibility, if systems are going to be qualified and certified for usability and accessibility

- Measurement for Qualification and Certification

- rigorous
- research &
- experiments

- informal
- evaluation

- Easy, variable

- Complex, reliable

- feasible
- reproducible

- conformance testing

## Current Voting Standards and Testing

- Current VSS has some accessibility standards, but only a usability appendix
- ITAs currently perform qualification tests
- Can we test for usability and accessibility?
- Standards must be clear, unambiguous, testable
- Requires procedures for testing the voting product against the standards (conformance testing)
- For example, inspection , demonstration, operation
- IEEE P1583 draft standards
- Task Group 3 has made some progress
- But, lack of resources and small vendor base have been a barrier to developing standards that are performance-based standards, benchmarks and conformance tests

## The HF Report

- Our report recommends an approach that will produce measurable voting systems standards for usability and accessibility
- Doesn’t need a lot of research
- Does need:
- Expertise in conformance test development
- Some applied research to develop user testing protocols
- Neutral third parties to accomplish this
- No cheap, quick fixes
- Could require some usability testing to avoid major usability blunders, but this is no guarantee

## 10 Recommendations

- Performance-based, high-level usability standards
- Complete set of user-related functional requirements
- Avoid low-level design specifications. Use only product design requirements that have been validated as necessary
- Applied research to support the development of usability and accessibility standards
- Review current requirements (Access Board, the current VSS, draft IEEE standards) for possible adoption
- Ballot design guidelines
- Guidelines for facility and equipment layout; design and usability testing guidelines for vendor- and state-supplied documentation and training materials
- Vendors should incorporate a user-centered design approach
- Conformance tests for voting products against the applicable accessibility requirements.
- Valid, reliable, repeatable, and reproducible process for usability conformance testing of voting products against the standards described in recommendation 1) with agreed upon usability pass/fail requirements.

## Most Critical Need

- A set of usability standards for voting systems that are performance-based, with
- Objective measures
- Conformance test procedures
- Then voting products and systems can be certified that they meet the usability standards
- This is the only way to guarantee high levels of usability

## RoadMap(Details in Report)

- Short term: encourage usability and user-centered design
- Long term:
- Use best of IEEE and other standards
- ...and ballot design guidance
- Develop user test procedures
- Collect user data to define performance baselines
- Develop performance standards and conformance tests