A Consideration of Voting Accessibility for

Injured OIF/OEF Service Members: Needs

Assessment

July 2012

Contract E4064914

2010 Voting Technology and Accessibility Research – Military Heroes Initiative 

CFDA #90.403

PREPARED FOR:

Election Assistance Commission 

1201 New York Avenue, N.W. 

Suite 300

Washington, D.C. 20005

PREPARED BY:

Human Systems Integration Division 

Electronic Systems Laboratory 

Georgia Tech Research Institute 

Georgia Institute of Technology 

Atlanta, Georgia 30332
Annotated Literature Review Results

Usability of Voting Systems

Arzt-Mergemeier, J., Beiss, W., & Steffens, T. (2007). The digital voting pen at the Hamburg Elections 

2008: Electronic voting closest to conventional voting. Paper presented at The International Conference 

on E-voting and Identity, Bochum, Germany.

The city of Hamburg revised its election laws so that each voter has more than one vote making 

the voting process more complicated. Tallying all of the votes would be complicated and time 

consuming using traditional ballots, so an electronic voting system called the Digital Voting Pen 

System (“Digitales Wahlstift-System”, or DWS) is proposed. The DWS was selected due to its 

security, verifiability, and closeness to the conventional voting procedure and therefore its 

acceptance among voters. The digital pen is used like an ordinary pen but contains a

microprocessor that scans the marks the user places on a special piece of paper with a specific dot

pattern printed in the background. The paper booklet is collected along with the electronic vote

for spot tests or, as needed, in depth verification of polling results.

Bederson, B. B., Lee, B., Sherman, R. M., Herrnson, P. S., & Niemi, R. G. (2003). Electronic voting system 

usability issues. Paper presented at the SIGCHI conference on Human Factors in Computing Systems, Ft. 

Lauderdale, FL.

Usability problems associated with paper and mechanical voting devices were brought to the 

public eye after the elections of 2000. In response, many precincts nationwide have begun to 

use electronic voting machines. However, electronic voting machines have usability problems as 

well. This paper discusses benefits and drawbacks of both kinds of voting machines, as well as 

the issue of technology acceptance by different demographic groups. Some groups, such as 

older adults and those of lower socioeconomic status, are less familiar with electronic devices, 

and may therefore less readily accept the new technology. They also may experience 

significantly more difficulty when using the new technology.

Bederson, B. B. & Sherman, R. M. (2002). Usability review of the Diebold DRE system for four counties in 

the state of Maryland. Retrieved from www.capc.umd.edu/rpts/MD_EVoteMach.pdf.

The results of a usability evaluation of the Diebold AccuVote-TS voting system are reported. The 

system, a direct reported electronic voting system, was evaluated using three methods: expert 

review, close-up observation, and field testing. The system exhibited both strength and 

weaknesses. Some of the weaknesses included the absence of a dedicated help button, unclear 

and long help instructions, and inconsistent terminology. Although most of the users responded 

favorably to the system, some users did report a need for improvements.
Byrne, M. D., Greene, K. K., & Everett, S. P. (2007). Usability of voting systems: Baselines data for paper 

punch cards, and lever machines. Paper presented at CHI 2007, San Jose, CA.

Computer-based direct recording electronic (DRE) voting systems are being adopted by 

jurisdictions in the United States to replace older technologies such as paper ballots, punch 

cards, and lever machines. The computer-based DRE systems could potentially improve the 

usability and accessibility of the voting process, but the only way to know if these systems do 

improve usability is to have baseline data on the usability of the older technologies. The authors 

performed an experiment to assess the usability of punch cards, lever machines, and two forms 

of paper ballots (a bubble ballot and an arrow ballot). They found that while there was no 

significant difference in the amount of time required to complete the ballots, the four methods 

did vary in error rates with users making the fewest errors when using paper ballots. 

Additionally, the bubble ballot was seen as the most usable by individuals who participated in 

the study.

Campbell, B. A. & Byrne, M. D. (2009). Now do voters notice review screen anomalies? A look at voting 

system usability. Paper presented at the 2009 Electronic Voting Technology Workshop / Workshop on 

Trustworthy Elections, Montreal, Canada.

The research summarized in this article builds on previous research indicating that 

approximately two thirds of voters do not notice discrepancies between the review screen on an 

electronic voting machine and the selections they intended. In the previous study, the 

instructions did not emphasize to voters the need for verification. The current study improved 

the instructions by explicitly instructing voters to review their choices before casting their ballot. 

Additionally, the review screen was redesigned so that it included party information and made 

undervotes more visually salient. The changes made to the instructions and interface did 

increase the detection rate but only moderately to 50%.

Conrad, F. G. (n.d.). Usability and voting technology. White paper for Voting Technology Workshop.

Usability testing assesses speed, accuracy, and user satisfaction. It has a long history of use and 

domains other than voting technology. Lessons learned from these other domains should be 

applied to the assessment of emerging voting technologies. Practical issues such as the 

participant recruitment and development of metrics are discussed. The paper concludes with a 

critical evaluation of several usability claims made by voting machine manufacturers.

Everett, S. P., Greene, K. K., Byrne, M. D., Wallach, D. S., Derr, K., Sandler, D., & Torous, T. (2008). 

Electronic voting machines versus traditional methods: Improved preference, similar performance. Paper 

presented at CHI 2008, Florence, Italy.

The usability of a computer-based direct recording electronic (DRE) voting system was 

compared to more traditional voting techniques including paper ballots, punch cards, and lever 

machines. The usability of each method was determined using efficiency, effectiveness, and user 

satisfaction metrics. The results indicate that there is little difference between the DRE and the 

traditional methods in efficiency (time to completion of ballot) and efficacy (error rate). 
However, the DRE was reliably superior to the traditional methods in terms of user satisfaction. 

These results indicate a disconnect between the subjective and objective usability of DREs.

Greene, K. K., Byrne, M. D., & Everett, S. P. (2006). A comparison of usability between voting methods. 

Paper presented at the 2006 USENIX/ACCURATE Electronic Voting Technology Workshop, Vancouver, 

Canada.

There is no baseline usability data for traditional voting methods, such as lever machines and 

paper ballots, to compare the new direct recording electronic (DRE) voting system to. In this 

study, the usability of three traditional voting methods was compared. The efficiency, 

effectiveness, and user satisfaction of two paper-based ballots (one open-response and one 

bubble ballot) and one mechanical lever machine were measured. The efficiency (time to 

completion of ballot) and efficacy (error rate) of the three methods did not significantly differ. 

However, there were significant differences in user satisfaction for the three methods.

Participants reported that they were most satisfied with the bubble ballot and least satisfied 

with the lever machine.

Herrnson, P. S., Niemi, R. G., Hanmer, M. J., Bederson, B. B., Conrad, F. C., & Traugott, M. W. (2006). The 

importance of usability testing of voting systems. Paper presented at the 2006 USENIX/ACCURATE 

Electronic Voting Technology Workshop, Vancouver, Canada.

The usability of six voting systems was assessed by expert reviews, a laboratory test, and a field 

study. Across the six systems, usability problems were identified in signing-on to the machines, 

navigating the ballot, reviewing and changing votes, marking write-in candidates, and casting 

the ballot. In the field study, most voters ignored the paper trails provided by the machines. 

Although most voters were generally able to vote as intended both inside and outside the 

laboratory, certain activities proved to be problematic; these included changing votes, detecting 

errors, and making a straight-party vote. The types of errors that occurred most frequently 

differed across the voting machines, suggesting that each machine had unique deficiencies.

Herrnson, P. S., Niemi, R. G., Hanmer, M. J., Bederson, B. B., Conrad, F. C., & Traugott, M. W. (2008). 

Voting technology: The not-so-simple act of casting a ballot. Washington, DC: Brookings Institution 

Press.

The authors evaluated five commercially available voting systems and one prototype system. 

Evaluation metrics included ease of use, speed, and accuracy. Expert reviews and observations 

of users indicated that mechanical interfaces, such as scroll wheels and buttons, and interfaces 

that present the entire ballot at once pose greater physical and mental challenges.

The authors recommend that usability testing include poll worker operations, including voting 

machine setup, maintenance, and shut down. These tasks can be unclear and difficult, and 

recovery from mistakes is often complicated.
Herrnson, P. S., Bederson, B. B., Lee, B., Francia, P. L., Sherman, R. M., Conrad, F. C., . . . Niemi, R. G. 

(2005). Early appraisals of electronic voting. Social Science Computer Review, 23(3), 274-292.

In response to the problems in the 2000 U.S. elections, new electronic voting machines have 

become more popular because of the speed and accuracy with which they are able to record 

and tabulate votes. However, there is little evidence about the interface between the voter and 

the voting system to support this transition. Some of the concerns include: the accessibility of 

the system; the effects of age and technical experience on usability; the potential for bias; the 

costs associated with purchasing, evaluating, and maintaining new systems as well as training 

election workers on the new technology; and the security of electronic voting machines. The 

authors evaluated the Diebold AccuVote-TS machine in a series of studies involving expert 

review, close observation, a field test, and an exit poll. The results of the study indicated that, 

while the systems worked well, there were several issues related to usability and useracceptance that need to be addressed.

Norden, L., Kimball, D., Quesenbery, W., & Chen, M. (2008). Better ballots. New York: New York 

University School of Law, The Brennan Center for Justice.

The authors provide substantial evidence that a large number of voters have been 

disenfranchised by poor ballot design. They document empirical evidence from past elections to 

support their case. The solution, of course, is to produce ballots that are more simple and easy 

to use. The report outlines ballot design problems, laws that interfere with good design and 

usability, and policy recommendations and directionsfor the future. A checklist for evaluating 

ballot usability is also provided.

Sherman, A. T., Gangopadhya, A., Holden, S. H., Karabatis, G., Koru, A. G., Law, C. M., . . . Zhang, D. 

(2006). An examination of vote verification technologies: Findings and experiences from the Maryland 

study. Paper presented at the 2006 USENIX/ACCURATE Electronic Voting Technology Workshop, 

Vancouver, Canada.

A technical review of vote verification systems was conducted for the Maryland State Board of 

Elections. The vote verification and post-election auditing capabilities of five touchscreen vote 

verification machines, to be used in conjunction with a ballot machine, were evaluated. Other 

secondary considerations of the evaluation included general functionality, security, reliability, 

accessibility, and privacy. The evaluation results concluded that the currently used Diebold 

voting system (ballot) could provide better vote verification by being integrated with any of the 

four vote verification systems. Only small modifications would be needed for integration of the 

voting machine with the vote verification machine.

Yee, K. P. (2007). Extending prerendered-interface voting software to support accessibility and other 

ballot features. Paper presented at the 2007 USENIX/ACCURATE Electronic Voting Technology 

Workshop, Boston, MA.

A pre-rendered user interface (PRUI) can be designed and audited independently of the voting 

machine manufacturers. This paper describes software to be used on a PRUI system in 

conjunction with accessible technologies, including touchscreen interfaces, alternate input 
devices, and concurrent speech output (i.e., synchronized audio and visual displays). A 

prototype was developed in Python.

Yee, K. P., Wagner, D., Hearst, M., & Bellovin, S. M. (2006). Prerendered user interfaces for higherassurance electronic voting. Paper presented at the 2006 USENIX/ACCURATE Electronic Voting 

Technology Workshop, Vancouver, Canada.

The authors developed a software program, which they call a pre-rendered user interface 

(PRUI), but can be published before election day. The PRUI is an electronic sample ballot that 

could enable the general public to participate in the verification, usability testing, and 

accessibility testing of the ballot to be used on election day. Problems that are identified by the 

public could be remedied before the election. By preparing the user interface apart from the

voting machine, the difficulty of software verification is greatly reduced. The prototype 

developed by the authors supports a variety of user interface styles.