Can Voters Detect Malicious Manipulationof Ballot Marking Devices?
Matthew Bernhard, Allison McDonald, Henry Meng, Jensen Hwa, Nakul Bajaj∗ , Kevin Chang, J. Alex Halderman
University of Michigan∗ The Harker School

However, BMDs do not eliminate the risk of vote-stealing
attacks. Malware could infect the ballot scanners and change
the electronic tallies — although this could be detected byrigorously auditing the paper ballots [50] — or it could infect
the BMDs themselves and alter what gets printed on the ballots.
This latter variety of cheating cannot be detected by a postelection audit, since the paper trail itself would be wrong, andit cannot be ruled out by pre-election or parallel testing [51].
Instead, BMD security relies on voters themselves detecting
such an attack. This type of human-in-the-loop security isnecessary in many systems where detection and prevention of
security hazards cannot be automated [18]. However, as several
commentators have recently pointed out [7], [20], [51], its
effectiveness in the context of BMDs has not been established.
Whether such a misprinting attack would succeed without
detection is highly sensitive to how well voters verify theirprinted ballots. Every voter who notices that their ballot ismisprinted and asks to correct it both adds to the evidence
that there is a problem and requires the attacker to change an
additional ballot in order to overcome the margin of victory.
I. I NTRODUCTION
Consider a contest with a 1% margin in which each polling
The threat of election hacking by hostile nations has place has 1000 voters. If voters correct 20% of misprinted
prompted a major push to ensure that all voting systems in the ballots, minimal outcome-changing fraud will result in an
United States have voter-verifiable paper trails, a defense rec- average of 1.25 voter complaints per polling place — likely tooommended by the National Academies [36], the Senate Select few to raise alarms. If, instead, voters correct 80% of misprinted
Committee on Intelligence [53], and nearly all election security ballots, polling places will see an average of 20 complaints,
experts. Guided by past research [8], some states and localities potentially prompting an investigation. (We model these effects
are implementing paper trails by deploying ballot-marking in Section V.) Despite this sensitivity, voters’ BMD verification
devices (BMDs). In these systems, the voter makes selections performance has never before been experimentally measured.
In this paper, we study whether voters can play a role inon a computer kiosk, which prints a paper ballot that the voter
can review before inserting it into a computer scanner to be BMD security. We first seek to establish, in a realistic pollingcounted [56]. BMDs have long been used as assistive devices place environment, the rates at which voters attempt to verify
for voters with disabilities, and a growing number of jurisdic- their printed ballots and successfully detect and report malicious
tions are purchasing them for use by all voters [24], [25], [37]. changes. To measure these, we used real touch-screen voting
BMDs have the potential to provide better security than machines that we modified to operate as malicious BMDs. We
direct-recording electronic voting machines (DREs), which recruited 241 participants in Ann Arbor, Michigan, and had
maintain the primary record of the voter’s selections in a them vote in a realistic mock polling place using the ballot
computer database and often lack a voter-verifiable paper trail. from the city’s recent midterm election. On every ballot that
Numerous studies have demonstrated vulnerabilities in DREs our BMDs printed, one race was changed so the printout did
that could be exploited to change election results (e.g., [11], not reflect the selection made by the participant.
[23], [31], [35]). In contrast, BMDs produce a physical record
We found that, absent interventions, only 40% of participantsof every vote that can, in principle, be verified by the voter reviewed their printed ballots at all, only 6.6% reported the error
and manually audited by officials to confirm or correct the to a poll worker, and only 7.8% correctly identified it on an exit
initial electronic results.
survey. These results accord with prior studies that found poor

Abstract — Ballot marking devices (BMDs) allow voters to
select candidates on a computer kiosk, which prints a paper
ballot that the voter can review before inserting it into a scanner
to be tabulated. Unlike paperless voting machines, BMDs provide
voters an opportunity to verify an auditable physical recordof their choices, and a growing number of U.S. jurisdictions
are adopting them for all voters. However, the security of
BMDs depends on how reliably voters notice and correct any
adversarially induced errors on their printed ballots. In order tomeasure voters’ error detection abilities, we conducted a large
study (N = 241) in a realistic polling place setting using realvoting machines that we modified to introduce an error into
each printout. Without intervention, only 40% of participants
reviewed their printed ballots at all, and only 6.6% told a poll
worker something was wrong. We also find that carefully designed
interventions can improve verification performance. Verbally
instructing voters to review the printouts and providing a written
slate of candidates for whom to vote both significantly increased
review and reporting rates — although the improvements may
not be large enough to provide strong security in close elections,especially when BMDs are used by all voters. Based on thesefindings, we make several evidence-based recommendations to
help better defend BMD-based elections.

voter performance in other election security contexts, such as
II. BACKGROUND AND R ELATED W ORK
DRE review screens [1], [15] and voter-verifiable paper audit A. Human-Dependent Security
trails (VVPATs) [48]. The low rate of error detection indicatesElections fundamentally depend on having humans in thethat misprinting attacks on BMDs pose a serious risk.
loop— as Stark [51] notes, the voter is the only one who
The risks notwithstanding, BMDs do offer practical advanknows
whether the ballot represents their intended vote — andtages compared to hand-marked paper ballots. They allow
thesuccessor failure of election security has the potential
voters of all abilities to vote in the same manner, provide atohave
history-altering
effects. The type of risk posited bymore user-friendly interface for voting, and more easily support
Stark,
whereinvotersdonotcheck their paper ballots to ensurecomplex elections like those conducted in multiple languages or
the
BMD
has
correctly
represented
their selections, is a postwith methods such as ranked choice [44]. BMDs also simplify
completion
error
[14],
in
which
a
user
makes a mistake (orelection administration in places that use vote centers [56],failstoverifythecorrectnessofsomething)after they have
which have been shown to reduce election costs and lower provicompleted
the
main
goal
of
their
task.
Voterswhoforget or do
sional voting rates [28], [42], as well as in jurisdictions that emnot
know
toverifythecorrectness
of
a
paper
ballotafter theyploy early voting, which can improve access to the ballot [30].
haveenteredtheirselections
on
a
BMD
miss
a
criticalstep in
Given these advantages and the fact that BMDs are alreadyensuring
theaccuracy
of
their
vote.
We
therefore
explorehow
in use, the second goal of our study was to determine whether
tocommunicate
this
risk
to
voters.
it might be possible to boost verification performance through
Cranor [18] describes five ways that designers can commuprocedural changes. We tested a wide range of interventions,
nicaterisk to a user who needs to make security decisions:such as poll worker direction, instructional signage, and usage
Warnings:
indication the user should take immediate action
1)of a written slate of choices by each voter.
2)
Notices:
information
to allow the user to make a decision
The rate of error detection varied widely with the type of3)
Statusindicators:
indicationof the status of the systemintervention we applied, ranging from 6.7% to 86% in different
4)Training:
informing
users
about
risks and mitigations
experiments. Several interventions boosted review rates andbeforeinteraction
discrepancy reporting. Verbally encouraging participants to5) Policies: rules with which users are expected to comply
review their printed ballot after voting boosted the detection rate
to 14% on average. Using post-voting verbal instructions while
Implementing indicators that reveal meaningful informationencouraging participants to vote a provided list of candidates to voters about the security status of a BMD would be next toraised the rate at which voters reported problems to 73% for impossible, as security issues are often unknown or unforeseen
voters who did not deviate from the provided slate.
to the operators. Although voter education about the importance
These findings suggest that well designed procedures can of verification might be an effective form of training, significant
have a sizable impact on the real-world effectiveness of voter coordination would be necessary to enact such a scheme atverification. We make several recommendations that election scale. Therefore, we focus in this study on the effectiveness ofofficials who already oversee voting on BMDs can employ warnings issued through poll worker scripts and polling place
immediately, including asking voters if they have reviewed signage.
their ballots before submission, promoting the use of slates
A warning serves two purposes: to alert users to a hazard, andduring the voting process, informing voters that if they find an to change their behavior to account for the hazard [62]. Thereerror in the printout they can correct it, and tracking the rate are many barriers to humans correctly and completely heedingof reported errors. Our recommendations echo similar findings security warnings. Wogalter proposes the Communicationabout the most effective ways to alert users to other security Human Information Processing (C-HIP) Model [61] to syshazards (i.e., in context [12] and with active alerts [21]) and tematically identify the process an individual must go throughredirect them to take action.
for a warning to be effective. The warning must capture andAlthough our findings may be encouraging, we strongly cau- maintain attention, which may be difficult for voters whotion that much additional research is necessary before it can be are attempting to navigate the voting process as quickly asconcluded that any combination of procedures actually achieves possible. Warnings must also be comprehensible, communicate
high verification performance in real elections. Until BMDs the risks and consequences, be consistent with the individual’s
are shown to be effectively verifiable during real-world use, the beliefs and attitudes toward the risk, and motivate the individual
safest course for security is to prefer hand-marked paper ballots. to change — all of which are substantial impediments in an
Road Map Section II provides more background about human environment with little to no user training and such a broad
factors and security and about previous work studying the role user base as voting.
of voter verification in election security. Section III describes
To maximize effectiveness, warnings should be contextual,
our experimental setup, voting equipment, and study design. containing as little information as necessary to convey the risk
Section IV presents our results and analyzes their significance. and direct individuals to correct behavior [12], [61]. Voters
Section V provides a quantitative model for BMD verification are essentially election security novices; Bravo-Lillo et al. [12]security. Section VI discusses the results, avenues for future found that, in the context of computer security, advanced andwork, and recommendations for improving the verifiability of novice users respond to warnings differently. Most significantly,
BMDs. We conclude in Section VII.
novice users assessed the hazard after taking action, whereas2advanced users assessed the hazard before engaging in theactivity.
There may be effective ways to improve voter verification
performance. Many studies have applied lessons from Cranor,
Wogalter, and Bravo-Lillo et al. to help humans make secure
choices in different contexts, including phishing [21], [41],browser warnings [2], [46], [52], app permissions [3], [40],and operating system interfaces [13]. In the context of phishingwarnings, for example, Egelman et al. [21] found that users
were far more likely to heed an active warning, or a warningthat disrupted their workflow, than a passive warning. Thissuggests that similar interventions applied in a polling place
may have a significant effect on voters’ ability to review andverify their BMD ballots.
Our study contributes to this literature by exploring theeffects of several modalities of warnings (oral and visual) on
human detection of malicious ballot modification.

BMDs statewide [24], [25], as have several counties and cities,
including Los Angeles County, the largest single election
jurisdiction in the U.S. [58].
There has been vigorous debate among election security
experts as to whether BMDs can provide software-independence(e.g., [7], [20], [51], [60]). However, the discussion has yet to
be informed by rigorous experimental data. Our work seeks tofill that gap by contributing the first human-subjects study to
directly measure the verification performance of voters using
BMDs under realistic conditions and with a variety of potential
procedural interventions.
III. M ATERIALS AND M ETHODSOur goals in this work were to empirically assess how well
voters verify BMD ballots and whether there are steps election
officials can take that will enhance verification performance.
To these ends, we conducted a between-subjects study where
we tested several hypotheses in a simulated polling place,following the best practices recommended by Olembo et al. [39]for election human-factors research. The study design wasapproved by our IRB.
We sought to answer several questions, all of which concern
the rate at which voters are able to detect that a BMD-printed
ballot shows different selections than those the voter picked:• What is the base rate of error detection?
• Is error detection impacted by:– Ballot style?
– Manipulation strategy?
– The manipulated race’s position on the ballot?
– Signage instructing voters to review their ballots?
– Poll worker instructions?
– Providing a slate of candidates for whom to vote?
In order to answer these questions in an ecologically valid
way, we attempted to create an environment that closely
resembled a real polling place. Nevertheless, it is impossible
for any experiment to fully recreate what is at stake for votersin a real election, and so study participants may have behaved
differently than voters do in live election settings. We went toextensive lengths to mitigate this limitation, and we find somedata to support that we did so successfully (see Section VI-A).
We used real (though modified) voting machines, printers andpaper stock from deployed BMD systems, a ballot from a realelection, and ballot styles from two models of BMDs. We
conducted the study at two city library locations, one of which
is used as a polling place during real elections.

B. Voter-Verifiable Paper and Ballot-Marking Devices

A guiding principle in election security is that voting systems
should be software independent [47]: that is, any software errors
or attacks that change the reported election outcome should be
detectable. Bernhard et al. [9] note that elections backed bya voter-verifiable paper record are currently the only known
way to provide robust software independence. Like BMDs,
voter-verifiable paper audit trails (VVPATs) and hand-marked
paper ballots are widely used in an attempt to achieve software
independence. However, each poses a different set of usability
and accessibility challenges.
Hand-marked paper ballots record the voter’s selections
without the risk of having a potentially compromised computer
mediating the process. However, voters often make mistakes
when filling out ballots by hand that can lead to them being
counted incorrectly or ruled invalid [27]. Moreover, many
voters have difficulty marking a paper ballot by hand due
to a disability or a language barrier. Ballots in the U.S. areamong the most complex in the world, further magnifying
these difficulties [38].
VVPAT technology also suffers from noted usability, privacy,
and auditability problems [26]. Most implementations consistof clunky printer attachments for DREs that are difficult for
voters to read, record votes in the order in which they are
cast, and use a fragile paper tape. In laboratory studies, Selker
et al. [48] and de Jong et al. [19] found that voters frequently
did not review the VVPAT, with Selker finding that only 17%
of voters detected changes between the selections they madeon the DRE and those printed on the VVPAT. While there has
been some criticism of Selker’s findings and methodology [45], A. The Polling Place[49], their results broadly comport with work by Campbell
To provide a realistic voting experience, we structured ouret al. [15] and Acemyan et al. [1] about voters’ ability to detect simulated polling place like a typical BMD-based poll site.
errors introduced in DRE review screens. The latter found that Three investigators served as poll workers, following the script
only 12–40% of participants successfully detected such errors. in Appendix A. Library patrons who were interested in voting
In part due to the concerns raised by these studies, BMDs began at a check-in table, where they were greeted by Poll
have become a popular choice for new voting system de- Worker A and asked to sign an IRB-approved consent form. Parployments in the United States. South Carolina and Georgia, ticipants were told they would be taking part in “a study about
together comprising nearly 9 million voters, recently adopted the usability of a new type of voting machine” and instructed3

Fig. 1: Polling Place Setup. We established mock polling places at two public libraries in Ann Arbor, Michigan, with three
BMDs (left) and an optical scanner and ballot box (right). Library visitors were invited to participate in a study about a new kindof election technology. The BMDs were DRE voting machines that we modified to function as malicious ballot marking devices.

on how to use the equipment, but they were not alerted that the ally a one- or two-person line formed as participants waited
study concerned security or that the BMDs might malfunction. to use the BMDs or the ballot scanner.
Each participant received a voter access card with whichto activate a BMD and was free to choose any unoccupied B. The Voting Machines
machine. There were three identical BMDs, as shown in
BMD voting systems are currently produced by several votFigure 1. On the last day of the study, one machine’s memoryingmachine manufacturers, the largest of which is ES&S. Over
became corrupted, and it was removed from service; all votesasix
month period, we repeatedly attempted to engage ES&S
that day were recorded on the other two machines.
indiscussions
about acquiring samples of their equipment for
The BMDs displayed contests in a fixed order, and voters
thisstudy.
However,these attempts were ultimately not fruitful.
made selections using a touch screen interface. After the lastInstead,weutilized
AccuVote TSX DRE voting machines,
contest, the machines showed a review screen that accurately
which
wepurchasedoneBay and modified to function as
summarized the voter’s selections and highlighted any unBMDs.
The
TSX
was
first
produced by Diebold in 2003 anddervotes. The voter could return to any contest to change theisstillwidelydeployed
today.
At least 15 states plan to use it
selections. A “Print Ballot” button ended the voting session and
in
at
least
some
jurisdictions
inNovember 2020 [57].
caused a printer under the machine to output the paper ballot.
The
TSX
runs
WindowsCE
andis designed to function as
Participants carried their ballot across the polling placeapaperless
DRE
or
a
VVPAT
system.
We developed softwareto the ballot scanner station, where they inserted them into
modifications
that
allow
it
to
print
ballots
in multiple styles usan optical scanner that deposited them into a ballot box.
ing
anexternal
printer.
Thiseffectively
converts
the TSX into a
Poll Worker B was stationed by the scanner and offered
BMD
—
and
one
we
could
easilycause
tobedishonest— whileinstructions if necessary. Next, the poll worker collected thepreserving
theoriginal
touch-screeninterface
usedbyvoters.
voter access card and asked each participant to complete an
In order to modify the machine, we built on techniques used
exit survey using a laptop next to the scanning station. The
survey was anonymous, but responses were keyed so that we by Feldman et al. [23]. We began by patching the firmware
could associate them with the voter’s on-screen selections, so that, when the machine boots, it attempts to execute a
program provided on an external memory card. We used thistheir printed ballot, and poll worker notes.
Poll Worker C, positioned separately from the other stations, functionality to launch a remote access tool we created, which
acted as an observer. They verified that participants moved allowed us to connect to the TSX over a network and performthrough the polling place stations sequentially, noted whether file system operations, run applications, and invoke a debugger.
The TSXes in our polling place were connected to an
they spent time reviewing their printed ballots, and recorded
whether they appeared to notice any abnormalities. The observer Ethernet switch using PCMCIA network adapters. A Python
was also tasked with noting participant behavior, specifically program, running on a computer on the same network, used the
how the participants completed each step in the voting process remote access tool’s API to poll each machine for newly voted
and any comments they made. The observer was available ballots. Whenever a ballot was cast, the program parsed theto answer participant questions and was frequently the poll selections, generated a PDF file based on them, and sent it to a
printer located underneath the appropriate voting machine. The
worker participants approached upon noticing a discrepancy.
Like in a real polling place, multiple participants could program could be configured to apply different ballot styles
progress through the voting process simultaneously. Occasion- and cheating strategies, depending on the experiment.

4For every ballot, the program randomly selected one race tomanipulate. In most experiments, selections could be changedin three ways: deselection in a voted-for race, selection in
an unvoted-for race, or changing a selection to a different
candidate. We ensured that some alteration would take placeon every ballot. For example, in a vote-for-one race wherethe voter had made a selection, the algorithm would choose
uniformly from the set of unselected choices plus no selection.
One experiment used a different strategy, in which choices
could only be deselected.
Both the voter’s original selections and the manipulated
ballot were logged for later analysis. Each voting session was
associated with a unique tracking number, which was printed
on the ballot along with a timestamp and encoded as a barcode.
As the final step in the voting process, participants fed theirprinted ballots into an AccuVote OS optical scanner, a device
used to tabulate votes in parts of 20 states [57]. The scanner
was intended to add realism to the experiment, but AccuVote
OSes are not capable of actually tabulating the ballot styles
we used. Therefore, we modified the scanner so that it simply
fed each ballot into the ballot box without counting it.
We mounted a barcode reader in a 3-D printed case above
the scanner’s input tray and positioned it so that it would
detect the ballot’s tracking barcode. (This setup can be seen in
Figure 3.) When the barcode was read, a Raspberry Pi would
activate the AccuVote OS’s feed motor to pull the ballot into theballot box. The Raspberry Pi also displayed the ballot tracking
number so that poll workers could associate the ballot withthe participant’s exit survey response and the observer’s notes.

(a) Regular Ballot

C. The Ballot
In order to ensure a realistic voting experience and increase
participants’ psychological investment in the outcome of
the mock election, we used races and candidates from thecity’s actual ballot for the recent 2018 midterm election. For
simplicity, we reduced the ballot to the first 13 races so thatballots would not require duplex printing or multiple pages.
We tested two ballot styles, which are illustrated in Figure 2.
One is a regular ballot that shows the entire set of candidates
in every race. The other is a summary ballot, which shows
only the voter’s selections or “NO SELECTION” if a choice isleft blank. Most BMDs print ballots that resemble these styles.
The specific visual designs we used mimic ballots producedby two models of BMDs manufactured by Hart InterCivic,
which also makes the voting equipment used in Ann Arbor.
The regular style is also the same design as the hand-marked
paper ballots most Ann Arbor voters use, ensuring that many
participants found it familiar. These designs are used injurisdictions that collectively have over 10 million registeredvoters [57].
The model of laser printer we used, Brother HL-2340, iscertified for use with Clear Ballot’s ClearAccess BMD system [43], so we chose paper stock that meets the specifications
for ClearAccess [16]. Summary ballots were printed on regularweight 8.5×11 inch letter paper, while regular ballots wereprinted on Vellum Bristol stock 67 pound 8.5×14 inch paper.

(b) Summary Ballot

Fig. 2: Ballot Styles. We tested two ballot styles: (a) a regularstyle, resembling a hand-marked ballot; and (b) a summarystyle, listing only the selected candidates. Both had 13 racesfrom the city’s recent midterm election. In one race, determined
randomly, the printed selection differed from the voter’s choice.

5

D. Participants and Recruitment
To gather subjects for our study, we approached staff at the
Ann Arbor District Library (AADL), who offered space for
us to set up our mock precinct. We conducted a total of threedays of data collection in July and September 2019 at two
library locations: the Downtown and Westgate branches. The
Downtown branch, where our study was held for two of thethree days, is an official polling location during real elections.
The AADL advertised our study through its social media
feeds and offered incentives to patrons for their participation,such as points for a scavenger hunt competition [5] and souvenir
flashlights [6]. We also set up a fourth voting machine outsideof the mock precinct where kids could vote in an election for
mayor of the library’s fish tank.1 Results from that machine
were not used as part of this study, but it served as a recruitment
tool for parents visiting the library with their children. In
addition, we verbally recruited patrons who happened to be at
the libraries during our study, using the script in Appendix B.
Participants were required to be at least 18 years of age
and to sign an IRB-approved consent form. All data collected,
including survey responses and behavioral observations, wascompletely anonymous. We informed participants that they
were not required to vote their political preferences.
E. Experiments
To explore what factors affect voter verification performance,
we devised nine experiments to run between subjects. In
all experiments, for every participant, one selection that theparticipant made on the BMD was not accurately reflectedon the printed ballot. Every participant within an experimentreceived the same instructions from the poll workers, following
the script and variants in Appendix A.
The first three experiments were designed to measureverification in the absence of protective interventions. Theyvaried the ballot style and manipulation strategy:

Fig. 3: Warning Signage. One of the interventions we tested
was placing a sign above the scanner that instructed voters toverify their ballots. Signage was not an effective intervention.

E1: Regular ballots We used the regular ballot style and thedefault manipulation strategy, in which a selection could beswitched, deselected, or selected if left blank by the voter.

E6: Script variant 2 When the voter approached the scanner,
the poll worker said: “Please keep in mind that the paper ballot
is the official record of your vote.”

Figure 3. We designed the sign following guidelines from the
U.S. Election Assistance Commission [55].
E5: Script variant 1 During voter check in, the poll worker
added this instruction: “Please remember to check your ballot
carefully before depositing it into the scanner.”

E2: Summary ballots We used the summary ballot styleand the default manipulation strategy. As discussed in Section IV, we found no significant difference in error detection
between regular ballots and summary ballots, so all subsequent
experiments used summary ballots.

E7: Script variant 3 When the voter approached the scanner,
the poll worker said: “Have you carefully reviewed eachselection on your printed ballot?”

Four further experiments tested interventions to determine
if they improved error detection. We tried posting a sign andhaving poll workers give different instructions at various times:

The final two experiments assessed whether reminding
participants of their selections during verification improved theirperformance. We gave voters a slate of candidates for whom
to vote that they could carry with them throughout the voting
experience. While we refer to this as a slate, a sample ballot thatthe voter filled in before voting could serve the same purpose.
Every voter received the same slate (Appendix C), which was
randomly generated and contained an even mix of parties.

E4: Signage A sign was placed above the scanner that
instructed voters to check their printed ballots, as shown in

E8: Slate with script variant 2 Voters were given the slate.
Poll workers encouraged verification with script variant 2.

1 Mighty Trisha unexpectedly beat Creepy Bob, leading some Bob supporters
to complain that the results were fishy [4].

E9: Slate with script variant 3 Voters were given the slate.
Poll workers encouraged verification with script variant 3.

E3: Deselection only To assess the sensitivity of voters to theway their ballots were changed, we limited the manipulation
to deselecting one of the voter’s choices at random.

6

ExperimentN

Were observed
examining ballot

Reported error
on exit survey

Reported error
to poll worker

Without interventions:
E1: Regular ballots
E2: Summary ballots
E3: Deselection only

31
31
29

41.9%
32.3%
44.8%

6.5%
6.5%
10.3%

6.5%
6.5%
6.9%

91

39.7%

7.8%

6.6%

30
30
25
31
13
21

13.3%
46.7%
92.0%
38.7%
100.0%
95.2%

3.3%
13.3%
16.0%
19.4%
38.5%
71.4%

6.7%
6.7%
16.0%
12.9%
38.5%
85.7%

150

64.3%

24.0%

27.8%

Subtotal/MeanWith interventions:
E4: Signage
E5: Script variant 1
E6: Script variant 2
E7: Script variant 3
E8: Slate with script variant 2
E9: Slate with script variant 3
Subtotal/Mean

TABLE I: Verification Performance for Each Experiment. Without interventions, participants’ verification performance wasremarkably poor: only 7.8% noted on an exit survey that their ballots had been altered, and only 6.6% informed a poll worker(averaged across experiments). The various interventions we tested had widely different effects, ranging from no significant
improvement (E4, E5) to a large increase in verification success (E8, E9).

IV. R ESULTS

demographics resembled the population of Ann Arbor voters(the city is among the most highly educated in the U.S.) [33].

A. Participant Demographics

We recruited 241 participants. The vast majority (220, B. Verification Performance91%) indicated that they were native English speakers; 19
To quantify verification performance, we collected threereported speaking twelve other native languages, including data points for each participant, which are summarized in
Hungarian, Korean, and Arabic; and two subjects gave no Table I. First, an observer noted whether the subject appeared
response. Participants who disclosed their age ranged from to examine the printed ballot for at least two seconds. Second,
18 to 84 years old, with a mean of 43.7 and a median of 42; the exit survey asked, “Did you notice anything odd about your
15 subjects did not answer the question. The percentages that ballot?”, and we recorded whether the subject’s response corfollow are out of the total number of responses to each question: roborated the discrepancy (i.e., correctly articulated which race
Respondents identified as male (84, 35%), female (152, 64%), was changed). Third, we recorded whether subjects reported
or other (3, 1%); two did not respond. Subjects reported their the ballot modification to a poll worker. Most experiments
ethnicity as Caucasian (187, 80%), Asian (17, 7%), African saw more participants identify discrepancies in the survey
American (6, 3%), Mexican American/Chicano (5, 2%), and than were reported to poll workers, but these differences
Other Hispanic/Latino (9, 4%); others reported not having any were not statistically significant. Where applicable, we referof these ethnic backgrounds (2, 1%) or were multiracial (9, to participants who by some means reported detecting the4%). Participants reported their level of educational attainment discrepancies as “noticers” and those who did not as “nonas some high school (1, 0.4%), a high school diploma (4, 2%), noticers”.
some college (20, 8%), a two-year degree (10, 4%), a four-year
1) Performance without interventions (E1–E3): With nodegree (80, 33%), a master’s or professional degree (92, 38%), interventions, we found verification performance to be consisor a doctorate (34, 14%).
tently poor. The three experiments involved 91 participants,
Most subjects indicated that they were registered to vote in and, averaged across the experiments, only 40% of participants
the U.S. (220, 92%), had voted in a previous election (216, examined their ballots, only 7.8% noted the error on the91%), and had voted in the November 2018 midterm election exit survey, and only 6.6% reported it to a poll worker. We(209, 87%). However, we note that, historically, 38–45% of did not find significant differences in performance between
non-voters have been found to falsely report having voted [10]. regular and summary ballots or between the tested attack
Compared to the population of Ann Arbor at the time of the strategies.
2010 census, our participant pool overrepresented Caucasians2) Effectiveness of interventions (E4–E9): The tested in(∆ = 7.6%) and underrepresented African Americans (∆ = terventions resulted in a wide range of effect sizes. Neither−4.4%) and Asians (∆ = −8.7%) [54]. The study population signage (E4) nor poll worker instructions issued before thealso overrepresented females (∆ = 13%) and underrepresented participant began voting (E5) yielded a statistically significant
males (∆ = −16%) [59]. In other reported aspects, participants’ improvement to any aspect of verification performance. In

7contrast, poll worker instructions issued after the ballot was3) Demographics: Comparisons of detection rates acrossprinted (E6 and E7) did have a positive effect, boosting demographic groups revealed that a strong indicator forreporting rates to 20% on the exit survey and 14% to poll verification performance was voting experience. Subjects who
workers (averaged across the experiments).
reported being registered to vote (N = 220) detected errors withThe largest performance gains occurred when participants their ballots 19% of the time, while their those who did not
were directed to vote using a slate of candidates (E8 and E9). (N = 21) detected errors 4.8% of the time. Those who reported
However, only E9 produced a statistically significant difference voting previously (N = 216) caught ballot discrepancies inin reporting rates (Fisher’s exact p < 0.001).2 Averaged across 19% of cases, again performing better than those who reported
both experiments, reporting rates increased to 55% on the exit not voting before (N = 25), who detected an error in 4.0%
survey and 62% to poll workers. E8, in which participants were of cases. If someone reported voting in the 2018 midterm
directed how to vote using a slate of candidates, saw detection election (N = 209), they detected problems with their ballot
and reporting rates of 39%, which is similar to results for DRE 20% of the time, whereas if they did not (N = 32), they detected
review screen performance found by Campbell et al. [15] and problems 3.1% of the time. This may indicate that familiarity
Acemyan et al. [1], in studies that similarly directed participants with the midterm ballot we used caused participants to feel
how to vote. With script variant 3, the use of a slate produced more invested in the accuracy of their votes; however, we dida significant difference (comparing E7 and E9, Fisher’s exact not establish this to statistical significance.
p < 0.02) for both review and report, but it did not produce a
Other demographic factors, such as age, education, ethnicity,
significant difference using script variant 2 (comparing E6 and and gender, had no correlation with detecting manipulation.
E8). This indicates that voters may be sensitive to the specific4) Ballot position: Noticing was correlated with ballot
instructions they receive about reviewing their ballots.
position (Pearson’s of −0.64), indicating that discrepancies
in more prominent races are more likely to be noticed. (RaceC. Correlates
0 was the first race on the ballot, so the number of noticers
1) Reviewing the ballot: Reviewing the ballot at all was decreases as the race position increases, hence the negative
significantly correlated with error reporting (two-sample permu- correlation coefficient.) On our ballot, the first five races
tation test p < 0.001 with 10k repetitions). Some interventions (Governor, Secretary of State, Attorney General, U.S. Senator,
do seem to promote reviewing: E6, E8, and E9 saw significant and Representative in Congress) were prominent partisan
increases (Fisher’s exact p < 0.004), although E7 did not.
contests with a high likelihood of name recognition. In the2) Time to ballot submission: Careful verification takes time, experiments with no intervention (E1–E3), 37 participants had
so one might expect that participants who noticed discrepancies one of these races manipulated, and five reported the error
took more time to cast their ballots. As an upper bound on on the exit survey, a rate of 14%. Additional experiments are
how long subjects spent verifying, we calculated the time from necessary to establish the strength of this effect when combined
ballot printing to ballot submission. (Due to clock drift on with interventions.
one of our machines, data from the third day of experiments
5) Undervotes: A metric that may inform voters’ ability
was unusable, and consequently E4 and E7 are excluded from and willingness to verify their ballot is how much care they
our timing analysis.) As expected, we find that noticers took take in filling out the ballot. There are two metrics we use to
an average of 121 s between printing and ballot submission examine this: whether a participant voted in every contest on(median 114 s), compared to only 43 s for non-noticers (median the ballot, and whether the participant voted in every available32 s). This difference is statistically significant (two-sample position on the ballot (e.g., in a vote-for-two contests, thepermutation test p < 0.004, 10k iterations).
participant selected two choices). Table II shows the rates of
We compared the submission times for two sets of experi- voting in every race and every position on the ballot, withments: ones with extra instructions to the voter (E5, E6, E8, E8 and E9 removed as they directed participants to vote inand E9; N = 84) and ones without (E1, E2, and E3; N = 91). every position. Voters who noticed discrepancies voted in every
The experiments that asked participants to review their ballots race or every position at a higher rate than those who did not,saw significantly more time spent between ballot printing but not significantly so (likely due to our small sample size).
and submission (two-sample permutation test p < 0.004, 10k Since these undervotes are visible to malware running on aiterations), an average of 83 s (median 72 s) compared to 50 s BMD, this correlation could be exploited by an attacker towithout (median 33 s).
focus cheating on voters who are less likely to carefully verify,
Notably, participants who were given a slate of candidates provided future work more firmly establishes this link.
to vote for had much higher submission times (two-sample
permutation test p < 0.004, 10k iterations). Noticers in theOverall
Noticers Non-noticers
slate experiments took an average of 119 s (median 111 s) and
Every race
64.3%
73.9%
63.0%
non-noticers averaged 55 s (median 52 s). This might be partly
Everyposition
43.0%
47.8%
42.4%
attributed to voters having to select unfamiliar candidates andwanting to check their work.
TABLE II: Participant Attentiveness. Voters who noticed thediscrepancy tended to vote in every race and ballot position
more often than those who did not.

2 All p-values were computed with a Bonferroni correction at a family-wiseerror rate of 0.05.

86) Partisanship: To assess the role partisanship plays in using Fisher’s combining function. We found that the effect ofdetection rates, we scored each ballot with a partisanship score, overlapping with a noticer did not significantly impact whether
where a vote for a Democratic candidate was scored −1 and a a participant noticed. This suggests that our interventions werevote for a Republican candidate was scored 1, and we take the more important than overlapping.
absolute value of the sum. There were 11 opportunities to vote9) Signage: One feature that did not correlate with improvedin a partisan way, so a participant who voted straight-party verification performance was the signage we tested (E4). Ourfor either major party would achieve a score of 11. Excluding observer noted that 11 of 30 participants in the signage
E8 and E9, where voters were directed how to vote, the mean experiment did not notice the sign at all. Only two participants
partisanship score for our participants was 8.3, and the median in this experiment detected the modification of their ballot
was 11. Although our BMD did not offer an automatic “straight- and reported it, and only one accurately noted the discrepancy
party” voting option, 105 participants achieved the maximum in their survey, suggesting that passive signage alone may
partisanship score.
be insufficient to capture voters’ attention and shape their
Intuitively, a voter expecting every selected candidate to be subsequent behavior.
from the same party might be more likely to notice a selection
from a different party. Looking at only these straight-party D. Participant Comments
Participants had two free-response sections in the exit survey.
voters, 15 out of 105 detected the errors. Of those, nine hada partisan race swapped to a different candidate of a different The first asked about anything “odd” they had noticed about
party, and six of those participants wrote in the survey that they the ballot. The second invited any additional comments. Of
had detected the change based on party. For example, one par- the 241 participants, 114 responded to at least one of theseticipant wrote, “voted GOP for governor / lieutenant governor prompts. We note several features of their responses.
1) Discrepancy reports: In total, 44 participants (18%) noted
but Libertarian was actually selected on the paper ballot.”
This suggests that choosing a uniform set of candidates in the free response section of the survey that they had identified
may help voters detect when something has gone wrong on some discrepancy on their paper ballot. Of these, 31 correctly
their ballot, although more work is needed to establish that identified the change, 12 gave no detail (e.g., “At least one ofthis is indeed the case, especially in more politically diverse my choices did not match who I picked”), and one incorrectly
populations. If this positive effect holds, it could be further identified the change (but did report that there was a mistake).
promoted with ballot designs that prominently display the party, We omitted this last participant from our “noticers” category
which could help voters see the information that is important where applicable.
Of the 44 participants who reported a change on theirto them while they review the ballot. On the other hand, BMD
malware could be designed to counter this effect by focusing ballot in the survey, five added that they thought it could
have resulted from a mistake they made. For example, one
cheating on voters who do not cast a straight-party ballot.
7) Slate voting: 34 participants were assigned an inter- participant reported: “I don’t remember voting for the member
vention which asked them to vote for a preselected slate of Congress and there was a vote. I very well may have butof candidates (with a partisanship score of 0). Of these, just don’t remember.”2) Attitudes about verification: Twelve participants menonly 26 participants voted exactly as directed. Of the eightparticipants who did not, four voted a straight Democratic ticket tioned either that they would only be comfortable voting on(partisanship score of 11), one voted a heavily Democratic ticket a paper ballot or that they were comforted by the fact that a(score of 9), two voted slightly Democratic tickets (scores of paper trail was created. Only three of these 12 participants
3 and 5), and one voted a non-partisan ticket (score of 0), noticed that their ballot had been modified, despite the factwhich only deviated from the slate in five positions. Of the that they recognized that the paper ballot was an importanteight participants who deviated from the slate, no participant tool for ensuring election integrity.
Several participants seemed to realize after casting theirdeviated by fewer than five positions, indicating that either
the deviation was deliberate or our instructions to vote the vote that the evaluation of their paper ballot was important; 13slate were unclear. Only one deviating participant managed participants mentioned in the survey that they did not review or
to notice the discrepancy on their ballot, leaving participants that they should have reviewed the ballot, although we did not
who deviated from the slate a 13% notice rate compared to ask them about it. This concern may have been triggered by oursurvey question about what they had noticed about the paper
the 73% notice rate for those who did not deviate.
8) Network effects: One potential feature of a live polling ballot, but it also might be an indication that our interventions
place environment is a network effect: will a voter who is did cause voters to think about the risk — albeit too late.
voting at the same time as a noticer be more likely to notice
The free responses also indicate that some participantsa problem on theirs? However, the number of people who assumed that the vote was completed and submitted on thenotice in a given experiment is a confounding factor: voters BMD, rather than the paper ballot being the official record of
are more likely to overlap with a noticer if there are more their vote. One participant wrote, “I was surprised to still havenoticers. To interrogate this, we ran partial hypothesis tests for a paper ballot, after using the touch system. I was expecting
each intervention using Fisher’s exact tests with permutations the results to be registered electronically.” This assumptionof overlapping with a noticer and noticing, and then combined may discourage voters from verifying the selections on their9paper ballot. Similarly, another participant, prompted by script
variant 3 (“Have you carefully reviewed each selection on yourprinted ballot?”), responded to a poll worker, “I checked it on
the screen, it better be right.”
Three participants expressed concern that they would notknow what to do if they noticed a problem with their paper
ballot during a real election. One person wrote, “Having theprintout be incorrect was confusing and it’s not clear how thatwould be handled in an election environment.”3) Feedback on the BMDs: We told participants that theexperiment was a study about a new kind of voting system,
and many left feedback about the interface and appearanceof the machines. In Michigan, where we conducted the study,
BMDs are available in every precinct, but voters must request
to use them. The vast majority of voters use hand-marked paper
ballots, so study participants were likely unfamiliar with BMD
voting. In their comments, 21 participants expressed liking thesystem, while only three disliked it. Although merely anecdotal,
this reflects previous findings that voters like touch-screen
voting equipment [22].

Number of problem reports Let d be the fraction of misprinted ballots that voters detect, report, and correct. Suppose
a contest had n ballots cast, and the reported fractional marginof victory was m. To have changed the outcome, the attacker
would have had to successfully modify at least n m2 cast ballots.
However, since some modifications would have been corrected,
the attacker would have had to induce errors in a greater numbermof printouts: n 2(1−d). Under our optimistic assumptions, if theattack changed the outcome, we would expect the fraction of
voters who reported problems, a, to exceed:a>md.
2(1 − d)voters who report a problem (%)

The model shows that the security impact of verification
is non-linear, because every voter who corrects an error bothincreases the evidence that there is a problem and forces theattacker to cheat more in order to overcome the margin of
victory. Figure 4 illustrates this effect.
With the 6.6% error detection rate from our non-intervention
experiments and a close election with a 0.5% margin (themargin that causes an automatic recount in many states) a
V. S ECURITY M ODEL
successful attack would cause as few as 0.018% of voters —We are primarily motivated by the threat of undetected less than 1 in 5000 — to report a problem. Small changes inchanges to election outcomes due to BMD misprinting attacks. verification performance around our base rate cause relatively
Prior work has shown that such attacks cannot be reliably ruled little change in the amount of evidence. More than doubling theout by pre-election or parallel testing [51], and we seek to error detection rate to 14% (the rate we found for prominentanswer whether voter verification can be an effective defense. races) only increases the fraction of voters who report a
If a voter reports that their printed ballot does not reflect their problem to 0.039%. However, larger improvements have an
on-screen selections, what should election officials do? Unfor- outsized effect: with the 86% error detection rate from ourtunately, there is not yet a practical way to prove that the BMD most successful experiment, at least 1.5% of voters (1 in 67)
misbehaved during voting. From officials’ perspective, it is also would report problems.
possible that the voter is mistaken, or even lying, and in a large Required detection rate Suppose election officials activate avoter population, there will always be some rate of spurious countermeasure if the fraction of voters who report problems
problem reports, even when BMDs are working correctly.
For these reasons, problem reports from voters can serve only
as evidence that something might be wrong with the BMDs.
1.2
If the evidence exceeds some threshold, officials could invoke
1% threshold
1.0
contingency plans. For instance, they could remove BMDs
from service to minimize further damage, perform forensic
0.8
investigations in an attempt to uncover the cause, or even rerun
the election if outcome-changing fraud cannot be ruled out.
0.6
Any of these responses would be costly (and none is
foolproof), so the threshold for triggering them should notverbal instructions
0.4
be too low. Moreover, attackers could exploit a low thresholdno interventions
by recruiting voters to fraudulently report problems, in orderslates0.2
0.1% thresholdto disrupt or discredit the election. On the other hand, if thethreshold is too high, outcome-changing fraud could be ignored.
0.0
0
20
40
60
80
100
To better understand how verification performance affects
error detection rate (%)
security in this setting, we construct a simple model. We
assume, optimistically, that the attacker has no way to guess Fig. 4: BMD security is highly sensitive to human performance.
whether a particular voter is more likely than average to detect Given a 0.5% margin of victory, we plot the percentage ofthe alteration, and so chooses voters to attack at random. We voters who report a problem during the minimal outcomefurther assume that whenever voters detect problems, they are changing attack as a function of the rate at which errors areable to remedy them and cast a correct vote by hand-marking detected and corrected. This model implies that using BMDs
a ballot. Except where noted, the model assumes that all voters safely for all voters requires dramatically improved verification
performance or very sensitive attack detection thresholds.
cast their votes using BMDs.

10

exceeds a threshold a∗ . For a given margin, the countermeasure
will be triggered by minimal outcome-changing fraud when:
2a∗

VI. D ISCUSSION
A. Limitations

It is challenging to capture real-world voter behavior ina mock election. However, our study followed establishedAn expensive countermeasure, like rerunning an election, will best practices [39], and we strived to create as realistic arequire a high trigger threshold — say, 1% of voters reporting polling environment as we could. It is impossible to know
a problem — to avoid false positives. With a 0.5% margin, exactly how well we succeeded, but the effect seems to havereaching a 1% trigger threshold would require an error detection been convincing: several people approached us to ask whether
rate exceeding 80%. A less expensive countermeasure, such there was a real election taking place that they had not heard
as an investigation, might be triggered by a lower threshold — about. Our participants also seemed engaged in the study; manysay, 0.1%. Reaching this lower threshold in an election with a expressed strongly held political preferences in our survey (so0.5% margin would require an error detection rate greater than much so that some refused to vote according to our slate), and29%. This suggests that using BMDs securely for all voters a large majority reported voting in the 2018 midterm. On thewill require large improvements to verification performance or other hand, the election used a ballot that was more than ninemonths old, which may have reduced participant motivation,
extremely low thresholds for triggering countermeasures.
and we had a few participants who reported that they did not
Minimizing BMD voting helps dramatically Securing against vote in our state or were otherwise unfamiliar with our ballot.
misprinting attacks is far easier if only a small fraction of voters It is also possible that our results were skewed due to selection
use BMDs than if all in-person voters do. This is because an bias and observer effect.
attacker would be forced to cheat on a much larger fraction
Another limitation of our work is that we drew participantsof BMD ballots in order to achieve the same change to the from a population that is locally but not nationally representaelection results. Moreover, if the population of BMD voters is tive. Our participants tended to be younger, significantly bettersmaller than half the margin of victory, it is impossible for a educated, more liberal, more likely to be female, and more
BMD misprinting attack to change the outcome.
likely to be Caucasian than the average voter in the United
Let b be the fraction of voters who use BMDs. We can States [54]. Future work is needed to validate our study inreplace m in the expression above with mb and let a∗ be the more diverse and representative populations.
fraction of BMD voters that must report a problem to trigger
Although our results suggest that certain interventions can
the countermeasure. In Maryland, which uses hand-marked boost verification performance, the data is too sparse topaper ballots but makes BMDs available to voters who request provide a high-fidelity understanding of the magnitude of thethem, 1.8% of voters use BMDs [34]. With a 0.5% margin, as improvements. In addition, due to time constraints, we werein the previous example, Maryland would reach a complaint unable to test the interplay of all combinations of interventions,threshold of 1% of BMD voters with an error detection rate of and some interventions appear to be sensitive to small changes
only 6.7%. If 5% of voters use BMDs, the error detection rate (e.g., the difference in phrasing between script variants 2
would need to be 17%. Our results suggest that these more and 3). Further study is needed to better characterize whatmodest rates of verification likely are achievable, in contrast to makes interventions work and how they interact before we canthe far greater accuracy required when all voters use BMDs. confidently conclude that any particular set of procedures will
This model overestimates security An attacker might use be effective in practice.
any number of features (including several of the correlations
we observed) to focus cheating on voters who are less likely B. Discussion of Findings
to successfully catch errors. For instance, an attacker could
Our study provides the first concrete measurements of voter
preferentially modify ballots that have undervotes or a mix error detection performance using BMDs in a realistic votingof selections from different parties. Attackers could also environment. At a high level, we found that success rates
selectively target voters with visual impairments, such as those without intervention are very low, around 6.6%. Some interwho use large text or an audio ballot. Other features, such as ventions that we tested did not significantly impact detection
how long voters spend inspecting the candidate review screen, rates among participants, although others improved detection
might also prove to be predictive of verification success. For drastically and may serve as a roadmap for interventions tothese reasons, our simplified model is likely to overestimate explore in further research. We discuss those interventions here.
the effectiveness of verification against sophisticated attackers.
1) Verbal instructions can improve verification: Notably, all
We also note that some attackers may merely seek to cast interventions that involved poll workers verbally encouraging
doubt on election results by causing highly visible errors or verification between the BMD and the scanner— those in E6–
failures — which are also possible with hand-marked paper E9 — resulted in higher ballot reviewing and error reporting
ballots. However, in general, BMDs are vulnerable to all classes rates. This, coupled with the fact that reviewing the printout
of computer-based attacks that affect hand-marked paper ballots was highly correlated with error detection across all of ourand to others, such as the misprinting attack discussed here, results, suggests that interventions focused on causing theto which hand-marked paper ballots are not susceptible.
voter to review the ballot carefully may be helpful. On thed>m + 2a∗.

11other hand, instructions at the beginning of the voting process after their ballot has printed but before it is scanned and ask(E5) and passive signage (E4) had no significant effect on them to review it. Signage with a similar message to our scripts
error reporting. This pattern of effects is supported by findings placed at the optical scanner (E4) or instructions before thefrom the usable security literature, which suggest that post- participants voted (E5) did not result in significant differencescompletion errors can be mitigated with timely interruptions in error detection; nevertheless, further study with additionalthat encourage individuals to take defensive steps [14].
variations is prudent before ruling out such strategies.
3) Encourage personalized slate voting: Although our
It is worth noting that we also found that these interventions
caused participants to take longer to submit their ballots, on study tested randomized slates, rather than personalized slates,
average about twice as long. This could cause longer lines at the effect size was so large that we tentatively recommend
polling places if these interventions are implemented without encouraging the use of personalized slates by voters. In ourcomplementary procedural considerations, such as having experiments (E8 and E9), participants who were directed tovote using a randomized slate (and did not deviate) reported
adequate space for voters to stop and review their ballots.
2) Effectiveness of slates: Directing participants to vote errors at a rate of 73%. If voters prepare their own slates
for a provided slate of candidates, combined with verbally at home (or use a printed slate prepared, for instance, by aprompting them to review their printouts, resulted in strongly political party or other organization), they can use them toincreased rate of error detection: 74% of participants who were check each selection on the BMD printout. We note that, since
given a slate and did not deviate from it noticed the errors. we did not directly test the use of personalized slates, further
This finding may suggest that encouraging voters to write down research is necessary to ascertain whether large performancegains are actually achieved. Furthermore, even if personalized
their preferences in advance can boost verification.
However, the slates we used functioned quite differently from slates are effective, the gain will be limited to the fraction ofslates likely to be used in practice. The choices we provided voters who can be induced to use them.
Slates have potential downsides and should be used with
were randomly generated and had no basis in the subject’s
preferences — in a real election, slates would reflect who the care. They have the potential to compromise ballot secrecy, so
voter intended to vote for, most likely created by the voter or we recommend providing a closed trash can, paper shredder,their political party [29]. It is possible that the success rate we or other means for voters to privately dispose of them before
observed was primarily due to participants carefully attempting leaving the precinct. Coercion is also a threat, but voters could
to follow our instructions and vote for unfamiliar candidates. be advised to prepare multiple different slates as a defense.
4) Help voters correct errors, and carefully track problems:
Further study is needed with more realistic slate conditions(i.e., asking subjects to write down their preferences) in order Verification-promoting interventions will be of little use if
action cannot be taken to remedy misbehaving BMDs —to assess whether slates really do help voters catch errors.
something that even our participants expressed concern about.
C. RecommendationsFirst, it is crucial that polling places have a procedure forSince BMDs are widely used today, we recommend several voters who want to correct their printed ballots. Several subjects
strategies for improving voter verification performance. While commented that they would not know what to do if something
we are unable to conclude that these strategies will enhance was wrong with their ballot in a real election, indicating thaterror detection to the point that BMDs can be used safely in this problem is present in current election procedures.
close or small elections, our findings indicate that they can help.
Second, detailed records should be kept about which BMD
1) Design polling places for verification: Polling place lay- the voter used and what the specific issue was, including
out and procedures should be designed with verification in mind. the contest and candidates involved (to the extent that the
As we have discussed, voters need time and space to verify their voter is willing to waive ballot secrecy). Problems should beballots. If tables or areas to stand out of the way are provided, treated as potentially serious even when the voter believes
voters will be able to carefully verify without causing lines they are at fault — we note that several participants in our
to form or slowing polling place throughput. The presence of study believed they had made a mistake even though the BMD
such a “verification station” might also encourage verification. actually was programmed to be malicious. Problem reports
Another practical concern is privacy. Several of our partici- should be centrally reported and tracked during the election,
pants expressed discomfort with the fact that we did not provide so that issues affecting multiple precincts can be identified asa privacy sleeve for their ballots (a requirement in Michigan), rapidly as possible.
and that the scanner accepted the ballots face-up only, with one5) Prepare contingency plans: What to do in the event
participant stating, “I feel like inserting the ballot face up in that BMDs are known or suspected to be misbehaving is athe scanning machine will make people uncomfortable.” Voters more difficult question. If an elevated number of voters have a
may not feel comfortable reviewing their ballots in front of poll problem with a single machine, it should be taken out of service,
workers but may be unsure where to go to review them privately. provided there are other BMDs available for use (especially
2) Incorporate post-voting verbal instructions: As all of for voters with disabilities, who may have no alternative).
our script-based interventions that took place after the ballot
If widespread problem reports occur— particularly problems
was printed (E6–E9) showed an increase in verification focused on a tightly contested race or significantly exceeding
performance, we recommend that poll workers interrupt voters the rate reported in past elections — officials could consider

12

taking most BMDs out of service and encouraging all remaining
voters who can to use hand-marked ballots. This raises logisticalchallenges: polling place would need to have enough ballots
available for hand-marking, or the ability to print ballots on
demand, and votes already cast on the BMDs would be suspect.
After the election, forensic analysis of the BMDs could
be performed to attempt to determine the cause of reported
errors. Unfortunately, such analysis cannot in general rule outthat a sophisticated attack occurred and left no digital traces.
Even if programming errors or attacks are uncovered, they may
be impossible to correct if officials are unable to determine
whether the effects were large enough to change the election
outcome. The only recourse might be to re-run the election.
Our findings show that, in the event of an actual error or
attack, the rate of reported problems is likely to be only the tipof the iceberg. In our non-intervention experiments, undetected
errors outnumbered reported problems by almost twenty to one.
Our results further suggest that an attacker who cleverly focusedcheating on voters who were less likely to verify could achieve
an even higher ratio of undetected errors. An effective response
requires either being very sensitive to reported problems —which increases the chances that an attacker could trigger false
alarms — or achieving very high error correction rates.
6) Educate voters about BMD operations and risks: Likein other human-in-the-loop security contexts, greater education
could boost voters’ awareness of the importance of careful
verification and boost error detection and reporting rates.
To this end, we recommend educating voters that the paper,
rather than what the BMD screen shows, is the official recordof their votes. Several of our participants said they realized
after scanning that they should have, but did not, review theirprintouts. Others stated that they had checked the review screenon the machine and that they trusted the paper to be correct.
It is likely that many participants incorrectly assumed that the
BMDs, rather than the paper and scanner, tabulated their votes.
We also recommend educating voters about the possibilityof BMD malfunction. Many of our participants seem not tohave even considered that the machine might have changed
their votes, as indicated by the voters who blamed themselves
for the misprinted ballots. Raising threat awareness could help
motivate voters to carefully inspect the paper, as well as givethem greater confidence to report any discrepancies they detect.
7) Consider the needs of voters with disabilities: Further
research is needed to specifically examine verification performance among voters with disabilities, but we offer some
initial recommendations here. Detecting errors in printed ballots
may be especially challenging for voters with impaired vision.
Designing BMD ballots for maximum legibility might help, andso might encouraging voters who use text-to-speech devices tobring them to the polls for use during verification. Jurisdictions
could also provide air-gapped accessible devices to read theballot back to voters, in case voters do not have their own textto-speech devices. These steps would have the added benefitof reinforcing the message that the content of the paper ballots
is what gets counted. If BMDs are to live up to the promise of13

better and more accessible voting, enabling all voters to verify
their printed ballots is a must.
8) Require risk-limiting audits: Even perfectly verified
paper ballots are of little use for security if they are notrigorously audited to confirm the results of computer-based
tabulation. Fortunately, risk-limiting audits [32] (RLAs) are
gaining momentum in the United States. Colorado, Nevada, andRhode Island mandate statewide RLAs, and states including
Michigan, Virginia, Georgia, and Pennsylvania are considering
implementing them soon [17]. RLAs and effective verification
are both necessary in order for paper to provide a strong defense
against vote-stealing attacks, and we recommend that efforts
to achieve both be pursued vigorously.
VII. C ONCLUSION
We conducted the first empirical study of how well votersusing BMDs detect errors on their printed ballots, which is alimiting factor to the level of security that a BMD-based paper
trail can provide. Based on the performance of 241 humansubjects in a realistic polling place environment, we find that,absent specific interventions, error detection and reporting
rates are dangerously low. Unless verification performance canbe improved dramatically, BMD paper trails, particularly whenused by all in-person voters, cannot be relied on to reflect
voter intent if the machines are controlled by an attacker.
Nevertheless, we also find that procedural interventions canimprove rates of error detection and reporting, potentially
increasing the security offered by BMDs. The interventions
we tested should serve as examples of what is and is not
likely to be effective, and we hope they will point the way
for further research and experimentation. These findings add
to the broad literature of human-in-the-loop security results
and recommendations, and they provide additional examplesof what does and does not work in human-centric security.
Our results should not be read as demonstrating that BMDs
can be used securely. Further work is needed to explore thepotential for attackers to predict which voters will verify,
and additional human-subjects testing is necessary to confirm
whether sufficient rates of verification success can be achievedin practice. The cost of implementing interventions andcontingency plans may also be prohibitive. Nevertheless, BMDs
do offer advantages, including uniform accessibility and ease ofadministration. We hope our work will help election officials
make better informed choices as they weigh these benefits
against the security risks of using BMDs for all voters.
ACKNOWLEDGMENTS
The authors are grateful to Jackie Fleischer Best, Eli
Neiburger, Emily Howard, Matt Dubay, and everyone at the
Ann Arbor District Library, without whom this study would
not have been possible. We also thank Philip Stark for advice
about our statistical analyses; Ben Adida, Monica Childers,and Ben VanderSloot for feedback about the experimental
design; and the anonymous reviewers. This material is based inpart upon work supported by the National Science Foundation
under Grant No. CNS-1518888, by the Facebook Fellowship
Program, and by the Andrew Carnegie Fellows Program.

R EFERENCES[1] C. Z. Acemyan, P. Kortum, and D. Payne. Do voters really fail to detectchanges to their ballots? An investigation of ballot type on voter error
detection. Proceedings of the Human Factors and Ergonomics Society,57:1405–1409, 2013.
[2] D. Akhawe and A. P. Felt. Alice in Warningland: A large-scale field studyof browser security warning effectiveness. In 22nd USENIX Security
Symposium, pages 257–272, 2013.
[3] H. Almuhimedi, F. Schaub, N. Sadeh, I. Adjerid, A. Acquisti, J. Gluck,
L. F. Cranor, and Y. Agarwal. Your location has been shared 5,398 times!
A field study on mobile app privacy nudging. In 33rd ACM Conferenceon Human Factors in Computing Systems, CHI, pages 787–796, 2015.
[4] Ann Arbor District Library. Classic shop drop! Plus, fish election results!,
Aug. 2019. https://aadl.org/node/396262.
[5] Ann Arbor District Library. Mock the vote, July 2019. https://aadl.org/
node/395686.
[6] Ann Arbor District Library. Mock voting @ AADL, Sept. 2019. https://
aadl.org/node/397364.
[7] A. Appel, R. DeMillo, and P. Stark. Ballot-marking devices (BMDs)
cannot assure the will of the voters, 2019. https://papers.ssrn.com/sol3/
papers.cfm?abstract_id=3375755.
[8] S. Bell, J. Benaloh, M. D. Byrne, D. DeBeauvoir, B. Eakin, G. Fisher,
P. Kortum, N. McBurnett, J. Montoya, M. Parker, O. Pereira, P. B. Stark,
D. S. Wallach, and M. Winn. STAR-Vote: A secure, transparent, auditable,
and reliable voting system. USENIX Journal of Election Technology and
Systems, 1(1), 2013.
[9] M. Bernhard, J. Benaloh, J. A. Halderman, R. L. Rivest, P. Y. Ryan,
P. B. Stark, V. Teague, P. L. Vora, and D. S. Wallach. Public evidence
from secret ballots. In 2nd International Joint Conference on Electronic
Voting, E-Vote-ID, pages 84–109, 2017.
[10] R. Bernstein, A. Chadha, and R. Montjoy. Overreporting voting: Why
it happens and why it matters. Public Opinion Quarterly, 65(1):22–44,
2001.
[11] D. Bowen et al. Top-to-bottom review of voting machines certified for use in California. Technical report, California Secretary ofState, 2007. https://www.sos.ca.gov/elections/ovsta/frequently-requestedinformation/top-bottom-review/.
[12] C. Bravo-Lillo, L. F. Cranor, J. Downs, and S. Komanduri. Bridging the
gap in computer security warnings: A mental model approach. IEEE
Security & Privacy, 9(2):18–26, Mar. 2011.
[13] C. Bravo-Lillo, S. Komanduri, L. F. Cranor, R. W. Reeder, M. Sleeper,
J. Downs, and S. Schechter. Your attention please: Designing securitydecision UIs to make genuine risks harder to ignore. In 9th Symposium
on Usable Privacy and Security, SOUPS, 2013.
[14] M. D. Byrne and S. Bovair. A working memory model of a common
procedural error. Cognitive Science, 21(1):31–61, 1997.
[15] B. A. Campbell and M. D. Byrne. Now do voters notice review
screen anomalies? A look at voting system usability. In USENIX Electronic Voting Technology Workshop/Workshop on Trustworthy Elections,
EVT/WOTE, 2009.
[16] Clear Ballot Group. ClearAccess administrators guide, 2015. https://www.
sos.state.co.us/pubs/elections/VotingSystems/systemsDocumentation/
ClearBallot/ClearAccess/ClearAccessAdministratorsGuideRev4-0-r0.
pdf.
[17] A. Cordova, L. Howard, and L. Norden. Voting machine security: Where
we stand a few months before the New Hampshire primary. Brennan
Center, 2019. https://www.brennancenter.org/analysis/voting-machinesecurity-where-we-stand-six-months-new-hampshire-primary.
[18] L. F. Cranor. A framework for reasoning about the human in the loop. In
1st Conference on Usability, Psychology, and Security, UPSEC. USENIX,
2008.
[19] M. De Jong, J. Van Hoof, and J. Gosselt. Voters’ perceptions of voting
technology: Paper ballots versus voting machine with and without paper
audit trail. Social Science Computer Review, 26(4):399–410, 2008.
[20] R. DeMillo, R. Kadel, and M. Marks. What voters are asked to verify
affects ballot verification: A quantitative analysis of voters’ memories of
their ballots, 2018. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=
3292208.
[21] S. Egelman, L. F. Cranor, and J. Hong. You’ve been warned: An empirical
study of the effectiveness of web browser phishing warnings. In 26th
ACM Conference on Human Factors in Computing Systems, CHI, pages
1065–1074, 2008.

14[22] S. P. Everett, K. K. Greene, M. D. Byrne, D. S. Wallach, K. Derr,
D. Sandler, and T. Torous. Electronic voting machines versus traditional
methods: improved preference, similar performance. In 26th ACM
Conference on Human Factors in Computing Systems, CHI, pages 883–
892, 2008.
[23] A. J. Feldman, J. A. Halderman, and E. W. Felten. Security analysis of
the Diebold AccuVote-TS voting machine. In USENIX Electronic Voting
Technology Workshop, EVT, 2007.
[24] M. Fitts. SC chooses new voting machines that will print paper
ballots but some fear it’s not safe. The Post and Courier, June
10, 2019. https://www.postandcourier.com/article_f86632ce-8b83-11e98dab-5fb7858906cc.html.
[25] S. Fowler.
Georgia awards new voting machine contract toDominion Voting Systems.
Georgia Public Broadcasting, July29, 2019. https://www.gpbnews.org/post/georgia-awards-new-votingmachine-contract-dominion-voting-systems.
[26] S. N. Goggin and M. D. Byrne. An examination of the auditability ofvoter verified paper audit trail (VVPAT) ballots. In USENIX Electronic
Voting Technology Workshop, EVT, 2007.
[27] K. K. Greene, M. D. Byrne, and S. P. Everett. A comparison of usability
between voting methods. In USENIX Electronic Voting Technology
Workshop, EVT, 2006.
[28] Indiana Fiscal Policy Institute. Vote centers and election costs: A studyof the fiscal impact of vote centers in Indiana, 2010. https://www.in.gov/
sos/elections/files/IFPI_Vote_Centers_and_Election_Costs_Report.pdf.
[29] D. Jones and B. Simons. Broken Ballots: Will Your Vote Count? CSLI
Publications, 2012.
[30] D. Kasdan. Early voting: What works. https://www.brennancenter.org/
sites/default/files/publications/VotingReport_Web.pdf.
[31] T. Kohno, A. Stubblefield, A. D. Rubin, and D. S. Wallach. Analysis ofan electronic voting system. In 25th IEEE Symposium on Security andPrivacy, 2004.
[32] M. Lindeman and P. Stark. A gentle introduction to risk-limiting audits.
IEEE Security & Privacy, 10:42–49, 2012.
[33] J. Mack. Who votes in Michigan? A demographic breakdown. MLive,2018. https://www.mlive.com/news/erry-2018/11/340b0f9c406363/whovotes-in-michigan-a-demogr.html.
[34] S. Maneki and B. Jackson. Re: Comments on Ballot Marking Devices
usage for the 2018 elections, 2017. Letter to Maryland State Board ofElections, citing SBE data.
[35] P. McDaniel, M. Blaze, and G. Vigna. EVEREST: Evaluation andvalidation of election-related equipment, standards and testing. Technicalreport, Ohio Secretary of State, 2007. https://www.eac.gov/assets/1/28/
EVEREST.pdf.
[36] National Academies of Sciences, Engineering, and Medicine. Securing
the Vote: Protecting American Democracy. The National Academies
Press, Washington, DC, 2018.
[37] National Conference of State Legislatures. Funding elections technology,
2019. https://www.ncsl.org/research/elections-and-campaigns/fundingelection-technology.aspx.
[38] R. G. Niemi and P. S. Herrnson. Beyond the butterfly: The complexityof U.S. ballots. Perspectives on Politics, 1(2):317–326, 2003.
[39] M. M. Olembo and M. Volkamer. E-voting system usability: Lessons forinterface design, user studies, and usability criteria. In Human-Centered
System Design for Electronic Governance, pages 172–201. IGI Global,
2013.
[40] S. Patil, R. Hoyle, R. Schlegel, A. Kapadia, and A. J. Lee. Interrupt now
or inform later? Comparing immediate and delayed privacy feedback. In
33rd ACM Conference on Human Factors in Computing Systems, CHI,
pages 1415–1418, 2015.
[41] J. Petelka, Y. Zou, and F. Schaub. Put your warning where your link
is: Improving and evaluating email phishing warnings. In 37th ACM
Conference on Human Factors in Computing Systems, CHI, 2019.
Colorado voting reforms: Early re[42] Pew Charitable Trusts.
sults. https://www.pewtrusts.org/ /media/assets/2016/03/coloradovotingreformsearlyresults.pdf, 2016.
[43] Pro V&V. Test report for EAC 2005 VVSG certification testing: ClearBallot Group ClearVote 1.4 voting system, 2017. https://www.eac.gov/file.
aspx?A=kOBM5qPeI8KZlJyADXYTieiXLwsxw4gYKIVroEkEBMo%
3D.
[44] W. Quesenbery. Ballot marking devices make voting universal. Centerfor Civic Design, 2019. https://civicdesign.org/ballot-marking-devicesmake-voting-universal/.

[45] W. Quesenbery, J. Cugini, D. Chisnell, B. Killam, and G. Reddish. Letterto the editor: Comments on “A methodology for testing voting systems”.
Journal of Usability Studies, 2(2):96–98, 2007.
[46] R. W. Reeder, A. P. Felt, S. Consolvo, N. Malkin, C. Thompson, andS. Egelman. An experience sampling study of user reactions to browser
warnings in the field. In 36th ACM Conference on Human Factors in
Computing Systems, CHI, 2018.
[47] R. Rivest. On the notion of ‘software independence’ in voting systems.
Philos. Trans. Royal Soc. A, 366(1881):3759–3767, October 2008.
[48] T. Selker, E. Rosenzweig, and A. Pandolfo. A methodology for testing
voting systems. Journal of Usability Studies, 2(1):7–21, 2006.
[49] T. Selker, E. Rosenzweig, and A. Pandolfo. Reply to comment on:
The Methodology for Testing Voting Systems by Whitney Quesenbery,
John Cugini, Dana Chisnell, Bill Killam, and Ginny Redish. Journal of
Usability Studies, 2(2):99–101, 2007.
[50] P. Stark. Conservative statistical post-election audits. Annals of Applied
Statistics, 2(2):550–581, 2008.
[51] P. B. Stark. There is no reliable way to detect hacked ballot-marking
devices, 2019. https://arxiv.org/abs/1908.08144.
[52] J. Sunshine, S. Egelman, H. Almuhimedi, N. Atri, and L. F. Cranor.
Crying wolf: An empirical study of SSL warning effectiveness. In 18th
USENIX Security Symposium, pages 399–416, 2009.
[53] United States Senate Select Committee on Intelligence. Report on Russian15

active measures campaigns and interference in the 2016 U.S. election,
2019. https://www.intelligence.senate.gov/sites/default/files/documents/
Report_Volume1.pdf.
[54] U.S. Census Bureau. QuickFacts: Ann Arbor, 2019. https://www.census.
gov/quickfacts/annarborcitymichigan.
[55] U.S. Election Assistance Commission. Designing polling place materials. https://www.eac.gov/election-officials/designing-polling-placematerials/.
[56] Verified Voting. Ballot marking devices. https://www.verifiedvoting.org/
ballot-marking-devices/.
[57] Verified Voting. The Verifier: Polling place equipment. https://www.
verifiedvoting.org/verifier/.
[58] VSAP. Voting system for all people. https://vsap.lavote.net/.
[59] Wall Street Journal. Election 2018: How we voted in the 2018
midterms, November 6, 2018. https://www.wsj.com/graphics/election2018-votecast-poll/.
[60] D. S. Wallach. On the security of ballot marking devices, 2019. https://
arxiv.org/abs/1908.01897.
[61] M. S. Wogalter. Communication-human information processing (C-HIP)
model. In M. S. Wogalter, editor, Handbook of Warnings, chapter 5,
pages 51–61. Lawrence Erlbaum Associates, Mahwah, NJ, 2006.
[62] M. S. Wogalter and K. R. Laughery. Warning! sign and label effectiveness.
Current Directions in Psychological Science, 5(2):33–37, 1996.

A PPENDIX A
P OLL W ORKER S CRIPTA PPENDIX B
R ECRUITMENT S CRIPT

Our poll workers followed four versions of the script below:
An investigator used the following script to recruit librarya baseline version, and three variants that each add one line. patrons to participate in the study:
VARIANT 1: Before the voter begins using the BMD, a poll Hello, do you have 10 minutes to participate in a study about
worker asks them to check their ballot before it is scanned. a new kind of voting machine that is used in elections across
VARIANT 2: Before the voter deposits the ballot, a poll worker the United States? This study will consist of voting using our
informs them that it is the official record of the vote.
voting machine and depositing a printed paper ballot into a ballot
VARIANT 3: Before the voter deposits the ballot, a poll worker box, and then filling out a survey about the experience. If you
asks whether they have carefully reviewed each selection. would like to participate, we will need you to first sign a consent
form. We will provide a flyer at the end of your participation
When Subject Arrives (P OLL W ORKER A)
with information about the study. We cannot make all details
Hello! Before you begin, please fill out this Institutional Review available at this time, but full details and research results will be
Board consent form. [ Point to form and pen.] If you have any made available within six months of the conclusion of this study.
We thank you for your consideration and hope you choose toquestions, feel free to ask.
participate!
You are about to participate in a study about the usability of
A PPENDIX C
a new type of voting machine. You will be using one of these
S
LATE
OF
C
ANDIDATESFOR D IRECTED VOTING C ONDITION
voting machines to make selections on your ballot, which will bea truncated version of the Ann Arbor 2018 midterm ballot. Once
We randomly generated a slate of candidates and provided
you are finished, your ballot will be printed from the printer a printed copy to voters in certain experiments. The handout
beneath the machine, and you can review your ballot and deposit voters received is reproduced below:
it in the ballot box over there. [ Point out ballot box.] Feel free to
RaceCandidate(s)vote your political preference or not; no identifying information
Bill
Gelineau
andwill be collected that could match you with your votes. If you
Governor and Lieutenant Governor
Angelique Chaiser Thomaswould like to quit at any time during the study, just say so.
VARIANT 1: Please remember to check your ballot carefully
before depositing it into the scanner.
You may begin at any time.
Before Subject Deposits Ballot (P OLL W ORKER B)

Secretary of State

Mary Treder Lang

Attorney General

Lisa Lane Gioia

United States Senator

Debbie Stabenow

Representative in Congress 12th District

Jeff Jones

Member of State Board of Education

Tiffany Tilley
Mary Anne Hering(Vote for 2)

VARIANT 2: Please keep in mind that the paper ballot is theofficial record of your vote.

Regent of the University of MIchigan(Vote for 2)

Trustee of Michigan State University(Vote for 2)

VARIANT 3: Have you carefully reviewed each selection on yourprinted ballot?

Justice of the Supreme Court(Vote for 2)

Jordan Acker
Joe Sanger
Mike Miller
Bruce Campbell
Megan Kathleen Cavanagh
Kerry Lee Morgan

After Subject Deposits Ballot (P OLL W ORKER B)

Judge of Court of Appeals 3rd District
Incumbent Position ​(Vote for 2)

Jane Marie Beckering
Douglas B. Shapiro

Thank you for participating! You are now finished with the study,
and should fill out the exit survey. [ Point to debrief survey
computers.]

Judge of Circuit Court 22nd Circuit
Incumbent Position ​(Vote for 2)

Timothy Patrick Connors
Carol Kuhnke

Judge of Probate Court Incumbent
Position

Darlene A. O’Brien

After Subject Completes Exit Survey (P OLL W ORKER B)

Judge of District Court 14A District
Incumbent Position

Thomas B. BourqueThank you for your participation! You are now finished. If you
have any questions about this study, you may ask them now,
although I am unable to answer some questions due to the natureof the research. Here is a debrief form. [ Hand subject a debrief
form.] If you think of anything after you leave, you can reach[me/the principle investigators] through the information on thedebrief form.
If you know anyone who might like to participate, please referthem here; we will be here [remaining time].
Thank you again for participating!

16