VVSG Tutorial[^1]

Usability, Accessibility, and Privacy Part 3

*\[Slide 1\]*

\[MODERATOR:\] The following presentation is Part 3 of the Next
Voluntary Voting System Guidelines Training Module on Usability,
Accessibility, and Privacy. This section of the next VVSG is located in
Part 1, Chapter 3, of the draft document. The presenter is Dr. Sharon
Laskowski of NIST's Information Technology Laboratory. This section
overview includes question and answer sessions with members of the
Election Assistance Commission's Board of Advisors and Standards Board.\
*\[Slide 2\]*

\[MS. LASKOWSKI:\] Continuing on with the blindness requirements, there
are also requirements that support blind people being able to
independently activate their ballots, submit their ballot, and perform
verification. The controls should be tactically discernable, and the
status should be discernable of the current state. *\[Slide 3\]*

For dexterity, these requirements are specific for features designed to
assist voters who lack fine motor control or use of their hands. I think
these haven't been changed from VVSG05, which support nonmanual input,
submission verification, manipulation of controls, etc. *\[Slide 4\]*

Mobility was directly based on ADA accessibility guidelines for
buildings and facilities. Again, this is almost directly from the ADA,
and this was in collaboration with the Access Board for basically
appropriate placement, spacing, floor space, clearances, obstructions so
that people in wheelchairs can have good access to the voting station.*\
\[Slide 5\]*

3.3.6 has to do with hearing. So for example, if there are any sound
cues, you have to have some kind of visual redundancy and no
electromagnetic interference with hearing, etc. There are also audio
requirements for volume control, etc.\
*\[Slide 6\]*

3.3.7 has to do with general support for cognitive disabilities. The
accessible voting station should provide support to voters with
cognitive disabilities. This is a 'should,' because there is a wide,
wide range of cognitive disabilities. Basically what you can do is be
cognizant of this, but there are certain ranges that we don't have good
design requirements for.*\
\[Slide 7\]*

Jim.

\[QUESTIONER:\] Yes, we have been requesting for many years a study on
the use of icons and pictures. Has that study been commissioned?

\[MS. LASKOWSKI:\] We have done some preliminary work, and I can
certainly summarize here. It's a very interesting aspect. So in a
particular standard, we don't want- with design, we can't go
specifically into what, say, party icons look like, etc. So we can sort
of put in general design recommendations. With respect to pictures, the
research shows that pictures can easily introduce biases, gender biases,
race biases, etc. When you are reproducing, say, pictures of candidates,
often on paper ballots, the resolution is poor, so you don't get very
good pictures. So there are definite human factors issues with using
pictures. With respect to icons, in particular, people with severe kinds
of cognitive disabilities, typically they have, say, icons in any
software that they use that are specifically designed for that person
and with a lot of training. Obviously we can't do that on a public
kiosk. Icons are subject to a lot of interpretation depending on
background. There are very few universal icons. There are some icons on
a voting machine such as arrows, for example. In general, animation is
an interesting issue. If there's animation on a Web page, you know, the
new icon flickers- very bad from a human factors point of view because
that distracts, and there has been a lot of eye tracking studies that
say that kind of animation to point out things distracts the user from
the task at hand.

\[QUESTIONER:\] I don't want us to go way afield here, but-

\[MS. LASKOWSKI:\] Okay, so I've done some preliminary work. The TGDC
did not ask for this study. So there is no study, but there are severe
human factors issues with icons and pictures. I think that's why the
TGDC did not encourage that study.

\[QUESTIONER:\] Well, there are states and countries and territories
that use icons, and it strikes me as peculiar that the impact of that
has not been studied. We're talking about elections; we're not talking
about other worlds.

\[MS. LASKOWSKI:\] Brian Hancock has a question/comment.

\[MR. HANCOCK:\] Well, the way I read HAVA, it doesn't, you know, it
talks about all disabilities, it doesn't separate cognitive as being a
'should' as opposed to a 'shall.' I understand whether there was a
discussion about making it a 'shall.' If so, why-

\[MS. LASKOWSKI:\] Oh, so the question is, in the general support for
cognitive disabilities, was there a discussion of making this a 'shall'?
That's because it would have been really impossible to test because of
the wide range of cognitive disabilities. So there was some discussion,
and this is purely a high-level goal.

\[QUESTIONER:\] Yes, I will submit to you more information about the
issue that Jim raised, because I think using a photo and the insignia of
the parties so that people can choose the candidate will solve a lot of
the problems of illiteracy and language, because people can associate
more easily the insignia and photo of the candidate and solve the
problem of the different languages, and people that don't know how to
read and so on and so forth. So I will send you more information.

\[MS. LASKOWSKI:\] Okay, the comment from, what was your first name?

\[QUESTIONER:\] Nestor.

\[MS. LASKOWSKI:\] Nestor, the question is that in dealing with
illiteracy, party icons and a candidate's photograph can help, and
language- Certainly audio is another alternative way to help with those
language illiteracy issues as well. And I should point out that there is
nothing in the standard that precludes the use of party icons or photos
in a state ballot. It's certainly not forbidden in the standard.\
*\[Slide 8\]*

Okay, 3.3.8 talks about English proficiency. And here, actually, the
question from Nestor ties directly in with that. Voters who have
proficiency in reading English can get instructions on ballots via the
audio interface. Question from Shelly.

\[QUESTIONER:\] Perhaps this would be a good area to include the
unwritten languages in this section as well, that would allow the--

\[MS. LASKOWSKI:\] To mention that, yes, I don't know if it's in the
discussion or not, but that certainly- . Yes, for unwritten languages,
the use of the ATI also is a method to deal with unwritten languages.
3.3.9: speech, merely says that speech is not to be required as part of
the user interface of the equipment, that is, you know, speech input.*\
\[Slide 9\]*

Okay, so now I would like to talk about the usability performance
requirements, which is new material. Finally, are there any other
questions on the accessibility section?

Okay, so our goal in developing usability performance requirements is to
develop a test method to distinguish systems with poor usability from
those with good usability, based on performance, not evaluation of the
design. And I can't emphasize this enough that the performance and being
able to measure the usability performance really gives you the bottom
line of how good the system is when it's actually in use and the voters
are interacting with it. You can have many good- follow many of these
small design requirements, and the system still might not have good
usability. It's no guarantee. So that's why we worked very hard on
trying to develop these performance requirements. And performance
requirements need a test method that reliably detects and counts errors
that one might see when a voter interacts with a voting system, and
because a test laboratory has to see if these performance requirements
are met, the test method has to be reproducible by a test laboratory.

And one advantage is that it is technology-independent. You could take
any voting system you want and apply the test method to see if the
system has good usability or not, totally technology-independent.*\
\[Slide 10\]*

Given that we could design such a test method, measuring voters
interacting with a benchmark. You pick a number, you pick some metrics
on how effective, how many errors did the voter make, pick a number, if
they do better, fewer than this number of errors, they pass the test.
They have good usability; more than that, they fail. So you can use the
benchmarks to do a pass/fail. And so then the values that you choose for
the benchmark become the performance requirements. So I need to tell you
about what are we measuring, how do we measure it, and how did we choose
our benchmarks.\
*\[Slide 11\]*

So we are talking about doing usability testing for certification in a
laboratory. We want to measure the performance of the system in a lab.
We want a control for other variables including what our test
participants are like and measure just the effect of the system on the
usability. So that means you need a standard test ballot designed in
such a way that you can detect many of the different types of usability
errors, that is also typical- because this is in the test laboratory,
that's typical of many types of ballots. You want to control the test
environment for lighting, setup, instructions; we don't allow
assistance. Then you use enough test participants in such a way that you
can, given repeating the test, you can reliably detect the same
performance on the same system every time you run the test.*\
\[Slide 12\]*

Jim, question.

\[QUESTIONER:\] What is the actual page?

\[MS. LASKOWSKI:\] Oh, oh, okay, I'm sorry. I was right at the end of
the accessibility section. I talked about, oh gee, let me back up.

\[MALE SPEAKER:\] Chapter 3, page 29.

\[FEMALE SPEAKER:\] Chapter 3, page what?

\[MALE SPEAKER:\] Page 29.

\[MS. LASKOWSKI:\] Oh, in the standard itself, okay. A lot of the
background of the test method is not there. It's on the slide itself. I
just completed the slide entitled "Usability Testing for Certification
in a Lab." There were two of those slides, and now I am on "Components
of the Test: Our Method Voting Performance Protocol."\
*\[Slide 13\]*

Does that help orient you? The title of the slide is "Components of the
Test Method Voting Performance Protocol." In the standard, it's 3.2.1.1.

Okay, so we need this reliable test method. And so we defined a test
protocol that describes the number and characteristics of voters that
are going to participate in this test. I call them test participants and
how to conduct the test. We've got a standard test ballot that's
relatively complex. We want to make sure we capture significant errors
as we evaluate the voting system with this ballot. Matt, question.

\[MR. MASTERSON:\] Sorry to interrupt. Who is designing this test ballot
because- ?

\[MS. LASKOWSKI:\] NIST, NIST designed the test ballot. It must remain
the same. All components of the test have to be the same, because we
have to be able to repeat this, reproduce it in different test
laboratories. So it's a standard test ballot. John has a comment.

\[MR. CUGINI:\] We specified the logical ballot, what the other guys
were calling ballot configuration, but the actual low-level design of
the fonts and the colors, the vendor decides that. We give them the
logical ballot and say, you now make this look as good as you can do.

\[MS. LASKOWSKI:\] Let me give the background on that. The manufacturer
knows they are going to submit for this test. We say give it your best
shot designing this ballot on your system. We didn't want them to claim
the test lab did a terrible job designing the ballot. *\[Slide 14\]*\
Jim, question.

\[QUESTIONER:\] I'm trying to understand something here so I'm going to
just-- I want to know if what I'm about to say is an accurate reflection
of what's in the standard. Regarding people with disabilities, testing
the usability of the system, there is nowhere in the standard where it
defines or explains which disabilities are included, how many people
should do it, that kind of thing.

\[MS. LASKOWSKI:\] Okay. Jim asks about what about people with
disabilities and the testing. For these usability benchmarks, you will
note they are in the usability section. We've only done the research on
developing the test method, and these are for testing the regular voting
station, not the accessible voting station. We do have research under
way because the benchmarks would be different, have different values for
the accessible voting station because it's different, for example,
timing might be different. I take that back. The effect on these
benchmarks, how accurate the voter is, one would hope, are the same, but
we don't have any data because we haven't run those experiments yet but
we are- we have that in the plans.

The major challenge for this pass was to be able to develop, to show
that we could develop a valid test method that has repeatable results.
We've done that, so the next stage is to perform some experiments with
various demographics of users with disabilities to see how they perform
against these benchmarks. But I don't have that data yet.

\[QUESTIONER:\] In terms of these standards, is that data going to be
available to inform what goes in the standard or is the standard going
to be completed without that data?

\[MS. LASKOWSKI:\] Well, I have-- all the data we have collected so far
is on the NIST Web site, and I can send you the link to that. As we
collect more data, it will be published on our Web site.

\[MR. SKALL\]: That data will be available before final disposition is
done.

\[MS. LASKOWSKI:\] Correct.

\[FEMALE SPEAKER:\] Will it be included?

\[QUESTIONER:\] Let me see if I understand. The data is over here on
your Web site, separate and apart from that is the VVSG 2. Correct?
There is no plan to take the data from what's on the Web site and use it
to inform the VVSG 2?

\[MS. LASKOWSKI:\] The data that we have so far inform what those
benchmarks are. One would expect or one would hope that the performance
of the accessible stations would be the same, because they are
effectiveness, accuracy benchmarks. So the benchmark numbers may change,
we are talking about perhaps making them more stringent, but we are
still collecting some additional data on some variations of the test.
Tom Wilkey has a question.

\[MR. WILKEY:\] Yes, this might be a question for Mark. Is this going to
be part of the overall test scripts to be developed for the labs?

\[MS. LASKOWSKI:\] Yes, is this going to be part of the test scripts?
The answer is yes, and the reason you are hearing about the test method
now is because we couldn't develop the benchmarks without the test
method, which is different than the design issues because you can put
the design in and then specify the test.

\[MR. SKALL:\] Let me just clarify Jim's question. We have a VVSG, and
we have further information that we are gathering, and we have some
information we have already gathered. The plan is definitely to allow
that information to impact the final VVSG, and we will work with the EAC
to determine exactly how that gets done. But the whole reason we are
continuing to do research during this public review period is so that
the input of that research can be- what we finally decide to publish for
the VVSG. So, again, as to how we do that, we will work closely with the
EAC to determine how to do that.

\[MS. LASKOWSKI:\] Wendy has a question.

\[QUESTIONER:\] Okay, I want to be really clear on this. The intent is
that you will have performance benchmarks for the regular equipment and
the accessible voting station?

\[MS. LASKOWSKI:\] Well, the intent of the TDGC was to use those same
benchmarks. People with disabilities ought to be having the same kind of
error rates one would hope on that benchmark, but I have not collected
that data and specified what that test for the accessible system and the
demographics, as Jim pointed out, for users with disabilities at this
point.

\[QUESTIONER:\] All right, so all of the performance requirements, the
intent is for them to be dealt with for all equipment, not just --

\[MS. LASKOWSKI:\] Yes, and as I said, the test method is
technology-independent so you can apply it to any equipment. As you will
see as I go through the test method and how we picked the voters, the
missing component is what would be the demographics for choosing voters
with disabilities for testing. I haven't done those experiments yet.
Question in the corner?

\[QUESTIONER:\] Wes.

\[MS. LASKOWSKI:\] Yes Wes.

\[QUESTIONER:\] My question falls under the time line that I understand
Mark is indicating is going to happen. If you are collecting this
additional data so you can set your benchmarks and provide input-

\[MS. LASKOWSKI:\] We have set the benchmarks already.

\[QUESTIONER:\] All right.

\[MS. LASKOWSKI:\] But we may adjust them.

\[QUESTIONER:\] What is available to look at what occurred if you are
doing it concurrent with the public comment period?

\[MS. LASKOWSKI:\] Brian.

\[MR. HANCOCK:\] Well, there are going to be two public comment periods.
It's going to depend a lot on when they get that information to us. The
public comment period is going to be starting shortly, an initial
120-day public comment period on this document we are talking about
today. From the public comments we receive, there will be changes made
to this document. That would be the point where I believe we would
incorporate that information into the second iteration of the public
comment process.

\[MS. LASKOWSKI:\] Yes, what's sort of missing, and we are working on
test methods. The test methods we are developing are not part of the
public comments. You need to know enough about how we generated those
benchmarks for this version to make comments on that particular
benchmark.*\
\[Slide 15\]*

Okay, so let me go through what the components of the test method are.
We have to define what our test voters look like, how many of them to
use. We've developed a standard test ballot specification that's put on
the voting system to be tested.

And how do you count errors? Again, remember we are measuring the system
usability. We tell the voters, we give them instructions, and say, vote
this way. Then we see if they can do it. Then we can count errors. In
fact, on our test ballot, there are twenty-eight voting choices. So we
can calculate how many of the twenty-eight did they get correct. Brit,
question?

\[QUESTIONER:\] When your voters are voting on this ballot, you say you
give them instructions on how to vote.

\[MS. LASKOWSKI:\] Correct. We give the voters instructions on how to
vote.

\[QUESTIONER:\] People make errors. Do you have two people, one doing
the voting and one watching for errors? How do you distinguish between
things within the system and just plain old vanilla human errors?

\[MS. LASKOWSKI:\] We are only looking at the human errors. So we have
observers and depending on the system, you can get the results out the
other end. So we can count the errors that way. So it depends on the
system, but either observe or use the system capability to count the
errors. We are not evaluating inside the machine; we are only looking at
what the voter did and what errors the voter made. Did I answer your
question, you look puzzled.

\[QUESTIONER:\] Yeah, you answered the question, but I-

\[MS. LASKOWSKI:\] Paul?

\[QUESTIONER:\] I'm going to follow up on that question. I think there
is some potential and perhaps you incorporate it in your test
methodologies. There are errors people make following a script, and then
there are errors people make because they can't figure out how to do
what they are trying to do.

\[MS. LASKOWSKI:\] Yes, we control for that, though, because all the
voters get exactly the same script. Jim, question.

\[QUESTIONER:\] The script that you use and the ballot length in the
United States is all over the lot. So if it's a real short script, you
are going to have different results than if you are dealing with
California, where there could be a five-page ballot.

\[MS. LASKOWSKI:\] That's correct. So what we have to do, so Jim makes
the comment that there are lots of differences in ballots. What we did
was, we did a survey of ballots, and we put together what we considered
was a moderately complex ballot that is designed to draw out most errors
one would see that voters make. Remember, we are not measuring on a
specific ballot and specific state; this is a general laboratory test
that is going to discriminate across systems, which are more usable than
other systems. Okay, so going through the components of the test method.
For this to be run in a lab, we specified strictly the test environment
in great detail and also the methods of analyzing and reporting the
results. We did some studies, and I'll show you some of the data in a
moment that allowed us to pick some threshold values.\
*\[Slide 16\]*

So, initially we said, well, we have this test method, script, protocol-
is this a valid method? In other words, can we detect differences and
produce the errors that we expected? Is it a sensitive enough test that
would make it valid?

We looked at two different systems. We used 47 participants, relatively
well-educated participants, and we were surprised that we actually,
given that set, that we detected all the different errors that could
possibly be made. So we said, okay, this test is definitely a sensitive
test to differences in usability across systems. It detected differences
between these two systems. One system did better on some things than the
other system. So then the question is, can we repeat this with different
test voters given this very controlled environment and set of test
voters? Each test is done with different test voters. What we did was
four tests on the same system using 195 participants, and we got similar
results on that system for each test. That means it's a repeatable- it's
a reliable test. *\[Slide 17\]*

So for this particular test, what were our demographics like? Remember
we got all the different kinds of errors, so basically we had 60%
female, mostly Caucasians, some African-American, some Hispanic.
Education was 20% some college, 50% college graduates, 30% post
graduates. Age range between 25 and 54. It was in the Virginia,
Maryland, D.C. area. We are doing some follow-up to look at performance
with a few different regions and with an older population, a slightly
less educated population. What we were trying to do is first set for
these initial experiments, a population that was easy to recruit.
Remember we have to send this test out to test laboratories. For them to
actually recruit these users, and they are going to need to do 100 per
test, to keep cost down, they have to be able to get those users.

The amazing thing was that we definitely detected errors, and we
detected differences between the machines. So our test instrument is a
valid test instrument. Jim, question.

\[QUESTIONER:\] Yes, 30% of the testers have postgraduate degrees. Does
that reflect the electorate?

\[MS. LASKOWSKI:\] No. We weren't trying to get a sample across the U.S.
population. I'll go into that a little bit later. What we were just
trying to do is detect usability errors of the system. Now we are doing
a few other experiments, as I say, with a slightly larger range of
demographics. The bottom line is, even with 30% post graduates, they
were making all sorts of errors on these machines, usability errors. So
the test was very sensitive.*\
\[Slide 18\]*

Okay, so we took four systems, a selection of DRE, EBMs, and PCOS. We
had 187 test participants. We were taking five measurements: three
benchmark thresholds which I'll go into in a minute and two values to be
reported. We found vast differences with this test protocol on these
user demographics.*\
\[Slide 19\]*

So let me first go over the performance measures. Then I'll show you our
data and show you how we chose the benchmark levels. We have what we
call base accuracy score. I tried not to put too much math in here. I
don't think that the calculations are that difficult. There are
twenty-eight voting opportunities. We could count how many were correct
for each participant. That gives us a base accuracy score. That's the
mean percentage of all the ballot choices that are correctly cast by the
test participants. We just do a score for each participant, you know,
twenty-six out of twenty-eight, fourteen out of twenty-eight, and then
take a mean of that. That gives us a base accuracy score to work with.
*\[Slide 20\]*

So then we used this in several different ways. First we calculate a
total completion score. Actually this one doesn't use the base accuracy
score. This just looks at the percentage of test participants who are
able to complete the process of voting and cast their ballot. What
percent--

\[QUESTIONER:\] You timed them?

\[MS. LASKOWSKI:\] We did time them, but we just report that metric.
Timing we found wasn't correlated to the accuracy. Some people are very
accurate and take their time; some people are fast and accurate. There
was really no correlation, so we present that because we think election
officials will find timing useful.\
*\[Slide 21\]*

We calculate what we call voter inclusion index, which is a measure of
the overall voting accuracy but we use the base accuracy score and the
standard deviation. The reason we are using a standard deviation is,
suppose two systems have the same mean accuracy scores, this base
accuracy score, but one had voters, some of which did really well and
some did really poorly. When you take the average, it might look the
same as a system that had very consistent score. So we said getting that
variability is not good, they should get a lower voter inclusion index.

Basically we used a formula that our statisticians at NIST recommended
that divides by the standard deviation. So take the base accuracy score.
We use a lower specification limit that we subtract off just to get a
nice spread of numbers so we don't get numbers real close together on
the system, divided by the standard deviation. If there is a large
standard deviation with the same base accuracy score, that system will
have a lower voter inclusion index.\
Okay. Again, the reason we are using a standard deviation is, suppose
two systems have the same mean accuracy score, this base accuracy score.
But one had voters some of which did really well and some did really
poorly. When you take the average, it might look the same as a system
that had very consistent score. So we said getting that variability is
not good, they should get a lower voter inclusion index. And so
basically we used a formula that our statisticians at NIST recommended
that divides by the standard deviation. So take the base accuracy score.
We use a lower specification limit that we subtract off just to get a
nice spread of numbers so we don't get numbers real close together on
the system, divided by the standard deviation. If there is a large
standard deviation with the same base accuracy score, that system will
have a lower voter inclusion index.\
*\[Slide 22\]*

We've got the perfect ballot index as our third measure. So this takes
the number of cast ballots that contain no errors at all and compares it
to the number of ballots that contained one error or more. It
deliberately magnifies the effect of even a single error. The reason we
have this measure is because, if you have got a high base accuracy score
but there may be one particular part of the system that causes a similar
error across many participants. The base accuracy score looks okay, the
voter inclusion index looks okay, but there may be a serious design
problem-

The perfect ballot index- the higher the value of the perfect ballot
index, the better the performance of the system on the perfect ballot
index.*\
\[Slide 23\]*

We also looked at average voting time. That's the time taken by the test
participants from the time they activate their ballot to casting the
ballot. We also gave them a questionnaire to look at the confidence. As
I told in answer to Jim's question, the voting session time was not
correlated to the accuracy, so we decided to just report the session
time. Voter confidence in general, even on the systems where the
accuracy was poor, the voters were pretty confident that they voted
correctly. I think that's useful information to collect, because if a
system shows that the voters are not confident, the election officials
purchasing that system might want to know about that, because that has
to do with public perception. We don't report that.\
*\[Slide 24\]*

So, I have a chart here that has our benchmark test results. Each row is
for four systems: A, B, C, and D. We had approximately fifty people on
each system- fifty different people on each system. And the second
column is the total completion score. Because these are all statistical
tests, we use a confidence interval. So all we can say is with 95%
confidence, the interval given, that is where the true value of the
total completion score is. So, for example, let's pick system B, our
completion score, we have 95% confidence that it was somewhere between
92.8% and 100%. And notice that for all four systems, we had pretty
good- , except for maybe system D, which had a confidence interval from
83 to 98%, pretty good completion score, which is what you want. You
want people to complete the ballot. For that reason, we chose our
benchmark for the total completion score to be 98%. John, correct me so
I don't have to bring up my sets -- 98%, correct? So I don't have to
flip between slides.

\[MR. CUGINI:\] Right.

\[MS. LASKOWSKI:\] Then we computed a base accuracy score. For example,
system A had a base accuracy score of 95% with a standard deviation of
11, whereas system D had a 92.4% accuracy score but a large standard
deviation of 19. When we run through that little formula I gave, we get
a range: system B has a 95% confidence interval that the voter inclusion
index is between .49 and .85, where system D is between .03 and .22.
Looking at this, we picked a voter inclusion index of .35. So for these
systems and these are all 2002 certified systems, two of them would fail
on the voter inclusion index. Ninety-eight percent, they all passed the
total completion score. So we are seeing a lot of variation and a lot of
errors in some systems, even with the demographics I outlined.\
*\[Slide 25\]*

Jim, question?

\[QUESTIONER:\] Were there any people with disabilities in your test
group?

\[MS. LASKOWSKI:\] No, there were no people with disabilities here. One
would expect that, though using accessible systems, they ought to get
the same performance on the accuracy effectiveness measures that able
people are getting. That's sort of the goal that you want to strive for.
As I said, we are going to run some of these systems through with this
test protocol with different disabilities to see what kind of results we
are actually getting on some of the systems. I don't have that data.

\[QUESTIONER:\] I have a question.

\[MS. LASKOWSKI:\] Yes.

\[QUESTIONER:\] Just looking at these results, it appears you have a
regional bias here. If all the test voters that tested this were from
the Maryland and Virginia area, and they don't have any experience with
the systems-

\[MALE SPEAKER:\] Talk into the microphone, please.

\[MS. LASKOWSKI:\] Oh, sorry. Looking at similar kiosk-based systems, we
have not seen evidence in other areas that regional differences for
these kinds of interfaces make a difference, but this question has been
raised, and we are planning some regional tests probably in the South
and Midwest.

Chicago does a lot of straight party voting, so maybe they get a
different result on our test ballot because of their experience. So we
are going to verify that conclusion, but just based on other similar
kinds of- what we see in other similar systems, that people have
studied, we don't see typically those kinds of regional differences.

\[QUESTIONER:\] How about the ballot itself? You know, ballots are
different.

\[MS. LASKOWSKI:\] Yes. That's a very good point. Ballots in a state are
different, so you can't- and this is a very controlled study, just to
pass/fail on the usability level. What we tried to do is make this
ballot have as many characteristics, general characteristics, that you
would see on most ballots in the United States. But because the ballots
in each state and election law and presentation of the ballot are
different from each state, you can't necessarily take these results and
apply them to your state. What you can do, is say you have a good feel
that if they passed them and certified to this benchmark, they have a
certain base level of usability. It gives you some assurance that you
are starting off with a good system that has good usability. That's all
you can say based on a lab test like this. Okay.

\[QUESTIONER:\] One more question. Can you tell us what these systems
are?

\[MS. LASKOWSKI:\] No. These were experiments just to test our test
methodology, and as I say, your mileage may vary depending on your
particular state and--

\[MALE SPEAKER:\] What systems were tested?

\[MS. LASKOWSKI:\] Yeah, I did say that. It was a selection: DRE, EBM,
PCOS.

\[QUESTIONER:\] Translate that jargon into words we know. The types of
systems were DRE --

\[MS. LASKOWSKI:\] DRE, where they filled out an optical scan and put it
into a precinct optical scan machine, counter, tabulator, and Electronic
Ballot Marker.

\[MS. DAVIDSON:\] Sharon, you might explain to them that there were
other tests that were done besides what you are doing, the professors
have done tests on equipment in different-

\[MS. LASKOWSKI:\] Oh, yes. Thank you, Donetta. Donetta says you might
point out there's been other research done on some systems, and I should
point out that in some of the academic research, we've seen similar
error rates. We can't compare the numbers per se because they use
different protocols, but we've seen similar kinds of errors and voter
confusion, and our tests were in line with all of that.

\[MR. SKALL:\] And also, just one more thing. This is Mark Skall from
NIST. This is really breaking new ground in voting, and this is fairly
complex. We understand that. Some of it is nonintuitive. As Sharon said,
we have worked with NIST statisticians and because this is really
breaking new ground, we have also had this peer reviewed by other people
to give us feedback, and so far we are trying to incorporate all the
comments we got.

\[MS. LASKOWSKI:\] We have some researchers from Census, Bureau of Labor
Statistics, who know their stats, know their human factors testing to
peer review all this work.\
*\[Slide 26\]*

Yes.

\[QUESTIONER:\] Can you describe the perfect ballot? I know you
described the perfect ballot index before, but as it is used here-

\[MS. LASKOWSKI:\] Yes. The perfect ballot index looks at the percent
of-

\[QUESTIONER:\] In the first column, the number of participants with
perfect ballots-

\[MS. LASKOWSKI:\] Right, right. Okay, a perfect ballot meant no errors,
what percent had absolutely not even a single error.

\[QUESTIONER:\] Is that perfection compared to a script or- ?

\[MS. LASKOWSKI:\] Yes, vote this way, did you get the twenty-eight
correct.

\[QUESTIONER:\] That's amazingly low.

\[MS. LASKOWSKI:\] We expected that we would have to do a lot more
testing and vary our demographics. Initially we just didn't want to
spend a whole lot of money until we saw what direction to go in. We were
kind of surprised.

\[QUESTIONER:\] You said post graduates --

\[MS. LASKOWSKI:\] Yeah, are they over thinking? I don't know.

So we calculate an index, you know, based on that, and our perfect
ballot index benchmark was 2.33. 2.33 is the benchmark we chose, so
system C would fail, because they had a very low perfect ballot index.

I put in the voting times, and confidence on a one-to-five scale was,
you know, three or higher. Even for system C, where they made errors,
they were very confident that they voted correctly as they were told.
*\[Slide 27\]*

So here's a summary of the benchmarks. There has been some discussion
that these were VSS2002. Are these tough enough standards? We are
collecting some more data on these. That's sort of up for looking at
some additional data and some discussion, but certainly it does weed out
some of the poor systems which was our initial goal. I saw two
questions. Jim?\
*\[Slide 28\]*

\[QUESTIONER:\] You won't tell us the brand and model, but does that
chart tell us what type of device, A, B, C, D?

\[MS. LASKOWSKI:\] No. Part of the agreement we had with the
manufacturers to supply us with these machines and also part of the NIST
mission is not to evaluate products per se, but to develop test method.
So the way we were able to get these machines to test was to say, we are
not going to release the results.

They are preliminary results. When it comes to certification, you know,
clearly they would pass/fail on these benchmarks and you would know, but
it was inappropriate for us to release that information based on this
preliminary data.

\[QUESTIONER:\] I'm just wondering if you have done any similar or if
you are aware of anyone who has done similar testing as far as how
accurately people following a script fill out a paper ballot?

\[MS. LASKOWSKI:\] Yes, there was a question of have we looked at any
research of how accurate people follow a script to fill out a paper
ballot.

I'm not aware of any information directly applicable, but as I say,
since everyone has the same script to follow, we control for that. John,
did you want to add to that?

\[MR. CUGINI:\] Yes. I'll add a little, and I just don't have the
citation at hand, but there was one test where they tested accuracy both
using a script the way we did and telling the voter, vote however you
want, and then tell us how you think you voted. So they used both the
kind of unguided method and guided method, and the results were pretty
accurate in terms of accuracy. They got pretty much the same error rate.
There is parallel research, and these error rates, you know, Sharon said
it, but I will say it again, the error rates we are finding are in line
with other research. These are not, these results may look bizarre, but
they seem to reflect reality as far as the research that's been done.

\[MS. LASKOWSKI:\] Brit had a question.

\[QUESTIONER:\] No. A comment on your system confidence on a scale of
one to five, if you converted that, I assume, four would be 80%. That's
a fairly common number on exit polls.

\[MS. LASKOWSKI:\] Brit said if you convert, four out of five is 80%.
That's a common number you get on exit polls. Thank you. It's good to
know. Chris.

\[CHRIS THOMAS:\] I just want to note that Mike Trowget of the
University of Michigan and the guy from the University of Maryland --

\[MS. LASKOWSKI:\] Paul Hernson.

\[CHRIS THOMAS:\] While they weren't per se paper ballots, they were
optical scan ballots, but they were also testing-

\[MS. LASKOWSKI:\] Mike Burns has some interesting results out of Rice
as part of the accurate voting project, as well. Wendy.

\[QUESTIONER:\] Do you have a copy of the ballot you used? I'm just
trying to get an idea--

\[MS. LASKOWSKI:\] Yes. Actually if you go to vote.nist.gov and go to
the TDGC meeting for August 17, and scroll down, it's under the human
factors, all the data is posted, and the ballot specification is there.
*\[Slide 29\]*

So if you look into your VVSG under Section 3.2.1.1-A, B and C, you will
see the wording for these benchmarks.

For example, total completion performance. The system shall achieve a
total completion score of at least 98% as measured by the voting
performance protocol.*\
\[Slide 30\]*

On the next slide, 3.2.1.1-D says that the usability metrics shall, the
test lab shall report these metrics for usability as measured, and they
will also report the voting sessions time and voter confidence, but we
did not put benchmarks for confidence and time because it really didn't
correlate to the accuracy score, and it really depends on the individual
voters and the length of the ballot, etc. Matt.*\
\[Slide 31\]*

\[MR. MASTERSON:\] Matt Masterson from EAC. Given your testing, do you
have any evidence on the possible costs of this kind of testing?

\[MS. LASKOWSKI:\] Okay. Let's see. The question was- and I've put up
the next slide because there is some relevant information there. What
about the cost?

When we collected this benchmark data, we used 50 test participants but
because we need to get the statistics to work out nicely, we need to
assume a normal distribution to calculate the VII. For that, you need
100 test participants and you also, by the way, get more narrow, smaller
confidence intervals, so you get a better number in these charts when
you use 100.

Given that you have test participant demographics that are relatively
easy to recruit, we are talking probably in the range of 10,000 to
20,000 to recruit for a test for a system. That's a very rough estimate.
It's just, you know, based on getting a test lab and getting, probably
have to recruit 150 or so to get 100 to show up.

Based on that, maybe paying \$50.00 per participant. You can do some
back of the envelope but well within line of current cost of testing. We
thought that was a very reasonable number of test participants. We were
also surprised that with 100, we could get such good repeatable
numbers.\
*\[Slide 32\]*

Britt?

\[QUESTIONER:\] When you said when the test lab shall report. There are
two types of reports. One report is proprietary. The other one is
public. Which report did you say- ?

\[MS. LASKOWSKI:\] I believe the view of the TGDC was that the pass/fail
would be reported on the three effectiveness benchmarks, and the actual
numbers for time and confidence, since they don't pass or fail on those,
would also be reported to the general public.

*\[Slide 33\]*

So, what we have now in these proposed benchmarks, they do obviously
weed out the poorly performing system. The advantage of having this kind
of test method is that as systems improve in the future, it's pretty
easy to raise the thresholds.

There is another point of view that these should be a forward-looking
standard, and we need to be held to a higher standard. Humans in such a
test will always make some errors, so until we get some more data, we
are not clear about what the upper bound is, how good can it get given
any kind of system, given a perfect system since humans will make some
errors.

For now at least, we have some benchmarks that we know will at least
weed out the poor performance systems in the certification process.\
*\[Slide 34\]*\
Alright, so how much flexibility can we allow in the test protocol? We
already talked about geographic. What about less educated population, an
older population? Remember that these benchmark tests are the numbers
themselves.

The thresholds are tightly coupled to the demographics of the test
participants. We are going to be collecting some information to see how
much flexibility we have with those demographics in the next few months.

We also are planning to do some tests with a population of disabled
voters to see what kind of performance we are seeing there as well.\
*\[Slide 35\]*

Jim, question?

\[QUESTIONER:\] Back to the benchmark studies: the devices that you use,
did you look at how widely deployed these devices, the different devices
were? Is there the same sort of threshold in terms of impact right now
on voters?

\[MS. LASKOWSKI:\] These systems are VSS2002 certified and are in use
across the United States. And were used in the 2000 .. 2002 election.

\[QUESTIONER:\] But my question was, you've got clearly different
results along different measurements. Are the machines that were tested
evenly used across the country, did you--

\[MS. LASKOWSKI:\] No, no, that's not the case. Let's see, they are
among the well-used systems in the United States.

\[ALLAN EUSTIS:\] A Federal Register notice went out, and all
manufacturers were encouraged to provide equipment to NIST in these
different kinds of studies. It was voluntary on the part of the vendor
community. We did get representation from the major manufacturers that
are in use since '96.

\[MS. LASKOWSKI:\] Clearly certain systems are in more prevalent use
across the United States than others, but these are in prevalent use.

Any other questions? Wendy.

\[QUESTIONER:\] Okay. Because everybody uses the same script, how do you
factor in for placement within a race, I mean it may be that the third
person on the ballot, if you vote every single time, is maybe an easier
thing to find than the fifth person. So how do you adjust the benchmarks
for that?

\[MS. LASKOWSKI:\] John.

\[MR. CUGINI:\] I'm happy to say that's one-- yeah, the question was,
how do you adjust for the ease of finding candidates that are high up on
the list, low down on the list, or down in the middle. I'm happy to say
that's one of the things that we thought of, and the test script, some
of the instructions say, vote for this guy and he's first, vote for this
guy and he's last, vote for this guy and he's in the middle.

Just to give a flavor and one thing that's maybe worth saying, there are
things like N of M contests, vote for these three guys out of seven,
vote for these two guys even though you are allow to vote for three, do
a write-in vote, referendum vote, so we tried, what David recalled
voting variations, the straight party vote, we tried to cover most of
the normal voting variations. We tried to, you know, not every exotic
possibility, but most of the common things are within the script.

\[QUESTIONER:\] In the design phase, did every race end up one race per
page or you know on some systems, it was multiple page or- ?

\[MR. CUGINI:\] Yeah, the question was, did every race wind up one race
per page? Again we, what did we say, it's a ballot configuration. It's a
logical ballot. Here's a contest with five candidates. Here's a contest
with twenty candidates. We then send that to the vendor and say you lay
this ballot out the best you can. We don't do the ballot layout. Offhand
I don't even remember what some of the ballot layouts looked like. Where
any contest spanned the page or not? Do you remember that? That's not
something we control.

\[QUESTIONER:\] Right, but it would, it could potentially account for
variances. If they provide a bad design, that may be the variance--

\[MS. LASKOWSKI:\] So if those chose a bad design, that means they are
recommending bad designs for their system to the election officials, and
they should fail.

\[QUESTIONER:\] That gets into this type of thing that is just not
factored into the benchmarking.

\[MR. CUGINI:\] No. You are absolutely right that you can't measure the
effect of the system per se. You can't say, you know, x is better than
y. It's always in the context of a given ballot and a given ballot
design. That ballot design obviously has some effect. You can't test
these things in the abstract. You have to have an actual scenario and
see what happens. You are sort of vulnerable to that scenario.\
*\[Slide 36\]*

\[NARRATOR:\] Additional explanatory presentations on the Voluntary
Voting System Guidelines can be accessed from the web site
vote.nist.gov.

[^1]: ^^ Certain commercial entities, equipment, or materials may be
    identified in this presentation in order to describe an experimental
    procedure or concept adequately. Such identification is not intended
    to imply recommendation or endorsement by the National Institute of
    Standards and Technology, nor is it intended to imply that the
    entities, materials, or equipment are necessarily the best available
    for the purpose.