VVSG Tutorial[^1]

Usability, Accessibility, and Privacy Part 2

*\[Slide 1\]*

\[MODERATOR:\] This is Part 2 of the Next Voluntary Voting System
Guidelines Training Module covering Chapter 3 of the VVSG, Usability,
Accessibility, and Privacy of Voting Systems. Parts 1 through 3 of this
VVSG training module are presented by Dr. Sharon Laskowski of the
National Institute of Standards and Technology's Information Technology
Laboratory. The presentation includes questions and answers from members
of the Election Assistance Commission's Board of Advisors and Standards
Board.\
*\[Slide 2\]*

\[MS. LASKOWSKI:\] Okay, on to 3.2.4-F: Conventional Use of Color. This
is a requirement that just says be cognizant of not doing something that
is nonconventional. Part of our test method work will be to develop a
color guide that will say what good use of color is and also address
best practice for color blindness and contrast on these interfaces.

And 3.2.4-G,,Icons and Language: this requirement says that if there are
any icons, and icons include things like buttons, for example, arrows,
all those are icons, that they must also have some linguistic text
associated with them, because icons are often very easy for people to
confuse the meaning of depending on their experiences.\
*\[Slide 3\]*

3.2.5, Perceptual Issues: this is a set of requirements to minimize
perceptual difficulties for the voter. Some are designed to assist
voters with poor reading vision. So note that this is a usability
section. We've addressed not only accessibility of those with low or no
vision but because we know we are trying to address this universal
usability in a wide range that many people, many of us wear glasses
here, do have some poor reading vision, and this is to also address
that.

And so these include things like screen flicker; resetting of anything
that's been adjusted by the voter at the end of the session so the next
voter sees this; ability to reset default values if someone has reset
something like size and other aspects, or language, they can then say,
oh, no, I made a mistake, they can go back to the default values;
minimum font size for the visual voter; editable ballot device;
available font sizes; and use of san serif font.*\
\[Slide 4\]*

Jim, question?

\[QUESTIONER:\] Yes, for the usability of the poll worker, is there a
requirement that when the device is turned on, I'm talking now about a
DRE, the accessibility features are turned with that same on/off switch?

\[MS. LASKOWSKI:\] We're not talking about the accessible voting
stations here. Jim asks about when a poll worker turns the accessible
voting station on, are the accessible features turned? For an accessible
system, those features are available. I don't think we specifically have
a requirement for accessible voting stations, but there is a set of
default values for that system. And so for the accessible version, there
would be a set of default values available turned on for the accessible
system at that point. So I think logically it follows, since the
accessible system- that those set of defaults for the accessible system
would be available.

Question, Jim?

\[QUESTIONER:\] Just a comment. In the real world, many of the systems
now have, you turn on the machine, then you turn on the accessibility
features, and we constantly run into problems with poll workers not
knowing how to turn on the accessibility features.

\[MS. LASKOWSKI:\] So I would have to think about whether we cover that
or not, Jim. That is an excellent comment to put in the public comments,
so that we make sure that we take a look at it, and we will make a note
of it. Wendy.

\[QUESTIONER:\] Would they be- is this something that the manufacturer
is going to have to create that the voter can adjust independently, or
that when it meets the requirements if they were available for somebody
to adjust?

\[MS. LASKOWSKI:\] So, Wendy asks about who exactly can adjust these
things. It depends on the system for the DRE- well actually, let me
refer to John Cugini for the nuance here.

\[QUESTIONER:\] Okay. Say a voter or a poll worker- if they design it so
only the poll worker could do it, that's acceptable?

\[MR. CUGINI:\] Yes.

\[QUESTIONER:\] Now, if they submit a design that only the poll worker
can change these things, that meets the requirements?

\[MR. CUGINI:\] Oh, I see what you mean. That's ambiguous. Yeah, that's
a good point.

\[MS. LASKOWSKI:\] We'll think about more precise wording for that.
Matt?

\[QUESTIONER:\] Which standard?

\[MS. LASKOWSKI:\] It's 3.2.5-B and C.\
*\[Slide 5\]*\
Okay, continuing with perceptual issues. Once we introduced paper IVVR,
then we were concerned about the legibility of paper ballots and
verification records. And so 3.2.5-G talks about the fact that paper
ballots and verification records, anything on paper, needs to be
legible. That legibility can be met in at least two ways, that is, by
supporting at least two font sizes. This is on not the accessible
system, this is on the system that's not the accessible system. And the
other alternative is with some support using magnification. This
magnification has to be compatible with the configuration itself.\
*\[Slide 6\]*

Wendy has a question.

\[QUESTIONER:\] On those font sizes, I'm not familiar of what millimeter
relates to. Can you put those into point sizes?

\[MS. LASKOWSKI:\] Well, you can't do point size because it depends,
given a point size, it's going to appear a different size on the screen.
You have to just measure it with a ruler.

\[QUESTIONER:\] Right, you know is it 6.4 or- ?

\[MS. LASKOWSKI:\] It depends, because it depends on the screen
displaying it. Three is about ten point, typically. Okay. Someone from
NIST has a metric ruler.

\[MS. GUTTMAN:\] I will measure here with a standard piece of paper. Let
me tell you what-

\[MS. LASKOWSKI:\] And Keith has a question.

\[QUESTIONER:\] The problem is most state laws advise ballot layout by
point size, not by millimeters. So now we've got basically two
standards.

\[MS. LASKOWSKI:\] For paper, yeah, but it also depends on the font
style as well. I should point out the use of millimeters was in VVSG05,
it's not something new. Barbara?

\[MS. GUTTMAN:\] A point of reference, when we measured, if you have
your agenda handy, I think this is about three millimeters like the text
for 'lunch.' I think it is three millimeters.

\[MS. LASKOWSKI:\] That's about ten point.

\[MS. GUTTMAN:\] So, six would be, you measure on the x diagonal to
determine the size of the letters. Is that what you were figuring,
Nelson? So that puts it into context of what size we are talking about.

\[MS. LASKOWSKI:\] We can add that into the discussion of how one can
measure on a paper ballot. Yes, question?

\[QUESTIONER:\] People reading this and wanting to comment on it, they
are going to want some frame of reference. You know I think,
particularly-

\[QUESTIONER:\]. This whole issue with the millimeter business and font
size, this is just déjà vu all over. When we were discussing the 2005
VVSG, we had roughly the same discussion and the same questions and the
same everything. So it sounds like we are not learning.

\[MS. LASKOWSKI:\] Sara points out that we had the same discussion for
VVSG05. Brian Hancock.

\[MR. HANCOCK:\] This goes even beyond that. This was done as an
interpretation to the 2002 standards a long time ago. So this is nothing
new. These millimeters have been in place probably since 2002.

\[QUESTIONER:\] Well, that doesn't mean we can't at least put a frame of
reference so that when people are reviewing these, they kind of have an
idea of what the range is.

\[MS. LASKOWSKI:\] So, Brian Hancock says that was in use in 2002. Wendy
says a frame of reference would be useful. So noted; we can make a note
of that and give an example.*\
\[Slide 7\]*

Continuing with perceptual issues, there are also requirements for
adjusting contrast ratio, good contrast for electronic displays,
accommodation for color blindness, and no reliance solely on color to
distinguish an instruction. Wendy has a question.

\[QUESTIONER:\] I'm sorry-

\[MS. LASKOWSKI:\] Your questions are excellent. No problem.

\[QUESTIONER:\] Alright, we can't always control the lighting in our
polling places. When you look at contrast- I mean trying to set up a
device in a location where it both assures privacy, but at times of the
day, the sun hits it and glare forms and stuff like that. So, I don't
know where you do some of these things, but in the testing of these
devices, you need to develop testing that not only, you know, that looks
at how these things are used in a real location, not only by real
people, but in these locations where we don't have laboratory lighting.

\[MS. LASKOWSKI:\] Yeah, so Wendy points out that lighting conditions
vary a great deal, and she took the words out of my mouth and answered
her own question. In the testing of this, we need to specify certain
lighting conditions under which to test. Some of this is well known by
color experts, contrast and color kind of go together, and we will be
working on that for the test methods. Remember this testing goes on in a
laboratory and like with many of these standards, if you've got sunlight
glaring on a screen, no amount of different kind of contrast is going to
help in that case if you position your voting station in that way.
Similarly, if you put an accessible voting station backed into a corner
so a wheelchair can't get to it, you've undone the accessibility, and so
there is only so far that we can take these standards. But we certainly
will be addressing, you know, some good contrast guidance. *\[Slide 8\]*

3.2.6, Interaction Issues: so these are designed to minimize interaction
difficulties for the voter. Page scrolling is discouraged. There should
be good feedback when a voter selects something. The size of the touch
screen areas, etc., should prevent accidental activation. There should
be a good enough size and separation, and if someone holds down a key
too long, there is no function of repeating keys.\
*\[Slide 9\]*

We've clarified some of the timing issues. This screen has a bunch of
definitions for timing, and I don't want to read them in detail, but the
definitions are here to help you, and they are defined in the prose of
the standard itself, about system response time from the beginning of
when a voter starts pressing a button, how long it takes for a voting
system to complete response, voter inactivity time, how much time should
the system wait before asking the voter if they are still there, and
after activity, how long to wait when an alert is issued that a system
has been abandoned, and it goes into an inactive state. So you should
read those definitions carefully to understand the timing issues under
3.2.6.1.\
*\[Slide 10\]\
* Question, Brit?

\[QUESTIONER:\] I want to go back to 3.2.6-A, no scrolling. This says
the voting system shall not require page scrolling by the voters. On a
DRE-type machine, the summary page may not fit on a single screen.

\[MS. LASKOWSKI:\] So you can do a 'next' to the next page with proper
indication. It has to be paged according to this.

Oh, I'm sorry Brit asks, well, what about if a page is too long to fit
on one page and now we have prohibited scrolling. Then you have got to
have some sort of next button and some accommodation to put up the next
screen.

\[QUESTIONER:\] You have declassified practically every CRT-type voting
system out there.

\[MS. LASKOWSKI:\] Okay. I believe this was in VVSG05, and we didn't get
comments to that effect. This went from a 'should' to a 'shall' and-
okay.

\[MR. CUGINI:\] We get into this in the discussion. I don't know if
people have read the discussion there.

\[MS. LASKOWSKI:\] We encourage people to read the discussion for the
scrolling. People mis-scrolling cause a lot of errors. Wendy has a
question.

\[QUESTIONER:\] So, if you have a multiple-page issue, I'm thinking of,
we have these long issues. In general, the paging takes you to the next
issue versus within that issue. I guess I'm- you know, there is this
scrolling between issues- if you are within an issue and the next is for
the next issue coming up, if you do one thing per page. I guess what I'm
saying is if you want one issue, per page, we don't want any scrolling,
and we want them to hit the next button to go to the next page.

\[MS. LASKOWSKI:\] Well, you can have different kinds of next buttons.
There are certainly ways to design this effectively. John Cugini has a
comment.

\[MR. CUGINI:\] Yes, just to go back to the page thing. That is a
'should,' because we are very aware you might have a contest for
governor with eight hundred candidates, you might have a long issue,
that's why it's a 'should.' And the discussion says, we realize, the
idea is other things being equal, if you have a race for governor with
five candidates and the next race has five candidates, it's probably not
a great idea to put two candidates on one page and three on the next.
That's the purpose of that one page.

\[QUESTIONER:\] That's not what I'm looking at.

\[MR. CUGINI:\] I know, but Wendy is sort of relating the fit everything
on one page to the no scrolling. So what the standards are saying, for
better or worse, that in no case should you make the logical page bigger
than the physical page. You fit everything on one page by having a
hidden part of the page down at the bottom and use a scroll bar. For
better or worse, the standards say, no, you can't do that. That's what
the no scrolling says. So it follows that there is a limited amount of
physical space on the page whether it's electronic or paper. The
'should' requirements says, try hard not to split contests over pages,
but that's a 'should' because, yes, if you have a very long referendum
or a very long contest, you will have to split a contest over pages.
Without defending those design choices, I mean, that's what the standard
says. I think it's consistent. Whether those are good design guidelines,
you know, you can debate that. I don't think there is a conflict between
those two.*\
\[Slide 11\]*

\[MS. LASKOWSKI:\] Okay. So I encourage you to read the definitions for
timing so that the timing issues in 3.2.6.1 will be understandable. Note
that the response time for all operations, system response time is less
than ten seconds but only for the visual VEBD, not the audio, because
obviously if you are reading a long contest or the like, that's going to
take longer, so you can't have a limit on that. Timing issues are pretty
straightforward, but there is a question.

The question is on 3.2.6.1-E, where do the two-to-five minutes come
from? We were looking at other sort of kiosks and what they do. We also
know that some states have limits. For example, Keith noted a
five-minute limit, and so we tried to put in a relatively flexible range
there. Note that that is just before an alert to tell the voter that
they need to do something. It's not that the system is going to shut
down.\
*\[Slide 12\]*

3.2.7 is alternative languages. These are when a voting system is going
to be certified for languages. These languages are declared by the
manufacturer. And so obviously election officials, depending on their
state demographics, must ensure that the voting system they deploy has
been certified for the languages that they need. We have requirements
that say that you've got to have complete information in the alternative
language and the voter control must be in the alternative language. The
auditability of records should be in English. We've got one of these
usability testing for the technical data package on alternative
languages by the manufacturer.\
*\[Slide 13\]*

Question?

\[QUESTIONER:\] Yes, I'm curious for the alternative languages. We have
American Indian, Alaskan Intuit languages that are unwritten. In the
2005 version of the VVSG, we had some language in there that allowed
unwritten languages with the VVPAT, the IVVR to be in an audio playback.
I don't see that here.

\[MS. LASKOWSKI:\] You wouldn't see that in the usability section. You
will see that in the accessibility section.

\[QUESTIONER:\] Accessibility, but in this section under 3.2.7-A2, it
says "all supporting information must be available in the alternative
language." So, let's see in 3.3.1, your talk under the accessibility,
it's allowing the audio read back but in there, it doesn't really
address alternative unwritten languages, and there is nothing in the
IVVR section in Chapter 4 that would address unwritten languages. I
think that it is important to have it included somewhere in here that
allows that VVPAT to be presented or the IVVR to be presented in an
audio playback for those unwritten languages.

\[MS. LASKOWSKI:\] Shelly points out that we don't specifically say for
unwritten languages that there is audio read back from the VVPAT. I
think it's implicit, but it's important enough probably to mention
specifically in a discussion.\
*\[Slide 14\]*

So under usability for poll workers, we have some general capabilities
that all systems must support for usability for setup, operation, and
shutdown. We expect that the system messages for poll workers are clear.
In other words, the plain language guidance applies to these messages as
well. We want the manufacturers to report on some usability testing for
poll workers. We will be working on some test methods for ensuring that
the documentation is usable for poll workers, and there are also some
requirements for making sure that poll workers verify that the system is
operating correctly, etc.\
*\[Slide 15\]*\
Wendy has a question.

\[QUESTIONER:\] On the usability testing, I really would appreciate if
you did not make them representative of the general population. I don't
think there is anybody-

\[MS. LASKOWSKI:\] No, actually, because we are working some of these
test methods for the usability of the documentation. We just had this
discussion of what are the poll worker demographics. So, yes, we
understand that, and that's under discussion now. Obviously we need some
older poll workers, for example, you know.

\[QUESTIONER:\] The other point is that the poll worker testing be done
with representatives of the general population when in fact poll workers
are not representative of the general population. So, I don't want the
requirements to be that.

\[MS. LASKOWSKI:\] Yes. We are being pretty flexible for what the
manufacturer tests and pretty loose in that they just report it. But
obviously if they are reporting - if you look at the CIF, there are some
specifications of how you choose, you know, your set, and they will have
to put that into the report. So that should embarrass them into doing it
properly. But we also will have some test methods with appropriate
demographics for poll workers in that the test lab will be doing to
certify the usability of the documentation and the ease of normal
operation of the system.\
*\[Slide 16\]*

Safety, we just defer to UL 60950-1 for equipment safety. That pretty
much covers it.\
*\[Slide 17\]*

So now we are working on Section 3.3, the accessibility requirements.*\
\[Slide 18\]*\
There are some general requirements relevant to a wide variety of
disabilities. There should be accessibility throughout the voting
session. We are going to be working on a test method for ensuring that,
because sometimes you can do little accessibility solutions, they don't
all work together. That includes good documentation of any accessibility
procedures that are needed for a particular system. Alternative formats:
you should have complete information. No dependence on personal
assistive technology, a lot of this has been in VVSG05, secondary means
of voter identification. New is the accessibility of paper-based vote
verification. We are emphasizing that because of the software
independence issue, and audio read back for paper-based vote
verification. We have made that explicit, because of the software
independence issue.\
*\[Slide 19\]*

And so, expounding on that a little bit more, we need to ensure that all
voters have a similar opportunity to verify their vote. So how does one
make the paper-based vote verification accessible? The requirement that
we have, 3.3.1-E, is if the accessible voting station generates a paper
record or some other durable human-readable record for the purposes of
showing voters, allowing voters to verify their votes, the system shall
provide a means to ensure that the verification record is accessible to
all voters with disabilities.\
*\[Slide 20\]*

Okay, so this means that we need to allow voters with visual
disabilities to verify, of course the rub is even if indirectly, the
content of the record, and the verification depends on the integrity of
the mechanism that reads the record to the voter. This means that the
audio has to be generated by the paper record, directly from it, just as
if you were reading from the paper record, and not depend on any
electronic or internal record of the ballot. That leads us to 3.3.1-E1,
audio read back for paper-based vote verification. Okay, the system
shall provide a mechanism that can read the record and generate an audio
representation of its content.\
*\[Slide 21\]*

So I now refer to requirement 4.2.4-A, which has to do with the IVVR
because it ties in with the security of that audio read back, right? If
you have an IVVR vote capture device that's using assistive technology,
how do you know that that read back is indeed reading, that you have
some assurance that it's reading back correctly? 4.2.4-A says that the
IVVR vote capture devices that support assistive technology shall
support observational testing. So I've put, just for helping you
understand, I've put in some definitions to understand what that
requirement means. An IVVR vote capture device is, just to reiterate, a
vote capture device that achieves software independence through
independent voter verifiable records, which means paper.

A vote capture device is any device that's used directly by a voter to
vote a ballot. Software independence, and you heard this yesterday, is
the quality of a voting system or voting device such that a previously
undetected change or a fault in the software cannot cause an
undetectable change or error in an election outcome. Observational test:
so this is new information in this tutorial. An observational test is an
operational test conducted on voting devices during an election by real
voters to establish confidence that the VVPR is produced correctly when
assistive technology is used. Okay, so the idea is that if you've got an
accessible voting station that is going to do read back in some way,
that device that's doing the read back, the assistive technology, is
also used by some sighted voters who both look at the paper and listen
to the read back, and they do it in such a way that the real voter is
voting. You just encourage them to use it, but you tell them, listen and
make sure that it's reading back correctly off the paper. So that gives
you very high assurance, because you can't identify that that voter is
sighted or not, that that read back is working correctly. That was our
compromise in terms of only being able to indirectly read back off the
paper record for the blind but to ensure that that read back device is
giving them correct information.\
*\[Slide 22\]*

Wendy has a question.

\[QUESTIONER:\] This is kind of one of those things that falls into-
okay now we are making, we are developing a standard that the election
official has

to--

\[MS. LASKOWSKI:\] It's a procedure. So, Wendy says that we are
developing something an election official has to take care of. It is a
polling place procedure.

\[QUESTIONER:\] So, I have to get some voters to do this. How do I do
that?

\[MS. LASKOWSKI:\] You can ask some poll workers when they cast their
vote to do it, for example.

\[QUESTIONER:\] I have places that are all student precincts, and all
the precinct and all the poll workers come from somewhere else.

\[MS. LASKOWSKI:\] That's a good question. I don't know the answer.

\[QUESTIONER:\] The requirements you are putting in, I'm not sure are
voting system standards. They are management issues, the testing-

\[MS. LASKOWSKI:\] Yeah, well, with respect to a lot of security issues,
they work in conjunction with good procedure. But, yes, this is an
issue.

\[QUESTIONER:\] I don't understand, okay, this is the read back. Surely
in the logic and accuracy testing, you test the reading of the ballot
and the read back of the ballot, so why is this additional step
necessary when you don't--

\[MS. LASKOWSKI:\] Because there's a software independence requirement;
so because it's software, you don't have good assurance.

\[QUESTIONER:\] A logic and accuracy test is done on the election setup.
It's the exact same setup as you do during the election.

\[MS. LASKOWSKI:\] Barbara Guttman has a comment.

\[MS. GUTTMAN:\] So the requirement is on the equipment that it be
capable of supporting observational testing. Obviously what kind of
testing a jurisdiction does is up to that jurisdiction to decide which
testing they do, right, if you want to. For observational testing to be
an extremely effective means of ensuring that the system is not
malfunctioning for vision-impaired voters, you need a remarkably small
number of people to do observational testing on the order of one per
polling place. So it's a very small number to establish that the read
back is operating correctly. But it is very important to establish that
the read back for the vision-impaired people is true voter verification.

\[QUESTIONER:\] Well, it's just as important that when they read the
first time, is accurate. I guess I don't understand the difference
between the operational testing or the fact that is it reading the full
ballot for the first time. I don't understand, you know, are they
getting every issue of that style, you don't' have an operational test
for that piece of it and -- do you see what I am saying. Do you test
that pre-election, is that what you are saying?

\[MR. EUSTIS:\] Matt, and then Jim.

\[MR. MASTERSON:\] My point is very similar to Barbara's but I guess
more sympathetic to what you are saying. In the discussions with the
TGDC, your exact point was brought up, that this is a procedural thing,
and that it shouldn't be in the VVSG. And so Barbara is exactly right.
The requirement is that the equipment supports this observational
testing if a jurisdiction should choose to use it. It is still up to the
jurisdictions whether they want to do observational testing or not. It
is only a requirement that if they choose to do it, that the equipment
support it. So, you can do your testing however you want to do it, but
the equipment will support this form of observational testing if you
want to do it.

\[QUESTIONER:\] I guess I don't understand it then. If all it is is a
real voter going to vote, doing it, having it read back to him and them
coming and saying yes, it worked. The machine has to do that anyway. Any
machine can do that. It's not a standard.

\[MS. GUTTMAN:\] Part of the structure for how the audit works is, if
you are going to compare your electronic cast vote record and your paper
record at some time, you need to verify that the contents of the paper
record are good. And this, for sighted voters, they can do that
directly, but non-sighted voters cannot do this directly, so you need
this other method- the read back. So you want to make sure that the read
back on actual Election Day conditions, is reading back correctly for
sight-impaired voters. So that's establishing that your paper records
are a high-quality record.

\[QUESTIONER:\] It's the voter that is supposed to be doing the
verification, right? So are you saying that the poll worker, when a
blind voter comes in, has to get a citizen to verify that the system is
working so the blind voter has the verification?

\[MS. LASKOWSKI:\] No, not at all, if I understand your comment. We use
some sighted voters just to make sure the equipment is operating
properly on their vote, so that when the blind voter votes independently
and verifies independently through the read back, they have a high
assurance that they are hearing it back correctly.

\[QUESTIONER:\] Okay, a distinction without a difference from our point
of view. What your rule does is it says we don't get to verify. This
burden on election officials is a smoke-and-mirror dance to get around
the fact that you are saying that we don't get to verify.

\[MS. LASKOWSKI:\] You get to verify through assistive technology just
like with the voting.

\[QUESTIONER:\] I don't verify off the paper.

\[MS. GUTTMAN:\] Yes, you do.

\[QUESTIONER:\] No, I don't.

\[MS. LASKOWSKI:\] Through assistive technology.

\[MR. EUSTIS:\] It is indirect verification.

\[MS. LASKOWSKI:\] It is indirect verification off the paper, not off
the electronic record.

\[QUESTIONER:\] You mean by an audio method.

\[MS. LASKOWSKI:\] By an audio method.

\[QUESTIONER:\] Now you have thoroughly confused us.

\[MS. LASKOWSKI:\] Let me try once more. So the process is, suppose you
are using a VVPAT system. The sighted voter can check the VVPAT record,
which is paper, to make sure things were recorded as they intended. If
you are blind, obviously you can't read directly off the paper. You need
assistive technology. So you need some read back mechanism that, say
optical scan, for example, that looks at that piece of paper, the VVPAT,
and reads it back to you. Not from the electronic record stored in the
DRE but off that paper, but since it's paper and that's the only SI
method we have at present, it still has to be through the assistive
technology.

\[QUESTIONER:\] I understand this isn't the place for argument; I just
want to point out that the law says one thing and you cannot create a
regulation, whether you call it superman or cha-cha-cha, that limits the
law.

\[MS. LASKOWSKI:\] I think Keith had a question next.

\[QUESTIONER:\] So, my operational security method is that I had some
guy do it at 9:00 this morning?

\[MS. LASKOWSKI:\] So Keith asked the question, you are suggesting
because my operational method for testing the read back mechanism is
that I had some guy do it at 9:00 in the morning. A minimal number of
people just making sure it is operating properly is enough to say that
that system is not, because it can't tell whether that voter is blind or
not, is enough to give you very high assurance that it's working
properly.

\[MS. GUTTMAN:\] After you have done proper logic and accuracy testing,
this is not the only piece; this is just an additional piece to provide
extra assurance.

\[QUESTIONER:\] Look, am I supposed to know this person's name, have him
sign an affidavit?

\[MS. LASKOWSKI:\] No.

\[QUESTIONER:\] I had some guy check this morning so it's okay. Really?
Tight as a drum?

\[MS. LASKOWSKI:\] Wendy has a question.

\[QUESTIONER:\] There is nothing that the equipment has to do different
during this observational test than it does for any other voter,
correct?

\[MS. LASKOWSKI:\] Correct. It does nothing different as it does for any
other voter.

\[QUESTIONER:\] I don't understand why you are saying the equipment has
to be able to do it, but not what it is if the equipment has to be able
to let a person vote and read it back. So I think that's my thing, I'm
trying to figure out why do we have this equipment standard when there
is nothing, all it is is- can somebody do it?

\[MS. LASKOWSKI:\] Because there is a requirement that says you must
have software independence. The read back mechanism is not
software-independent, so this was the work-around.\
*\[Slide 23\]*

So what about VVPAT? Okay, VVPAT is one implementation of IVVR, and if
you look at Section 4.4.2, I'm not going to go over it here, it contains
requirements specific to VVPAT. All relevant usability and accessibility
requirements apply to a VVPAT implementation.\
*\[Slide 24\]*

3.3.2, Low Vision: note that for what we are calling poor vision, there
is in the usability Section 3.2.5, also apply in terms of adjusting
contrast, addressing color blindness, etc. In addition, in the
accessibility section, we have adjustability of the color saturation,
distinctive buttons and controls, and also synchronized audio and video,
so that someone with low vision can sort of see but also hear, and
that's often for someone who has low vision, that's useful. Synchronized
audio/video also addresses help for people with certain kinds of
cognitive or disabilities or things like dyslexia.*\
\[Slide 25\]*

3.3.3 are requirements to deal with voters who are blind. Again, the
manufacturer needs to supply some indication of some usability testing
in their software engineering process. There are requirements for the
audio tactical interface that say you have equivalent functionality,
that the voter can ask for things to be repeated, to pause, to resume,
being able to transition between next and previous, skip over referenda
wording, for example.\
*\[Slide 26\]\
*Any questions? Matt.

\[MR. MASTERSON:\] I think I already know the answer to this, it sort of
goes back to the previous debate. But I assume that there are no current
voting systems that offer that kind of read back functionality- because
these standards are designed for-

\[MS. LASKOWSKI:\] These were in VVSG05. Oh, you are going back to the
read back. Well, you could, for example, if you've got optical scan, if
you've got an Automark, you could set aside a separate Automark and run
it through. I think it's Automark or one of the electronic ballot
markers, I forget which one. You can put it back through another machine
to read back directly off the paper. That's one example. I don't think
there's an implementation reading off the VVPAT currently that I've
seen.*\
\[Slide 27\]*

Continuing on with blindness: there are other audio features and
characteristics supported like T-coil couplings; initial volume
settings; a range of volume, range of frequency; audio must be
intelligible; you can control the speed, etc., all the typical things
you would expect with a good audio interface.\
*\[Slide 28\]*

Jim.

\[QUESTIONER:\] Yes, I have a question. Later on, I think it's Section
3.3.5, where you are dealing with mobility, you cite the ADA and the
ADAG, but you do not cite that under this section that deals with
blindness, and I am wondering if there was a reason for that omission.

\[MS. LASKOWSKI:\] If we've omitted a reference, we certainly would
appreciate the comment to include it. I don't think there was a
particular reason. We did work with the Access Board, and they didn't
note that we had omitted it. So we could have easily overlooked that. If
you would in the public comment, put in that citation, we would
appreciate it.

*\[Slide 29\]*

\[NARRATOR:\] Additional explanatory presentations on the Voluntary
Voting System Guidelines can be accessed from the Web site:
vote.nist.gov.

[^1]: ^^ Certain commercial entities, equipment, or materials may be
    identified in this presentation in order to describe an experimental
    procedure or concept adequately. Such identification is not intended
    to imply recommendation or endorsement by the National Institute of
    Standards and Technology, nor is it intended to imply that the
    entities, materials, or equipment are necessarily the best available
    for the purpose.