VVSG Tutorial

VVSG Security Requirements Part 3[^1]

*\[Slide 1\]*

\[NARRATOR:\] This is Part 3 of the Security Training Modules for the
next VVSG document currently under public review. This training module
will be presented by Barbara Guttman of NIST's Information Technology
Laboratory. The module is an overview of Chapter 4 of the VVSG, Security
and Audit Architecture, and Chapter 5 of the VVSG, General Security
Requirements.

\[MS. GUTTMAN:\] So, as you gather, the format is first, we are going to
give you a little bit of the background on these terms. Now we are going
to do our part to walk through what's in the VVSG and explain the
security concepts in there.

*\[Slide 2\]*

I'm going to start again with the disclaimers. You may have noted we are
actually taking some advantage of some real-world examples so we will be
using real product names for example purposes.

*\[Slide 3\]*

The agenda, there are two security chapters: Chapter 4, which covers the
audit architecture, and Chapter 5, which are the general security
requirements. *\[Slide 4\]*

The first point I want to make is that with the security parts, they all
need to work together. Nelson explained to you that security has to work
together both from a technical and a procedural and a management point
of view, but also these components need to work together. And that's
actually why we spent so much time on cryptography, is that it ends up
as the basis of so many other controls. So you might find that when you
are looking through the sections, some of them are a bit hard to
understand out of context with the other.

*\[Slide 5\]*

That actually says the same thing that crypto is-- I tend to get ahead
of myself on this one-- crypto is used to support a lot of other things.
But the same thing holds true in the relationship between the other
controls. So you have to kind of look at the big picture, and that's
also part of the defense in depth strategy that it is, from a security
practitioner's point of view, you always presume at least one of these
will be bypassed. So you need to have these multilayers going through
your system.*\
\[Slide 6\]*

And one of the pieces that we actually thought was especially confusing
is that the documentation requirements are elsewhere. They are not in
with the security requirements because, of course, these security
things, they are a little bit complicated to set up. They are pretty
much worthless if the election official cannot set them up correctly.
Let me point out there are two kinds of documentation requirements.
There is one which addresses- and actually I have a separate slide on
this one-

*\[Slide 7\]*

\- Which is in the technical data package, which explains the sort of
overall approach to security so that when a lab looks at the system,
they know what to test, especially when we talk about this open-ended
vulnerability testing. It explains the whole context of the system. So,
it's more from a security architecture point of view.

*\[Slide 8\]*

And then you also have user documentation. Well, how do I actually set
up this feature so that I can use it correctly? These are both very
important pieces of documentation, and you really have to think of these
security pieces together, that you need documentation people can really
use to set this stuff up correctly. *\[Slide 9\]*

That's a little sort of background overview to the whole of security
Chapters 4 and 5. And with that I'm going to get into Chapter 4, which
is security and architecture. I have some advantage here, which is that
you've already gotten several introductions to this, because this was a
particularly difficult issue that the TGDC wrestled with, about how to
address the security and audit architecture. It's divided into two basic
parts. One is sort of an overall approach to audit, and then there are
sort of two basic components: what requirements are put on electronic
records the system keeps and what does it put on this concept that has
been introduced before, the IVVR records, these independent voter
verifiable records.

*\[Slide 10\]*

What does this mean? There was a resolution: the SI resolution that
systems must- changes must be detectible. What does that mean in reality
land? Some of us dwell in reality land, some in fantasy land. Election
officials tend to live in reality land. In practice, this means
auditable. If we are talking about whether it's a detectable, well, how
do you detect things? You detect things through an audit. So SI in
practice means auditable.

*\[Slide 11\]*

Why does the TGDC, why do they want SI? And some of the concepts David
had talked about in particular earlier is that- the first concept is
with the computer: it's actually kind of easy to make the screen say one
thing and doing something else underneath. This is pretty common to do-
that has been done in computer systems. People, they lie, all the time.
The hard part is, election officials will point out, is making changes
that are plausible.

*\[Slide 12\]*

So the first question people ask is, okay, computers often don't show
you exactly what they are doing inside. Wouldn't a test lab catch this?
And I think David explained part of this when he was talking about
testing. Testing is very difficult for software. One of the things he
mentioned is, you know, in voting systems, you have your core logic,
your counting logic, which is actually much more straightforward than
your user interface code. User interface in voting systems, I mean,
that's one of the beauties of having computerized systems. You are
capable of having a very rich user interface capable of supporting all
kinds of different ways of interacting with the voter. Well, you know
what? This adds a lot of lines of code and a lot of complexity. It's a
good thing, but I just wanted to point out, it's very hard to test.*\
\[Slide 13\]*

And I want to give you some examples that I'm hoping will help you
understand the concerns computer scientists have. So I am going to give
you three examples of famous software that wasn't doing what we thought
it was doing.

*\[Slide 14\]*

So the first one, I thought, would be one you are pretty familiar with.
In North Carolina, you had the system. There was a configuration setup.
It looked like it was recording votes. It did everything right. The
voters pressed the submit vote button. It submitted the vote and it went
into the black hole of doom because there was a memory configuration
error, and it just simply wasn't recording those votes. So that's one
example, it was an error- it was not malicious- of the voting system
looking like it was doing one thing when it was actually doing something
else.

*\[Slide 15\]*

And the second example I'm going to give is something that's very old.
It is deep in the heart of what computer scientists learn about systems.
This is an example, it's from the 70s. It was a radiation machine. It
was used to actually deliver radiation to cancer patients. So this is
the kind of machine that people actually spent a lot of time designing,
doing it right, and the machine- it's a very sad story because the
machine killed people. It burned people and it killed them because the
user interface- if you were actually really good at doing the user
interface, if you typed really, really fast, you were a really good
technician and you typed it in, you ended up bypassing, getting into a
place in the system where you should never be, where the machine it
looked like it was delivering dose X, it was really delivering ten times
that dose. And so the machine, everything looked right. The technicians
were doing what they were supposed to be doing. Everything on the screen
looked right, but the machine was actually doing something else. And I
bring this example up because, one, it was a user interface example
which is the most complicated part of voting systems, but also just to
give you a flavor of where computer scientists are kind of coming from
when they talk about their concerns about software quality. It was a
very odd error. It was very hard to recreate, to actually get to this
point because, of course, when people were trying to recreate the error,
they would make sure they typed things very carefully, but you couldn't
make the error happen when you typed it very carefully. You had to type
fast.

*\[Slide 16\]*

So my next example is something most people are familiar with, which is
pfishing attacks. You get a nice e-mail. It says you need to go change
your password. This one purports that it comes from E-Bay. And I think
if I click it, it says that you are going to that Web site, that nice
piece of blue, whereas-

So it says you are going here to E-Bay, but really what's behind there
is this arbitrary address. You are going there, and then they are going
to steal your information and do bad things to you. And that's an
example on the Web, but just an example to show you that there are
plenty of malicious attacks where something looks like it's one thing,
but it's really something else.

*\[Slide 17\]*

So the question is how does the VVSG address the issue of auditability?
The first thing was- one thing that came up a couple of times which is-
this is an equipment spec. It defines capabilities of the equipment.
What election officials do with it is actually their business or the
business of their jurisdiction, their state, whatever they are. This
requires equipment that can be audited. I feel that's something we can't
say too many times.

*\[Slide 18\]*

The VVSG itself lists four different kinds of audits that the equipment
needs to support. And there is a fifth type that the TGDC is considering
putting in their parallel testing which is yet another kind of audit.
However, this one had absolutely zero requirements on the equipment
itself. So basically whatever equipment you have now, you can do
parallel testing with it. They actually did consider putting some
requirements in, but the TGDC considered those too Draconian and dropped
them. And Wendy, you had asked a great question, like observational
testing, like how many actual requirements does this levy on the
equipment? The answer is one. I wrote it down. It's one, it's very
minor, the actual requirement that it levies is there can't be a way for
a poll worker to signal to the machine that it's being tested. So it's a
very minimal set of requirements, because observational testing is
primarily using the system the way it is.

But there are these other three types of audits to be supported. These
should basically be familiar to you: poll book audit, the right number-
I had so many voters come through. Do I have so many votes coming out
the other end? Are my counts correct? Can I compare, now this is a new
one, can you compare your electronic and your IVVR records and get the
same numbers? And also addressing not only just totals but are your
totals in the right bins? So, you could, if someone were maliciously
trying to attack the voting system, they might not change the number of
voters, but move voters from one, basically, from one precinct to
another so they are voting in the wrong contests. So you need to be able
to keep track of rather a lot of things, if you want to do a rather
complete set of audits. So those are required when we get to the
electronic and the paper records- that they provide enough data so that
you could actually do a meaningful audit depending on which attribute
you need to audit.

*\[Slide 19\]*

So as I said, there are two types of-- I mean the essence of auditing is
you compare two things or sometimes more than two to see if they are the
same. I am going to address the electronic records first.

*\[Slide 20\]*

There are some general requirements on electronic records, which is that
they must be in an open format. That is, whatever format it is, it has
to be published so that it is possible for you to read them without just
using equipment that came from that vendor. They must be printable. They
don't have to be printed all the time but they must be capable of being
printed if you need them. And our first use of cryptography, they must
be digitally signed for both integrity and authenticity. That provides
you rather a rich set of security features because when you are
accumulating votes, you can, if you have equipment that will support
this, you can know that the subtotal votes you are bringing in- they can
be signed so you know what exact piece of equipment they came from,
which means you can check, did you get totals from all the machines you
are supposed to get them from or are you missing some? And you can know
that they came from that machine, and they came from that machine
unchanged. So as you are looking at your accumulation function, this is
a really nice thing to be able to do. *\[Slide 21\]*

In general, the electronic records focus on making sure all the relevant
data is in there. One of the things, Dave pointed this out, and I'm
going to reinforce it because it tends to be confusing because there is
a difference between ballot style and ballot configuration: ballot
configuration being just the choices that you have and ballot style
addressing the specific user interface, like whether it was an
alternative language. In this section, it addresses ballot configuration
choices. So when you look through, make sure- well, did we make sure we
got all of the information that needs to be included in each electronic
record? It addresses configuration, not ballot style, which is the more
common term that people are used to using. So it includes things you
might think it would include, like what are the totals- the totals for
each ballot choice that was available to the voters? How many ballots
were read? How many were counted? How many were rejected? How many
overvotes or undervotes or write-ins? It also, Brit, includes your
ballot counter total, which is 435. So I looked that up for you. And it
addresses it and this is another important point that Dave brought up,
remember that a DRE is also- remember we don't distinguish necessarily-
some of these address things that are done by tabulators and EMS, which
would be a tabulator that has tabulators that report to it. So as you
accumulate, you have some additional requirements for keeping all of
these records straight.*\
\[Slide 22\]*

And now we get to the slightly harder concept, because you all are sort
of used to working with these electronic records already, which is the
independent voter verifiable record. So what is an independent voter
verifiable record and where did we come up with such a long name? The
issue is it has to support direct verification by the voter. It must
provide support for hand auditing. We also have various security and
operational properties that I'll walk through. And my first issue is,
well, doesn't this mean paper? This was brought up before and the answer
is no. One of the specific things we looked for was just because paper
is all we know today, why would we limit ourselves? Let's put some
flexibility in here and hopefully having flexibility in our standard
will encourage the marketplace to come up with better solutions. That
was the impetus that drove this. This is not a paper requirement; it is
truly an independent voter verifiable requirement.

*\[Slide 23\]*

So let me go through what some of these requirements are that are on the
IVVR for any system that is an IVVR system. It has to be capable of
direct review and that's by both the voter and the election official. It
must be capable of supporting a hand audit. It must be able to support a
recount. However, you might note that that does not say a hand recount.
There is an assumption that recounts would likely use automated means to
help them along. So the capability is just for a hand audit, but recount
can be automated. But it has to be able to support a recount. It has to
have some durability. It has to provide some tamper evidence. It has to
provide some support for privacy.

*\[Slide 24\]*

It has to have sufficient information on each physical media you have,
so for instance, it must have what ballot configuration this is so that
when you are left at the end of the day with a pile of these things, if
you can imagine you had a pile of these cut sheet things, you actually
know what they go with. So each record must have sufficient information
on it.

No code book is required. That is, the IVVR cannot just say "President
43." It has to say who choice number 43 is. So it must be readable
independently without resorting to a code book, because there are, of
course, some approaches out there that use a code book and these are not
allowed.

Support for multiple physical media: that addresses this cut sheet issue
or like on your paper roll, you may need more than one paper roll during
an election- so what information has to be available on each separate
physical piece of media. Yes.

\[QUESTIONER:\] You said ballot configuration and not just the
selections. So that means that each voter's complete ballot is recorded?

\[MS. GUTTMAN:\] The question is how much information you really have to
put on there in terms of the ballot configuration, not just the
selection. So, President Smith, you don't have to list all the choices
they didn't make, but enough so- because in a race you could have, you
know, the same names show up more than once. So you have to have enough
information so that it stands alone as a record.

\[QUESTIONER:\] With selections for President, it's not necessarily for
each voter's cast ballot. It's not the complete ballot.

\[MS. GUTTMAN:\] It doesn't have to be the complete ballot, the complete
cast vote record. Yes, you also had a question.

\[QUESTIONER:\] My question is, is this standard mandating individual
cut ballots?

\[MS. GUTTMAN:\] No. The question is does it mandate individual cut
sheet ballots? No, it does not. It allows for paper rolls.

\[QUESTIONER:\] But you just said it had to be an individual cut ballot.

\[MS. GUTTMAN:\] I think she was asking if it was. Oh, I shouldn't speak
for you, you speak for yourself.

\[QUESTIONER:\] I was talking about what was recorded.

\[QUESTIONER:\] I'm talking about what you said, our presenter, I
thought I heard you say when you were reading off the--

\[MS. GUTTMAN:\] Oh, on the slide, it says sufficient information
(ballot configuration, not just selections). She was asking about that
specific bullet on the slide. Does that answer your question?

\[QUESTIONER:\] No. He thought you said cut paper ballots. He thought
you were referring to cut paper ballots.

\[MS. GUTTMAN:\] I'm sorry I was unclear. The standard allows for both
paper rolls and cut sheet ballots. In either case, each individual piece
of physical media, which could be the end of a paper roll or the end of
sheet one of a multi-sheet ballot, has to have sufficient information on
it that it can stand on its own. Okay, I'm sorry about that. And if you
have a cut sheet system, the voter has to be able to accept or reject
each sheet individually. The standard also does allow for there to be
nonhuman-readable information on the IVVR, for instance, if you had a
digital signature or a bar code to support automated recounts. It does
allow for that, but whatever information is on there must be in a public
format.

*\[Slide 25\]*

Oh, a question, good.

\[QUESTIONER:\] On the multiple physical media where the IVVR has to
support each individual piece of that media to be accepted or rejected,
one of the issues I was concerned about there is that if you have an
entire ballot that, say, includes five individual sheets, there needs to
be somewhere in the VVSG, in discussion or something that, if you are
going to allow that voter to cast their ballot, you need at least five
accepted to complete your ballot set. If you could accept or reject
individually, you could have a cast vote record that wouldn't be a
complete set of those ballots if one of them would have been rejected.
You want the full set. At some point, you might have individual sheets
in there that are rejected, but by the time they cast their ballot, you
need that full five.

\[MS.GUTTMAN:\] You need the full five or you have a voting session that
is not completed.

\[QUESTIONER:\] That's correct.

\[MS. GUTTMAN:\] Make a note of that, Nelson.

\[MS. DAVIDSON:\] Does that include absentee?

\[MS. GUTTMAN:\] Well, your absentee ballot, I mean, you only have the
one paper ballot that's sent in. So yes, I think it would cover that.
That's a good question, Donetta. I have to think about that one, because
that's not an equipment spec. So I'm not sure it does. However, it would
probably be in keeping with best practice. But it's not specifically an
equipment spec, so it would probably be out of scope. There's another
one for you, Nelson.

*\[Slide 26\]*

So in addition, after we have this section, so any IVVR system must
conform to all of those IVVR requirements. Of course, we happen to know
that there are two IVVR-type systems out there, and there are some
additional requirements specifically on VVPAT systems. *\[Slide 27\]*

Op scan- they don't have as many issues beyond what was already covered
in IVVR. So in VVPAT, Sharon had already addressed VVPAT and
accessibility, and I don't think we need to go into that anymore here.
We have already talked about the observational testing there, and that
there are many operational requirements which I'm going to get into.
Here I did put specifically in that paper rolls are allowed because I
knew that would be an important question. Matt.

\[MR. MASTERSON:\] Matt Masterson from the EAC. I guess my question is
that, okay, we have IVVR and now you are creating requirements for the
two pre-existing IVVR things. But if we are trying to encourage
innovation, how is the EAC going to handle these possible other forms of
IVVR if--

\[MS. GUTTMAN:\] If it complies, if it meets IVVR and if it's neither a
VVPAT nor an op-scan, it meets IVVR. It meets the standard.

\[MR.MASTERSON:\] Then why have these additional--

\[MS. GUTTMAN:\] Because we know that there were some problems with
VVPAT, so we wrote standards to preclude these problems from happening
in the future. And you will see most of them are operational
requirements to make VVPATs better systems. If they meet IVVR, they meet
the basic security requirements for auditability.

*\[Slide 28\]*

Let me show you an example of some of the VVPAT ones. For instance, we
require- like printer-computer interactions, that that should be a
public format between the two. If it's not, you haven't really broken
the whole security thing but from an operational point of view, this is
a huge advantage.

These are ways to make VVPAT systems better, because we have some
experience with VVPAT. You all know what can go wrong with some of them,
and John Wack, who was the original author of these, spent a long time
studying what went wrong for people and what requirements they want so
these things will be better. Wendy, did you have a question on that?

\[QUESTIONER:\] When you said that the printer-computer interaction has
to be public--

\[MS. GUTTMAN:\] The printer port- You have to use a USB or-- that's
what it means. Use a USB; don't use a proprietary printer port that
nobody else can.

So, there's this section first, the components and definitions define
what a VVPAT is. So in the printer, the other things I pointed out
before, is for error handling so that if you have a printer error, you
want to know the status of the vote in question so that election
officials have some clarity of what they are supposed to do with this
error. The protocol of operations that the machine- . And some of these
actually just sort of clarify what's in IVVR but make it specific for
VVPAT. You must display the vote choices so that the voter can compare
them, can accept or reject it, and then also you start to address what
happens if you have multiple rejections: at what point you need a
parameter that election officials can set for after the voter has
rejected the 29th attempt to vote, and this is state-specific. So what
it requires in there is that there be a parameter that is state-setable
for when it is that an election official needs to be called in. Yes.

\[QUESTIONER:\] If the purpose is to audit the system to make sure it is
working properly, it seems to me that if the voter- the first time the
voter says the VVPAT doesn't match their cast ballot, you would need to
take the system out of operation.

\[MS. GUTTMAN:\] Well, no because people make mistakes. When you get
your display screen, now if you have a DRE, you already get a summary
screen that says these are them. People go back because they made
mistakes. So if you are at your display screen, if the voter saw that
the display screen and the printed screen were not the same, they might
want to call the poll worker over and say, something is really wrong
here. But they might just change their mind.

\[QUESTIONER:\] But see, then what seems so confusing is what a VVPAT is
for, because if the VVPAT leads to audit and confirms that there have
been manipulations, you have your summary screen, you want to make
changes. But if you are going to allow voters to continually make
changes off the VVPAT-

\[MS. GUTTMAN:\] This addresses, for systems- I'm sorry, the question
was, you have the summary screen to address when the voter recognizes
that they made an error and want to change their choices versus you have
the VVPAT record which is supposed to be an audit record. However, these
can happen at the same time. So therefore, there is a tunable parameter
which you might want to use in your jurisdiction. But yes, I mean this
would be procedural but I think an election official would be
hard-pressed to explain if they actually saw a difference in the VVPAT
record and the summary screen. You've got a problem, you need to pull
that piece of equipment off and you need to wonder is this an endemic
problem? Is this a systematic problem or is this a lone case of a
malfunctioning machine? But hopefully, this won't happen to you. Does
that answer your question?

So then what you have with the human-readable contents is that the VVPAT
record itself must be-- the print on it must be machine-readable back
in. This supports both accessibility and it supports if you are going to
later need to do a recount and need to do some- quite handy for you if
the human-readable version is also machine-readable. Then there is
another parameter for potentially linking the electronic and the paper
record. There is some very nice kind of auditing you can do if you can
pull specific records. However, this also might violate state law. So
what we have in there is the requirement for the ability to have this
that can be turned on and off locally, depending on what the
jurisdiction needs, because it does give you some auditing advantages;
however, it also has some problems. So we left that as a capability to
use or not use.

Then there is also something to address paper roll privacy, you know,
seals and things like that for the machine. Also if there is an error
that occurs during printing, in the paper roll that what the voter
already has selected should, of course, be inside the machine, inside
the roll, in case of an error. And also there is a requirement for
vendors to provide a spooler if there is a paper roll.

*\[Slide 29\]*

PCOS did not have many additional requirements beyond what's already in
all of the rest of the VVSG. Mostly there was a specific allowance of
nonhuman-readable marks, either record identifiers, batch information,
integrity checkmarks, but these are allowed to be on there.

*\[Slide 30\]*

I'm happy to answer any questions now. Yes.

\[QUESTIONER:\] I would just like to make a comment really. The audits
that you referred to earlier, the poll book audit trail, it is my
understanding those audits are not mandated, they are optional.

\[MS. GUTTMAN:\] They are provided in- or like the poll book audit and
the hand audit and the vote count audit, are they required? The answer
is no. It is required that the machine be able to support such audits,
but you in your state might, you might just call them something
different. You might call them five separate audits versus the three we
called them. Each state is allowed to use its own terms. We had to
provide a context for what the goals of what these records were doing.

\[QUESTIONER:\] As a follow-up, so we may simply choose not just to call
them something different but not to do them at all.

\[MS. GUTTMAN:\] No, no, your state may require you not to do them,
which is the beauty of our federal system. One more question, over here.
Oh, and two more.

\[QUESTIONER:\] The continuous paper roll: our state has concerns about
the secrecy of that. So the guidelines talk about procedure; what kinds
of procedures are envisioned to deal with the secrecy of the ballot on
continuous rolls?

\[MS. GUTTMAN:\] I'd actually just probably have you ask this gentleman
because he has more experience with them. The question was how you
handle secrecy of paper rolls? From an equipment point of view, we put
in a requirement that they have to be able to have seals on them.

\[MALE SPEAKER:\] In our state, it's very simple. All of the voting
machines in a polling place are electronic voting machines. We don't
record the order in which voters come in to vote. We have no way of
knowing. They are allowed to take whatever machine they want when they
go vote and there is no record of what they have seen. There is no way
to go back afterwards and determine this.

\[MS. GUTTMAN:\] Which also is an issue, if you go to electronic poll
books, you are going to want to make sure your electronic poll books are
not also keeping track of the order of votes, which is something to keep
in mind.

\[Mr. EUSTIS:\] Can you summarize his answer?

\[MS. GUTTMAN:\] His answer was that they don't record the order the
voters came in and they let voters select which machines to use. So you
get some randomization. Yes, you would also like to speak to that.

\[QUESTIONER:\] There is a contradiction in the guidelines. You just
read a standard that allows voter paper trails. There is another
standard that says that the vote has got to be cast in such a way that
no person can identify another person's ballot without their assistance.

Those two are conflicting because when you vote on a VVPAT paper roll,
if you record- if I see you vote-- when you leave, I walk over and I
record the serial number of that machine and the public counter on that
machine, those two numbers uniquely identify your ballot.

\[MS. GUTTMAN:\] As you well know, this was something that the TGDC
spent quite some time arguing about. In fact, at one point they had
decided to go the other way, but it being not a perfect world, they
decided that it was better to allow paper rolls, given that election
officials were developing procedures to address them. But yes, it's--

\[QUESTIONER:\] But in another place, you say that the system has got to
prevent that and the system cannot prevent that. As long as those two
numbers uniquely identify a ballot.

\[MR. WACK:\] Well, I know we talked about this earlier-

\[MS. GUTTMAN:\] This is John Wack now.

\[MR. WACK:\] We talked about this earlier. Does the public counter have
to be printed on the paper roll or could it just be displayed on the
screen or something?

\[QUESTIONER:\] It's just displayed on the screen. It's not printed on
the paper roll but if I know, if on this serial-numbered machine, you
were the fifteenth voter, I'd go down that roll to the fifteenth counted
ballot, and that's yours.

\[MR. WACK:\] Oh, okay. But that's unavoidable, I think.

\[QUESTIONER:\] I'm not arguing for or against either one of those two
requirements. I'm saying that they are conflicting, and one or the other
needs to be taken out.

\[MR. WACK:\] Another thing I will add is I can't remember which
particular requirement, but there is a requirement also to ensure that
the paper roll gets stored in a well-strengthened canister, which would
make it more difficult for someone at the polling site to look at the
paper roll.

\[QUESTIONER:\] There's no question about it and he does a really good
job with this, but still the point is that no matter how good a job it
is, somebody has access to those rolls, and whoever that somebody is,
given those two pieces of information, they can reproduce the ballots.

\[MS. GUTTMAN:\] So it definitely places a procedural burden. Did you
have a question too or anybody else?

\[QUESTIONER:\] Well, when I think over some of these things
particularly on the poll book audit, although it is a "should," you are
collecting certain information, and I was trying to go back and find it.
I think over in Chapter 7, sometimes there may be some things-- there
were a couple of things in there that said the activation device shall
not keep certain information, but I think maybe requires the poll book
audits-

\[MS. GUTTMAN:\] Okay. We tried to make sure we were consistent between
Chapters 7, 6, and 4. We might have missed a few spots. So if you send
us an e-mail, that would be good or put it in the public comment record,
because it is certainly possible that we missed a few.

\[MR. WACK:\] Can I just speak to that real quickly? In Chapter 7, what
we basically said was the electronic poll book records cannot be
combined with a DRE's records to clearly show how people voted. I think
they are consistent, I mean what Barbara said earlier was that, when he
said he doesn't keep track of the order in which people use the VVPAT,
she mentioned something about he may need to make sure his electronic
poll book doesn't also keep the same order, which then would keep things
in sync there.

\[QUESTIONER:\] Yes, that's one of the issues I had with Chapter 7
yesterday, was certain things assuming you had electronic poll books,
but it is being called the activation device which gave you something
different in a different system. I think again both of those functions
need to be in together to make sure. In Chapter 7, a lot of the
requirements were for an activation device for being- It was assuming it
was an e-pollbook when it's not always an e-pollbook. I'll have to find
the requirement.*\
\[Slide 31\]*

\[NARRATOR:\] Additional explanatory presentations on the Voluntary
Voting System Guidelines can be accessed from the Web site:
vote.nist.gov.

[^1]: ^\*^ Certain commercial entities, equipment, or materials may be
    identified in this presentation in order to describe an experimental
    procedure or concept adequately. Such identification is not intended
    to imply recommendation or endorsement by the National Institute of
    Standards and Technology, nor is it intended to imply that the
    entities, materials, or equipment are necessarily the best available
    for the purpose.