Un-official Closed Caption Transcript:

3/19/2006 Technical Guidelines Development Meeting

Please stand by for captions.\
\
If we could take our seats. We would like to start as soon as possible.
Thank you.

Dr. Semerjian: Good afternoon. I call this afternoon session of the TGDC
back to order. I would request that Mr. Greene call the roll and
determine if a quorum is present.

The afternoon roll call. Williams.\
Here.\
Williams is here.

Berger.\
Here.\
Karmol

Here

Craft.

Craft is here.

Gale. Gale is here.

Elekes.\
Here.\
Gannon is here.

Harding?\
Harding is on his way.\
Miller? Miller is here.

Purcell? Here

Quesenbery is here.

Rivest is here.

Schutzer is here. Turner Buie is present by telephone.

And Smerjian. Here.

We have 14 in attendance. That\'s more than enough for a quorum.

Chair: Thank you Mr. Greene. At this time I'll call on Dr. David Flater
of the NIST Information Technology Laboratory for the second half of the
Core Requirements and Testing subcommittee Preliminary Report. David.

Dr. Flater: Thank you. Normally one would receive lots of style points
deducted for having a presentation that\'s too short. I understand today
a general amnesty has been announced. So I shall omit needless words. If
the Committee sees I'm passing an issue they would like to discuss, slow
me down. My presentation covers the material that appears in the binders
as two documents. One contains requirements on casting, counting and
reporting, plus a section on closing polls. Another is the conformance
clause. I'm going to focus on the requirements for casting, counting and
reporting with respect to closing polls, I'm just going to talk about
early voting and how that relates. With respect to the conformance
clause I \'m going to talk about the classification mechanism introduced
into the standard. First the casting section. Casting section is broke n
down into six subsections. First, ballot activation, which previously
has been only appearing on d r e systems. This is the behavior in which
the system has all of the ballot formats available to it. And it
delivers to the voter the ballot format appropriate to that voter.
General voting functionality is just the interactions with the voter,
which has a greet deal \-- great deal of over lap with the human privacy
area. Voter variations is one of the areas where the requirements have
been substantially expanded versus the previous standards. Earlier there
were comments about how we should look at requirements and testing for
behaviors that are of interest to individual states that do not appear
in the old standards. Well, I'm happy to inform you that NIST
anticipated this need. In the documents in front of you, they included
voting variation requirements that cover such things as cumulative
voting, n of m voting, and straight party voting, which previously did
not get a lot of language in the old standard. It said vendors shall
describe how the system might or my not support these. There are now
requirements if you claim this functionality, this is what the system
must do. Recording votes deals with when the voter hits the cast ballot
button and other events surrounding that. Redundant records has to do
with the historical requirement that the DRE shall retain more than one
copy of the cast vote record. And respecting limits has to do with the
fact that a tab later should stop before there is the threat of
overflowing a counter. Now, these names for these different subsections
are, of course, tentative. They will be replaced with whatever words are
most effective in communicating the intent. One of the major adjustments
made in the casting requirements is to expand them to include a class of
voting devices that we have called EBMs, or electronically assisted
ballot markers. These are devices that provide some sort of electronic
interface to the voter and at the end of the interaction with the voter
produces a voted ballot on paper. Earlier there was some discussion
about expanding the standards to cover vote by phone. And I \'m again
proud to announce that NIST has anticipated this need. Vote by phone
system I believe satisfies the requirements to be an electronically
assisted ballot marker. It\'s providing an electronic interface to the
voter which is similar to the audio interface on the DRE. At the end of
the process you get a paper ballot. Now, we are aware of two different
variations of electronically assisted ballot markers. With one of these
variants, the poll worker gives to the voter a ballot that\'s pre
printed with the appropriate ballot format already. And what the e b m
does is assist the voter in filling in the ovals as it were. The EBM
does not have the capability to serve the voter with other ballot
formats. But there is a sub class of systems which we\'ve called
electronic ballot printers, which in fact do have the ballot formats
available to them. And they print an entire ballot. You do not have to
supply this equipment with the pre- printed ballot that has a ballot
format chosen. The interaction is much like the DRE. The poll worker
assigns a ballot format, but the actual production of a ballot of that
particular format is done by the equipment. EBMs therefore can support
ballot activation. And the requirements that previously applied \-- or
EBPs (Electronic ballot printers) rather \-- can support ballot
activation. And the requirements that previously applied only to DREs
with respect to ballot activation have been adjusted to include EBPs in
their scope. All EBMs on the other hand support interaction with the
voter that the d r e support. So those DRE requirements have been
adjusted to include all EBMs in their scope.

John Gale: I had a question. Was this at the direction of the TGDC, this
merger of the EBMs with the DRE standards?

Flater: I wouldn\'t call it a merger. There was discussion at some point
about we have to extend the scope of the existing standards to cover
these new kinds of technologies and systems that are appearing that we
don\'t know how to apply the standards to. I believe that the discussion
was at that general level. EBMs, EBPs, including vote by phone, fall
into that category. This really, although it\'s significant in terms of
it\'s impact, in terms of meddling with the standard, it wasn\'t that
significant. It\'s simply a matter of observing that these requirements
that \-- meaning all systems that provide an electronic interface to the
voter which supports a certain kind of interaction \-- produces a piece
of paper that\'s not a ballot. The electronically assisted ballot marker
produces a paper ballot. It\'s the genuine document that registers the
vote, that\'s cast by the voter. Where the verifiable paper audit trail
document is a document. It is not a ballot. The important thing I would
think for the system is number one, the way it handles that condition is
entirely predictable and entirely determinable. So there is no question
as to whether that ballot is going to be saved or whether it will be
cancelled in the event of catastrophic failure. It needs to be a known
quantity. It needed to be tested. And then the states will have to look
at the way that\'s handled or specified it the way that\'s handled when
they buy their systems.

Committee question: What happens when the equipment becomes unusable?

Flater: I don\'t know of any states that have a specific statute as to
when the equipment becomes unusable. But most have a statute as to
abandoned ballots. In the condition where this ballot was left and not
recovered until several hours later, it would fit the description of an
abandoned ballot t.would be handled consistent with however the state
handles. Of an unfinished ballot. We could test it that way too.

I have a question for you or for \-- this is something we have to deal
with in the feature. I suggest this is question for some future time.
The question I would have \-- I'm aware there are jurisdictions in which
if a ballot is abandoned that they bring in two witnesses and they page
through the ballot and they cast that half finished ballot. My question
would be how do you \-- I mean, if the measure of voter intent is just
to cast the ballot,, I don\'t understand the integrity of the half
ballot that the voter did not cast.

Mr. Williams: First off. You cannot determine the voter\'s intent.
Period. Can\'t be done. You can come to objective conclusions as to
whether the voter made a clear indication of his choices. And we have
beat that language to death in here before. The State of Florida beat it
to death in court several times. The state election code will probably
address how to handle that. If not, then their court cases hopefully
will. If not, they have opportunities for doing new stuff in the future.
How you handle \-- I mean, obviously the voter \-- we know the voter
left the dead machine with a half completed ballot on it. Did he leave
with the belief that the vote was counted or not? Did he leave with a
mark he needed yet to vote or not? What are the rights and
responsibilities of the parties involved in that? That\'s an issue for
the state election code. That\'s not something we should be dictating in
our federal standard. Unless congress wants to pass a law that controls
elections.

Another case is where one machine dies instead of all of them. In which
case, it will be reasonable to offer that voter the opportunity to cast
the ballot on other machines. You don\'t retain that vote.

Flater: In the interest of time I will move on because I think this is a
more detailed discussion than we have here. It might be a topic for a
subcommittee meeting.

Finally, in the old standard there was a prohibition about overflowing
counters and tabulators. Unfortunate ly it was buried in the testing
standard part of the document. I've moved that into the main part of the
requirements, clarified it and generalized it. To say for x from, f
district r precinct.....Or you can use the merge ballot approach. In the
merge ballot approach, all of the parts are included in a single ballot
format. The voter is instructed to only vote in one set of applicable
contests. The draft standard does not prohibit this. But it also does
not require the extra logic that the system would have to include to
correctly process this kind of a ballot. So this would remain an
extension to the standard just as it was in previous versions of the
spec. If the vendor does this they shall describe how they do this. But
you can support open primaries without doing this.

Committee Question: Do we have an example of a state that uses a merge
ballot process? Or is this \-- or is this the emerging \--

Flater: I cannot cite the example. I know there is at least one brand of
voting equipment that supports this. Now, I guess if the equipment
supports it, I guess there\'s a reason.

Okay.

It\'s not necessarily a good assumption. (laughter).

Flater: It\'s just as well that I didn\'t write this into the draft. \--
the question, do you want to recall this person yes or no. Assuming that
this person is recalled, who should the replacement be. The simple
answer is to have these be two independent questions. On the other hand,
you can have this \-- there are some jurisdictions in which the voter is
not entitled to vote on the replacement unless the voter \-- another
jurisdiction the voter is not entitled to vote on who the replacement is
unless the person voted in the affirmative for the recall request. As
requested in the merged ballot approach, we have complexity in the
process here. It\'s a single ballot format. Sometimes the voter is
entitled to vote in a certain contest and sometimes not. My guess is the
additional complexity is specifying this in the draft is probably not
worth the cost. If there\'s no comment on that. I will move it right
along.

Closing polls. I'm just going to mention here some issues having to do
with early voteing that were discussed. One of the things that was
brought up with the previous standards is how do we support early
voting? \-- \-- (inaudible) Therefore, the closing poll section does not
deal with early voting. Along the way of discovering this, it came out
that some of the old requirements were perhaps too loose. They said the
system shall permit unauthorized \-- that shall prohibit unauthorized
opening of the polls and prohibit unauthorized early reporting. All the
feedback we received said this should never be authorized. So those
requirements have been adjusted simply to say the system shall prohibit
the reopening of the polls or early reporting . A lot of the rest of
what came out with respect to early voting was procedural and dealt with
ballot accounting at the end of the day. You should make note of the
ballot counter and make sure the next day it doesn\'t change. \--
recommendations will be disposed in that manner. Counting section.
Counting section has five sub divisions. One again it is about voting
variations from the tabulation perspective.

Section about ballot separation and rejection. Separation has to do with
the requirements in some systems to \-- for example, separate ballots
that contain write ins with the assumption that they will require manual
counting layer. Rejection is this action that the system performs when
it\'s presented with a ballot that has some problems such as over
voting, it kicks the ballot back out to the voter in a precincts count
er environment, explains what\'s wrong, gives them a chance to fix it or
submit as is.

Paper jams-self -explanatory.

Clarification is the system shall make it blatantly clear when a ballot
jams in the reader whether or not that was ballot was counted so that
the election judge who is clearing that jam knows what to do with that
ballot. Accuracy builds on the general counting accuracy requirement
going into detail about some optical scan issues with manually marked
paper ballots that I'm going to talk about some more.

Finally there are some requirements on consolidation, chiefly about the
time requirement for DREs. With respect to ballot separation and
rejection. The requirement to separate write ins maybe showing its aged
a little bit. If you are using an electronically assisted ballot marker,
it is feasible these could decode in form the write ins name that the
candidate is providing. In such systems it\'s unnecessary to separate
write in ballots that can be tabulated with the rest. Initial
adjustments can be made and will be fine tuned later.

Committeee member: Okay. I don\'t think an electronic ballot marker
would ever be separating out write ins. After the ballot is marked it's
taken to a tabulator which deals with that issue, however it deals with
it.

Flater: Agreed. This is about tabulators. The requirements in the
counting section are requirements on tabulators. It says EBMs may encode
write ins in machine readable form.\
The intent was that the requirement on the tabulator to separate ballots
containing write ins may need to be adjusted in the context of a system
that includes EBMs because it may no longer be necessary to separate
those ballots.

Let\'s go ahead with the presentation and then let\'s come back.

Okay. Rejection behaviors. In the 2002 spec there was one set of
language about rejecting blank ballots and a slightly different set of
language about rejecting ballots containing over votes and under votes.
With the language that the election official shall be able to turn on or
of the rejection behaviors. In the 2005 VVSG a set of requirements is
duplicated in several places saying that the system shall reject ballots
containing over votes and under votes and the language about election
officials being able to turn it off disappeared. However, there is one
place remaining in the VVSG where it still has the old formula, that the
election officials can turn it on or off. In the draft, I brought these
requirements back together in one place. The requirement to be able to
turn on or off these behaviors is retained and also clarified to address
what sounds to be the most common practice, which is to reject ballots
containing over votes and blank ballots but not to reject ballots
containing under votes. And the reason there is that 80 percent of
voters do not vote in the dog catcher race and we do not want to reject
all those ballots because it will cause a long line in the precinct.

Additionally, there\'s been a couple of should requirements added for
suggestions for future improvement. One is to reject ballots that are
only blank on one side because apparently there this is a common voter
mistake on a two sided ballot to fill out one side only. Also to reject
ballots containing marginal marks. Those are a bad news in a mark sense
environment. We will talk more about that. If the system contains
ballots with marginal marks, it can go along way toward preventing lot
of problems we would rather not get into. \-- I have 5 minutes? I \'m in
big trouble.

Moving right along. Most important thing. I swear I would talk about the
classes. I'm going to have to spend at least 3 of my 5 minutes on that.
We dealt with the marginal marks issue on optical scanners. The old
standard said you shall accurate ly read the vendor specified mark and
you shall ignore extraneous perforations smudges and folds. There were a
lot of issues there. What we\'ve got now is you shall accurately detect
vendors mark and the standard mark which shall not challenge the
equipment that\'s out there now. It\'s a bench mark to show we have a
large range of marks that can be read. There\'s still issues with
marginal marks.

Mr. Chairman: This is JR Harding on the conference line.

Okay, JR Thank you.\
\
Flater: I'm going to skip right forward to the classes.

Mr. Williams: Before you leave, I have a comment on that. This is Britt
Williams and I \'d like to call the committee\'s attention to page 102
in the requirements section. This morning, Commissioner Davidson and Tom
Wilkey, great Americans, they both talked about adding complexity and
expense to the voting system unnecessarily. And Tom talked about that in
terms of complexity in terms of literacy. But I would like to call your
attention to section 4.8- 8 as an example of what they are talking
about. The requirement that a ballot scanner be able to provide feedback
to the voter that identifies specific contests or ballots when an over
vote ballot is suggested. That says over or under. What\'s going to get
returned is an over voted ballot. There\'s not a ballot scanner in
operation that has the ability to communicate like that with a voter.
These things do not have screens on them. They do not have nice printers
on them. They cannot issue those kind of instructions to the voter So
this requirement obsoletes every single optical scanner of voting
scanner currently in operation.

Dr. Flater: This requirement is in HAVA.

Dr. Williams: let me finish and then we will see. So this adds
considerable complexity and cost.

Dr. Flater: it is also a requirement in the 2002 VSS

Mr. Williams: I understand that. The only people that this affects are
those who over vote a ballot and it gets returned to them. Now I am the
"BW" in the comment at the bottom of the page. When a voter submits a
ballot to a ballot scanner, they fully expect it to go through. Thy do
not deliberately submit bad ballots. And so when it comes back it's a
little bit startling and a surprise. What do you do here? What happens
now is that the poll worker goes over and explains it to him. Now who
benefits from this? Well if your ballot is to be corrected, the only way
you can correct it is to spoil that ballot and issue a new ballot. So
the poll worker has got to get involved. So the only person you could
benefit from this by avoiding the poll worker is somebody who over votes
a ballot and when they are told they have over voted, says "what the
heck, I am going to cancel it anyway. I am going to vote it anyway." So
here is a requirement that adds considerable cost and complexity, yet
benefits only that miniscule number of voters that care so little about
the process that they deliberately vote an over voted ballot.

Dr. Shutzer: I am a little confused. I am thinking of a DRE voting
machine. Or is this the scanner we are talking about? Do we scan the
ballot immediately? We don't. The voter scans it. So I have a ballot now
that is scanned. And I can have the system detect whether there is a
system of dots in there that determines if there is an over vote or
there is a system of dots that constitutes an under vote- mainly some
things that are not voted for, in which case one can design that system
to have it go back to the voter to determine if they want to continue on
or not. I don't understand why you can't handle both.

Ms. Quesenbery: And that is what happens. I think what Britt is saying
is that it can't tell you what the problem is. It just sends it back and
tells you there is a problem. It does not say for example on race 5 that
you voted for two instead of three candidates.

Ms. Purcell: Mr. Chairman, the (optical scan) system we have comes back
and tells you, at least the poll worker, that you have over voted in a
specific race. So you know where you have over voted.

Dr. Williams: But this is a cryptic message. It's not one the voter
would understand.

Ms. Purcell: It's printed on a tape very easily and the inspector is
instructed to show it to the voter as to what it says on the tape.

Secretary Gale: Well in Nebraska we have quite a few counties that are
smaller counties in terms of the number of voters. We have put ES&S M
100s (Op scans) in those counties. They do have a message when the
ballot is cast. For example, it will say too many votes in the Secretary
of State's race. So you can accept or reject. You do know why it is
being rejected. You don't necessarily know how to get help to get a new
ballot. That is where a poll worker would have to help. But you know why
it came back at you.

This is Alice Miller. It is obviously system specific. Ours (District of
Columbia) just says over vote. It does not say where. It does no say
what race. It just says over voted ballot.

Mr. Craft: Mr. Chair if I may. This is a perfect example of what I was
discussing earlier where it would be very helpful if instead of
proposing a change to the standard, NIST brought to the Committee an
analysis of this issue, an analysis of how the voting system vendors are
handling this for each currently certified system, what the vendors have
in the works for future versions on this and then we could make an
informed decision. It is these types of lapses that every time we try to
run through this from a 30,000 foot level bring us down to two feet off
the deck.

Dr. Semerjian: These are brought to this committee, these are not
finalized and I presume the subcommittee members are participating in
these discussions. Is that correct?

Dr. Schutzer:Yeah, I mean, to me this gets to a point this is the
following. Up to now, it seems like the vendors have been making
decisions as upon how to treat these situations on their own. You buy
it, and you get with it, whatever the vendor\'s decision is. As opposed
to thoughtfully thinking out what you think the right answer ought to be
and calling in the specifics and it\'s true that there may be some
equipment that doesn\'t hands that will way now but if you don\'t
address it appropriate now you\'re going to forever be in the situation
where you\'re leaving it up to the vendor.

Dr. Flater: If I may raise a clarification at this point? This
particular requirement that\'s causing so much consternation, this is
not one that NIST introduced without consultation. Up there on the
screen are the requirements that appear on the 2002 VSS, the language is
similar to the problems causing problem now. The adjustments were to
make some of the words sound more like they do in HAVA.

Dr. Semerjian: This sounds like something that we\'re not going to
resolve at this moment. So I would recommend that we take that under
advisement.

Dr. Williams: I think we\'re missing the point here. We\'re not \-- I
don\'t mean to nitpick this specific issue. What I'm saying is in all of
our work we could be could go in his sent what we\'re doing as it
impacts cost and voting system. Every election related person on this
panel and in the audience has brought this up and pointed out that these
things are bought by jurisdictions with very limited resources and
operated by people that \-- with very minimum training and so forth. So
complexity and cost are big issues. And when we\'re talking about adding
features to the voting system, we should do that complexity and cost
analysis. And that\'s the point I'm trying to make and I'm not sure
we\'re doing that.

Dave KarmoL: Just as a point of clarification. When I look at the
statute, Section 301 13i, it says if the voter selects votes for more
than one candidate for a single office notify the voter that he has
voted for more than one candidate.So it requires notification of a over
vote, I don\'t see where it does for under vote.

It just says it must be notified it.

Says the voting system shall. I don\'t know what \--

That\'s generic but, however, we define voting system.

Ms. Quesenbery: So just to add, I don\'t know what to this. We\'re also
started talking about a usability issue which constitutes effective and
adequate notification. Is it a code which the code worker has a
translation sheet. Is it a screen that pops up. Is it a voice that booms
out over the loud speaker. I mean, obviously not. So I think one of the
things that we might want to be really clear about and a good example of
an intersection what do we need by an adequate notification and one of
the questions are we setting a minimum standard, an optimal standard or
optimal standard, there must be something that notifies them but that
done mean that\'s system not get more elaborate and do something more
fancy if they thought there was a market for that. So this is a good
example where it\'s not just a requirements issue but what is an
acceptable way to implement the requirement that doesn\'t violate
privacy that is usable for the poll worker, that is usable for the voter
and so on.

Dr. Semerjian: Okay. Well, clearly there is a topic here that needs
further discussion and analysis. So I suggest this staff take that under
advisement and let\'s provide with the presentation.

Dr. Flater: I'm going to take one minute to address a topic that I
promised I would address which was classes in the conformance clause. In
the old standard, there were categories of systems broadly speaking
paper based DRE precinct counts and central counts and these were
described in the conformance clause of VVSG 2005. This is a problem
which is we now have systems out there in which they may contain DRE
devices and paper-based devices side-by-side. The old standard just
talks about paper-based systems and DRE\'s, it doesn\'t tell you what to
do when you have a mixture of the two. So to resolve this, we are
getting additional precision in the conformance call to define different
types of voting devices, different types of voting systems and how you
get from one to the other. And requirements are going to be scoped very
precisely depending on what sort of devices, what sort of systems and,
therefore, this problem won\'t go away. End of presentation.

Chair: Now we believe that the proceeding preliminary reports of
technical support entitled Core requirements and Testing sub committee
preliminary reports for next VVSG responses to resolution 24.05, 25005,
27.05, 29.05 and, 2105 and 31.05. My script reads unless there are
supplemental directions or corrections on technical support on related
work product will develop consistent with this preliminary report. But
clearly, clearly there are some directions and perhaps corrections. Do
we need any further discussion, or will we consider these taken on under
advisement and the subcommittee members will take the comments and the
suggestions made under consideration for their future work? Mr. Craft,
did you have a comment?

Mr. Craft: Yeah, Mr. Chairman. I've got a bit of a concern perhaps on
the peace maker side. I mean, you\'ve got some very talented payroll
here on the front row which has brought us a pile of 400 pages
representing the last several months of their lives. And who are sitting
here today chafing at the bit to go forward with the next stage of their
work. And I'm wondering, I guess, a question for staff is do you feel
you have sufficient direction from the committee and the discussion
today to start moving forward? Or does the fact that we\'ve departed a
little bit from the plan you brought in cause a problem we need to talk
about before we get out of here today?

Dr. Schuitzer:Well, I think I've heard enough discussion about just some
of the points picked at random that it probably would be worthwhile
having a more in-depth discussion and not just necessarily of the
subcommittee but to really satisfy people that they had time to read
this, digest and plow through the points.

Dr. Semerjian: Mark, could we suggest that perhaps you have one of your
committee meetings dedicated to this particular topic, notify all
members of TGDC and make sure that they have an opportunity to
participate if they so choose to clarify any issues and make sure that
there is a consensus in the direction that we want to proceed? Is that
acceptable?

By committee meeting I presume you mean CRT subcommittee meeting.

Nominally but make sure that it\'s open to all TGDC members, not open,
but I mean, they are specifically notified.

You didn\'t mean an upcoming TGDC meeting.

No. We want to make sure everybody\'s notified and provide some
flexibility in terms of the time and date of the meeting so that
significant number of pg DC members will have an opportunity to parts
pay the and if not, they can perhaps provide some comments or \--

That might be even better. I mean, if you gave everybody a chance to
read it, provide some commence and then give your team sufficient time
to address those comments in terms of their response to it. And we have
a quorum, we might be able to resolve this, at least everybody would be
comfortable that they\'ve seen the rationale, heard what the concerns
are and earring it\'s going to result until the same document or a
modified document based upon how the outcome is. I mean, that would be
my suggestion. Is that \--

I think it\'s an excellent way to proceed.

I hear the frustration because the committee meetings go on week after
week. But I think part of it is that this is such a big chunk to bite
off that not only do you have a lot to read, but it\'s a lot to
understand why each of it exists whereas you\'ve been able to work on if
over time in a focused way, we\'re trying to absorb it through a giant
water hose. So we can take any issues that are sort of big issues and
make sure that, you know, the committee knows this is what \-- we\'re
looking at this now and. So if you want to read and comment on it,
now\'s the time as opposed to waking till December. Then maybe we\'d get
a little more response because I've heard some feedback that maybe
we\'re not as responsive as we could be also.

Right.

Dr. Semerjian: All right. So what I'm hearing is clearly we\'ve
identified some issues of concern. You have an opportunity now that
these been brought up to the surface TGDC members have an opportunity to
study the material in more detail, preferably provide some written
comments that NIST staff can take you know advisement and then with some
action items in mind, on the part of this staff and some responses
let\'s say to those suggestions, then a subcommittee meeting can be
organized which would be made, you know, broadly open to all members of
TGDC and knack make sure that you are all notified of time and date and
encourage your participation in that to make sure that your concerns are
heard and addressed in this work product. Is that \--

Yes, one other thing just for consideration is as discussed earlier that
we have \-- we\'re lacking as much as interaction as a group such as
vendors and many of the comments I'm hearing here are making assumptions
as to what a vendor might or might not say about some of these things.
And so I don\'t know how best to address that. But it might not be a bad
idea to also get some early vetting of that here that and we can discuss
that as well. I mean, I don\'t know it adds time but it produces a
document that has been more thoroughly tested.

Well, I mean, the reason these meetings are open is to give an
opportunity for vendors to provide comments in writing and \-- I mean,
that could be at any time but I mean, perhaps the vendor can be notified
of the meek not for input but that it\'s sort of submitted in time for
consideration at that meeting. Is that a reasonable proposition.

I think that misses the mark. I mean, the mark that I'm after is, number
one, getting the vendors in involved earlier, so that when NIST staff
brings us the idea of the problem, we have the vendors\' input in that
already. As far as the vendors having access to the public meeting, no,
the vendors really don\'t have an opportunity in this current format to
weigh in and give this group information if we\'re discussing something
like we were a few minutes ago as to how the various systems notify a
voter of over votes. There are people in this room that can tell us
exactly how all the systems do that. And we could make a decision and
move on. But it\'s a public meeting and they can\'t speak. So if we\'re
going to do that format, then NIST needs to have proper conversations
with them, involve them in the subcommittee meetings and get some
information before the committee if you don\'t want to go that route.
Then you\'re going to have to change the structure of this committee so
that during our meetings we can take appropriate testimony from
knowledgeable individuals to support our decisions.

Mark, would you like to comments on that?

Yeah, please. There are a couple issues here. As far as getting the
vendors involved, that\'s something as I said when I stood up there that
we\'re really trying to do the best way we possibly k I think the idea
of having the vendors participate in subcommittee meetings, think
that\'s not All of you know the TGDC structure. The meetings, the public
can view but they cannot speak and participate and I don\'t think we
could give one group that opportunity without making the whole public
give them the same opportunity and that would be chaos. And I believe it
would violate the TGDC resumes. We try to get vendor input as much as we
can. We put things on the website and call them up with answers. I think
Dr.Semerjian's idea of asking a question of a vendor that concentrates
on a single point would be useful. Just to say give us input, maybe it
works to some degree but obviously everyone has limited time. So I
certainly second the doctor\'s idea of vetting this as much as possible.
Getting all the issues out on the table, publishing what we know and
asking the vendors to provide information so we could then discuss that
as a subsequent subcommittee meeting. I think that\'s the only legal and
useful way to proceed.

I think in the investment VVSG 2005 we were under a lost time pressure
and we couldn\'t afford to take long periods of time, et cetera.
Everybody knows how it worked. I think this year perhaps we could be
little more flexible and solicit more pro actively, not simply putting
something on the Web page and then say, you know, anybody who\'s
interested, send us your comment. That may be, in fact, specific topics
that may be discussed at some subcommittee meetings could be advertised
so that the vendor community can be aware of that specific interest. Get
their input in time, you know, not the day after the meeting so that we
could be more proactive in soliciting input and participation by the
vendor community. I think I see no reason why we can\'t do that.

Yes, Mr. Berger.

I've been reflecting what I've seen take place in other processes, and
I've observed in the IEEE operating under ANSI rules and Dave Karmol may
wish to make some comment on documents with this complexity, any final
document almost always is put out for a written ballot which specific
comments supporting whatever the ballot is. What that allows for is
compilation of the comments from the different balloters and then in the
in-person meetings really focus on the items where clearly there\'s
concern and especially if there\'s conflicting comments. What that also
allows, and this happens quite frequently, is those who may not be
qualified to vote can submit comments and those can be compiled either
together or separately. It\'s not easy but it tend to work through the
process pretty well. It also allows observe the makeup of this
committee, there\'s certain organizations named t allows distribution of
documents throughout the organization so you really get the collective
input of the organizational membership as opposed to the individual
who\'s here on be a specific day. So through the we may want to consider
written ballots on documents of this complexity prior to a meeting, and
then focus our meeting on the issues where comments are grouped.

I don\'t quite understand the concept of a written ballot in this\--
these are guidelines that were that we\'re developing for recommendation
to the EAC and we don\'t have a broad membership like ANSI or IEEE, this
is the group that makes the final decisions. Can you clarify?

Let me give you a different example. I've participated in national
academy of engineering and national academy of science reports that have
been provided. We\'ve been asked to develop. And we do provide the first
draft, but after we provide the first draft before it\'s ever released,
it does go out for outside review and comments. We get back to comments
and I'm stuck reading it the rest of the authors are responding back. We
don\'t necessarily have to accept every comments but we have to consider
it, give some weight to it, rationale, response back and it does provide
a stronger document and pretty much prepared for one of those comments
you\'ve got back.

Aren\'t we having that process? That\'s why it took us, you know, \--

Yeah.

Seven months to get from the final draft form to the actual release of
the standard.

That\'s \--

The guidelines.

That\'s just what I was going to say. Exactly what we\'re describing now
is the public review professionals that the EAC puts forward after we
give them our recommended draft guidelines or standard and to impose a
similar process on this body, yes, you\'d get more input but the delay
would be dramatic. And we are talking about draft standards and
sometimes very informal stages, I think to introduce that type of
process on this body would just kill any schedule we possibly have right
now. And I think the public review that the EAC conducts works very well
to actually take care of the issues that I think you\'re just
describing.

Mr. Chairman?

Yes, Mr. Harding.

I simply need to move on, but in a sense of moving on, the \[
Indiscernible \] does a very good job of what we would call an advisory
group who are guidelines and the subsequent standards and ultimately-\[
Indiscernible \] national standards so if people participate in those
standards they very much need to feel like they\'ve participated in the
development of that and not have \[ Indiscernible \] shoved down its
throat and maybe we can have the advisory group to each of our three
working committees in which a member of that advisory group then works
\[ Indiscernible \] each of our working committee to make sure the work
product and the draft product are going back and forecast and we have
the value of that input. And I would leave it to your direction to
charge our city chairs to establish an advisory groups for subcommittees
or \--

Thank you.

Mr. Berger.

I will like to report to the committee in the interest of its
information but also because I think it\'s a good process, specifically
on the EMC requirements, to two weeks ago there was a meek of the IEEE
standards development committee and the staff of NIST afforded me to
take those requirements here and that committee is in the response of
response commence and Alan Goldfine are setting up a meek to receive
those. They\'re making sure the requirements are in the best current
thinking of that field. And I think I both agree and disagree with Dr.
skall we have to watch what we do about the timing of the process. As we
get towards final decisions being a little more deliberate and inviting
of detailed comments is probably well advised in the long run. And I
guess I'm not as convinced that the public comments process is as
efficient at that as necessary but \--

Let me just remind you that there are big thick copies of all the
documentation that\'s being discussed here outside for public
consumption. So I mean, we are to my knowledge, sharing all the
information, you know, interim information that\'s been developed with
whoever is interested in showing up here as well as obviously on our
website. Dr. Williams.

Yeah. When we were developing the first set of standards, we were
working under incredible time constraints. And so I was willing to
accept this glow in the dark kind of approach to things where we came up
here and we got four inches of paper that we\'d only seen for two or
three days. We\'re still operating in that mode yet we don\'t have those
time constraints any more. Why can\'t we take more time and spend more
time, give us more time to review these things, more time to formulate
intelligent responses to them instead of run up here and grab a
four-inch thick document and shoot from the hip?

Let me ask you then. How long would it take \-- how far in advance do
you think we should send that material and you would promise me to read
those before the meeting?

I'm not going to promise you that I'll responds to the entire document
but I will at least read the entire document and select those areas that
I feel like I could make a contribution to.

That\'s what I mean.

I don\'t even have time to read this document.

Could I respond quickly? I mean, one of the things that we\'ve tried to
do is make the material available on the Web page and make it available
all the time and we want to do there as well. We want to go beyond that
and, you know, basically publicize the telecons to all the subcommittee
that better identify issues, contain a bigger overview. But what I'm
hearing, we keep coming back in circles to this one issue which is how
does the TGDC digest all of the material and how do we all move forward
in an orderly fashion? And I don\'t think we\'re going to solve that
today, but I do think NIST and the TGDC needs to discuss this and agree
upon methods we can take to move things more forward. Now we put things
out on websites and we\'ve done a better job of identifying issues and
getting agendas out. And it\'s worked in some cases. It hasn\'t worked
in other cases. But that was something we tried but I think we need to
talk, we need to do a separate informal telecon or do some meek after
this issue because I don\'t think we\'re really going to settle it right
here.

If I may?

Yes.

I think John the answer as to how we get pass this is number one, within
these 400 pages there aren\'t that many issues. We shouldn\'t be hit
with 400 pages and have to sort through the 400 pages that \-- or part
of those 400 pages that rise to the level requiring a decision of this
board. We should have confidence that past directives to NIST are being
followed. We should have a good feeling about how those are going. We
should have new issues where NIST needs direction brought to us and we
should be briefed with appropriate research to make informed decisions
about those issues. Those are the things I think we have beat this horse
just about to death today. This committee doesn\'t feel that it\'s
getting. And no, there\'s nobody up here who can afford to read a
400-page document slowly enough to comprehend it to look at all the
issues to different terms, look beyond it to the research behind it. We
have other jobs. I think in working with the committee, the research
staff at NIST is going to have to do a little better job of giving the
committee the feeling that our previous motions have been carried
forward and understanding of how those are going, and then understanding
of the actual issues coming before us. And to have an issue before us
where nobody in the room who knows about the facts behind the issue is
allowed to speak about it, that kind of format simply isn\'t going to
work.

Well, we have to take a break. We still have two more presentations.
Perhaps the presenters will take into consideration the comment just
made and rather than covering all the material in your presentation,
programs you can try to focus on even more controversial or issues that
have not been addressed before.

I can say that \--

I can say for the HFP presentations coming up we have no new standard
sections to report at all. We\'re simply recording on ongoing process
and that work.

That will be a short presentation then. Okay. We do need to take a, you
know, make a decision whether we are accepting the report as made or,
you know, with the suggestions or the modifications you know made, but
we do have \-- we have to get a feel from this committee whether the
reported work is on track or \-- I hope we\'re not off track but clearly
there are some issues that are not being \-- right.

I move we accept.

I move we accept the report on requirements as written.

Do we have a second.

Second.

any further discussions. I think we discussed it. All those in favor.

ai.

The report is accepted as written. We\'ll take a 15 minute break and
then please don\'t go too far. So that we can get started immediately if
we\'re going \-- if we expect to finish up our work today. Thank you.
Ms. Quesenbery

(AFTERNOON BREAK)

Can we start to take our seats so we can finish up our sessions?

Can everyone take their seats please so we can start it. I think we\'re
missing a couple of TGDC members, but we\'re running very late so I
think we\'ll get started.

good afternoon, I think I have about 7 minutes to talk but no worries,
it\'s a top-down presentation so I think the key here as you listen to
the talk is to note that these are progress reports. There are no
drafts, standards, guidelines here. It\'s a report on the progress on
the research and if you recall that for the VVSG 05 we did a lot of new
requirements so we\'re now switching to the research gear for the next
set of requirements. So at our last meeting, this is aside from the last
meeting, in fact, you heard about the research underway that these all
DREams directly the resolutions and I unfortunately strip off the
resolutions to make the slide less busy but the last time they were all
in here. And in particular I'm rock on the progress usability
requirements, the testing with actual voters and as a conformance test
and how to form a benchmark and research on plain language guidelines
for balance instructions and error messages. I'm not going to talk today
about guidance for ballot design and I action design. This is some very
preliminary work we\'re doing for design for democracy and it\'s not yet
ready for primetime. Usability standards is kind of pervasive what we
do. We\'ve been looking over the outlines and different restructuring of
the VVSG and that\'s something we do on an ongoing basis and we will
continue to look at refining the accessibility guidelines and, of
course, work on test methods. I am talk about on my last slides some
specific issues that we\'ll be looking at. So in the documents that are
in your packet, there is an overview of the research methodology for the
performance benchmarks. It\'s rather dense. If you look at headings you
can get a notion of the I. and there\'s also a short paper for what
makes for a good metric because we\'re going to be measuring as we
collect data and how we measure usability. And that\'s a pretty short
method and 20 guidelines in a 40 page, not very dense report written by
Jenny Redish who\'s one of the world experts on plain language, so it\'s
written in plane language as well. So in order to kind of give you an
overview of some intuition into what an usability conformance test would
look like I kind of scripted out what it would look like so the voting
test laboratory recruits some voters according to demographics that have
been specified very carefully and they\'ve set up voting equipment
according to the test specks with a test ballot or ballots, again
predetermine for that test, the voters are brought into the lab.
They\'re given precise instructions on how to vote their ballot choices
according to a test script, and the people administering the test follow
a script in how to introduce this. Voters cast their ballot being
observed by the testers and their errors and time recorded that possibly
fill out a satisfaction questionnaire. We\'re not sure that\'s going to
be some of our final metrics but we\'re collecting some data in our
experiment because that can be rather subjective. And we figure out
error rates based on metrics and time and compare them against some
benchmark. Now we\'re going to have to figure out what that benchmark is
so we\'re doing this summer some testing, pilot testing of the concept
and also to give us some idea of where these benchmarks might lie. The
voter climate fails or passes the test. And this slide really is for the
usability professionals community, usability engineers because I want to
just a make point here is that we\'re doing an usability conformance
test to a benchmark. And this is different than what usability engineers
are used to seeing. They see formative or summative testing where
they\'re improving the design. We are doing something somewhat
different.

And there\'s a number of steps that we\'re going to be doing in order to
design experiments to test our test protocol, et cetera. I'm not going
to go over the details here but there\'s a lot of iteration that we\'re
going to have to be doing.

And you should note that we\'re testing the protocol. We\'re going to
bring in and probably IT rate but initially around 30, 50 participants,
and we\'re going to test all our protocols and scripts use a similar
population because we want to see if we can get reliability because
we\'re validating our test protocol and actually in this initial test
just validating that we\'ve got the right concept that this is going to
work.

That\'s all I'm going to say about the usability conformance testing.
This would be a good time to ask any or clarification that you might
need. And I'd be certain, the committee, we take any question you have
later on and talk to you about it.

Sharon, thank you, first. This is jr. First hide like to say thank you.
This is really good. Second, specifically my question on page No. 3, you
mention bringing the voter in at the lab. The lab is a sterile
environment. Is there any reason why we can\'t create a mock of voting
precincts or, you know, doing the experiment in the church where
they\'re not \--

There\'s two points here. One, that is conformance test of the
equipment. So we have to control all the variables. So, yes, it must be
tested in the lab for that reason. But I suspect you\'re alluding to the
fact that there are other issues in deployment and accessibility issues
when you do testing. For this initial test we\'re just looking at use be
the of the system that is not designated as accessible. Follow on work
which I will allude to on my last slide is really looking at special
requirements for developing performance test for accessibility and we
have to look at other environmental factors as well.

This is Whitney. I would add, lab as term of art and it just means the
place you\'re doing the testing. It does not mean a place with white
coats and sterile environments. The plab could be a church rec room or
some other appropriate facility that\'s easy for the community
population.

We have to \-- conformance says we have to control the environment so in
some sense it\'s a sterile lab, if the church stakes has to be the
church station mock up every time across the equipment.

Would the term controlled environment be more acceptable?

Yes, I mean, I think perhaps people have perception of a laboratory like
a \--

This is a testing lab. We\'ll specify what the environmental conditions
are very precisely in terms of lighting, et cetera.

The other part that the doctor made that we shouldn\'t slide over is the
point of this \-- this is a development of a test method so we\'re
really looking forward to volume two but the point is it be a repeatable
test that anybody following the protocol should get repeatable results.
So there are some \-- as she said a lot of issues about how to constrain
that test.

Yeah, we have to consistent to constrain it so we get reliable results.

I've got to get a question in.

Let me just to follow my script, let me point out that this report
responds and work carried on responds to the resolutions 205, 305, 405,
505, 605, 805. I guess I don\'t have to do this next time if we have the
matrix. 105, 1105. So unless there are supplemental directions or
corrections, the technical support and related work product will
continue to be developed consistent with this preliminary report. So are
there any questions? I guess there are.

Dr. Laskowski, a big element in the use be the. System is the
instructions given to the voter by the polling place worker and I think
a key element of the usability is how well the voter can use the system
in spite of fairly poor inconsistent instructions so is that going to be
another dimension.

We do have another resolution that talks about polling place, et cetera.
We\'re going to script out typical instructions but not poor
instructions because we\'re doing very spars instructions and unassisted
voting because we\'re trying to capture the usability of that equipment.
Now you\'re right. In terms of deployment it could muck up the
usability. You can\'t test that in the test lab on the equipment which
is why we want to make sure that we refer to other supplemental guidance
for poll worker training, et cetera and points that out clearly and
document that elsewhere.

But kind of another metric\--

Otherwise you\'re using so many different variables that we would he
want really get a good reliable valid measure of the use usability of
the system.

I was wondering if there was going to be another dimension for this.

Not for the conformance test itself but I can certainly see for follow
on work and we do have another resolution that talks about these other
issues of deployment.

I know that you probably haven\'t read the 30-page report that this is
the two-minute summary of. There\'s a lot of good detail in there. To
let you know when the-when the next HFP meeting is and invite more
discussion then.

We are hypothesizing.

I'd move acceptance of the human progress report?

Not quite done.

Thank you.

Two more points. But thanks for the vote of confidence.

I guess they were trying to accelerate things.

I got the message. The other report we had was a report, a study that he
Jenny Reddish did for us and basically we know that the \-- a clear easy
to understand ballot and interaction instructions are important part of
the voting process. If the voters captain understand how to use their
voting materials chances are they may not be successful in voting. Let
me make a following point. That everyone benefits from clear
instruction. We note that the could go anything sanity skills vote
we\'ve got aging population, tired workers. We have got a whole array of
different cognitive disabilities. When you explain things as clearly and
simply and directly as possibly minimize mistakes, you make things
clearer. And let me also note that a lot of these populations are not
going over to the accessible voting machine. My parents certainly
won\'t. They\'re aging. But they don\'t need help. And sometimes I come
in and tired I get confused easily from instructions all the time. But I
don\'t need an accessible voting station. I need clear language. And so
I think this is also responding to the fact that there are people with a
whole range of contra disabilities that we need to design for and we
need \-- and I believe that this Mr. Capture a large number of those
people, clear instructions. Let me also note that poll workers also
benefit from clear instruction material. We are not focusing on that
right now.

Would Sharon, Mr. Chairman, one more question on that.

Go ahead, Mr. Harding.

Sharon, would we have any picture kind of directions to complement the
written word to deal with the literacy in some of the cognitive issues
you alluded to?

That\'s another research issue. Once you start introducing icons and
pictures, there\'s some research that needs to be done because of
introduces bias and pictures for all the instructions and navigation as
well as say the candidates. There\'s a lot of issues there. This focuses
only on the language. That\'s another research topic. That\'s sort of on
our list further down our list of priorities.

Thank you.

We\'re trying to get the biggest coverage possible first and then go
down to, you know, further and further down into the population.

So basically what we did was to look at lots of paper ballots and 4-D
re\'s and look at the ballots and the ballots and the instructions and
the messages to see if there was, indeed, room for improvement and we
found serious gaps. Based on that gap analysis, I'm not going to \--
there\'s some material in the view graph to talk about that and as I
say, I'm not going to talk about it here because I don\'t have time. But
20 guidelines were written just based on that gap analysis, no usability
studies or testing. So, for example, and it\'s based on best practice
from other domains. One guideline is to put warnings about the
consequences, before, not after the voter\'s likely to act. On a DRE,
this is a tesble. Some go to scope of the instructions that\'s out of
scope of the equipment per se. So the next \-- and I've got some
examples here. Again I don\'t have time to go over there but the
examples are I'll go with each of the 20 guidelines so you can look at
\-- they\'re kind of fun to look at. Small improvements make big
differences in clarity. So what\'s missing from this preliminary report
are couple things. First is which things are testable and that could be
developed further to go into the VVSG and the second thing is that
guidelines really need to be tested in the context of voters working
with the ballots and equipment. What we\'ve done so far is just on best
practice and other domains so in the next we will to look research do
voters look at instructions on the ballots, how does it affect the
wording behavior. Which words do voters understand, do they understand
cast a ballot, a contest erase, do they understand partisan and how does
that affect their voting success? So the next step is to do some
research here to make sure we\'re on the\-- that we develop guidelines
that, indeed, specifically work in the voting arena. So if there are any
quick questions about this, I'll address them right now, and then I'll
wrap up with the future directions. Okay. Doesn\'t look like any red
lights are on. I'm going to continue. \>\>\> So we\'re going to continue
our work in validating our usability test protocols and develop
benchmarks. A we are going to continue our research for guidelines on
clear instructions. We\'re going to continue our work on looking at
trying to develop guidelines for ballot design and interaction design
that would go into the equipment standard, always doing usability of the
standard itself and other specific issues as they arrive, for example.
These have arisen. We\'ve got carry over items from the public commence.
We\'ve got to go through those. There\'s also some issues about
usability of some of the security approaches and through the some of the
talks today we\'ve got illusions maybe we\'ve got to look at the
usability issue, the usability perspective and so as those are
identified we\'ll look at them. I bring up vote by phone because I
believe there\'s some guns that could be put in equipment standards
what\'s the most usable way to time the audio interface? Are there
dexterity issues that could be addressed and improved with the vote by
phone and what about the control the interaction, is it done the
simplest way possible and there\'s a lot of research in the interactive
voice recognition field that we might be able to pull up to to use.
There\'s still further dexterity issues, some about ballot submissions
et cetera, so we\'ll continue to monitor that and address those points
as they arise. J. R. had a question about going into the polling
location and other sort of issues for usability testing. A lot of which
have to do with accessibility testing. The usability test we\'re talking
about developing for the corn form as the conformance test for address
usability because we need a slightly version of the test because the
equipment is different, the benchmarks are going to be different. If
you\'ve got say the audio interface, we know that\'s going to be slower
so what\'s an acceptable rate for that? So there\'s some specific issues
and also how do you define the demographics for classes of disabilities.
And just always looking for what can be moved from the accessible
systems requirements to general requirements? I know some in the current
version of the VVSG there were some font and color things that would you
know up in the accessibility section and these I think could easily be
moved into the general equipment section with really no cost to the
vendors or very little cost. Some of that is already addressed, in fact,
on most of the DRE\'s. and that\'s the wrap-up of my talk. Any other
questions or clarifications?

Any other questions or comments?

Again I would move acceptance of the report?

We have a second. Any further discussion? If not all those in favor in
adopting this preliminary report say aye.

Aye.

Those opposed Unanimous, thank you.

Thank you, Sharon.

Thank you.

Okay at this time I call on Dr. Nelson Hastings, Mr. John Kelsey and Mr.
John Wack of our laboratory to present the transparency for the next
iterations. They promise they\'ll be short but no pressure.

Good afternoon. I just want to first take a moment to acknowledge Quinn
dang and his support in helping to create the cryptography and the draft
requirements for access control. So basically I'm going to go through
very quickly some of the draft requirements that we\'ve created in crip
grief as well as access control and talk about some of the draft
requirements as we schedule the next areas in security.

Cryptography can support Integrity, confidentiality and validation and
we wanted to consolidate general requirements into one location so if
you go back and look at VVSG 2005 you see that their cryptography
requirements inside valid so we wanted to consolidate those common
requirements. This section doesn\'t talk to or set forth requirements
related to voting protocols. Those will be developed under independent
verification requirements. Some of the topics covered or that types of
algorithms that can be used key asymmetric key. Authentication,
validation and Cryptography. Security is discussed here. This is one
area where it changes the security strengths of a given Al go rhythm and
key length changes over time. So what we\'ve tried to do in the
discussion sessions provide links to NIST websites that are kept
up-to-date with that information. Key management requirements as well as
some general application requirements. First sample requirement
basically says cryptographic operations will be performed in a FIPs 140
cryptographic mod you\'ll. Many of the requirements can be meant by
using a validated cryptographic module. It leverages a well-established
program here at NIST called the cryptographic module validation program
which has over 200 plus modules that could be used to be integrated into
voting systems. Next one is a requirement related to the key management
policy. This documentation requirement on the
\-\-\-\--\*\*\*\*\*\*&&&&&&&

Testing labs. Security policy template, similar to the model key
management document discussed earlier. Identification, awe then case,
authorization, logging, events that should be logged, access control
requirements, and communications which probably should be more of a
remote access so you may want, may only want certain types of capability
to be accessed remotely. Sample requirements is the first one talks
about modes of operations that have been defined in the document,
there\'s a table, I believe, that defiance what each one of these modes
means so you have pre-loading open, so we\'ve coordinated with the crt
folks that they have developed, and the second requirement basically
says that you might, you could apply different access controls for each
of the modes, so in pre-loading, you may want to, administrator should
be able to up load a definition file. second requirement talks about
possible groups and roles that, within the voting system itself, so as
the user, a voter or is the user a poll worker, the administrator, so
we\'ve defined several different roles and would like your feedback on
the role and modes of operation if we, if we define too many, we
haven\'t defined enough, we would like your input back on that. again,
we\'ll continue to develop these and refine these requirements based on
your comments pride we are doing some additional research to check how
far we deviate from the VVSG 2005, and the i.e. EE standard, and
there\'s an ANSE standard that, on role base access control, and we need
to research that and see how to best leverage that standard for the VVSG
2007. next requirements that we are looking at, and this is subject to
change, is event logging system, event longing communications and
software division, and looking to have draft requirements in the June
time frame. other items that are still left on the table are software
installation, set up validation, physical security and those things, and
at this point, I'll open it up for discussion.

for the sake of, you know, since we are under time pressure, let\'s go
on with the 3 presentations and open it up for discussion.

Use this over here.

Hi, can you hear me? So i'll try to run through this quickly. This is a
talk on open ended evaluation voting systems, and I'm John Kelsey. So to
put this in perspective, a year ago, I gave a much less specific talk on
the same topic for you guys. And then I talked about what we were going
to try to accomplish, in this work, we made some progress on this.
That\'s still a lot to be done, so we, the history here is there\'s a
TGDC resolution that told us to look at open ended, kind of add a
significant amount of an open ended search for vulnerabilities in the,
in the voting system evaluation, and so then what we\'ve done, we\'ve
had this preliminary threat presentation and paper from last year, and
the open ended vulnerability testing presentation outlined a high level
idea of this. More recently, we\'ve done the NIST threats to voting
workshop which was a great success, I thought it was, and we\'ve been
doing some work with the Brennan center on a more formal thread
analysis, and this is pushing towards the same goal of figuring out how
to make voting stronger by knowing how to attack them, so kind of, in
that context, what I'm going to do is talk about what open ended testing
is and why we need it, and talk a little bit about how we plan to do it,
and a lot of this is in the air, we are doing things that haven\'t been
done on this scale before, so we need to go slowly and learn what we are
doing as we write the standards and get operational experience with it.
I'm going to try to sprinkle this talk with technical questions and
policy questions we need to resolve . These are questions I hope you can
shed some light on. So, at a high level, open ended testing \-- the easy
way, and the standard way of trying to verify that something complies to
standard is a checklist approach, smart d checklist approach, but the
question you are asking is does the system conform to the standard? So
we want to make sure that the voting system has the right kind of
security controls, right? It has, you know, the, you can\'t get in there
and mess with the memory cards, without breaking the lock or breaking
the seals, stuff like that, and you wants to make sure the controls are
configured controlled or installed correctly, lock really works, you
can\'t pry open the door without opening the lock, something like that.
Open ended testing is different. With checklist testing, you really, you
just go down a list and say does this comply with the requirements? Does
this \-- open ended testing is somebody trying to find a way to break
the system, to find a way around security control, and so you were
looking for basic design flaw that let\'s I break the thing. It has the
lock, but you can get around it somehow. Okay. An analogy I used, the
analogy that I've been using, you can have a policeman come and check
your home to see if it\'s secure, right? And he will say this door has a
bad lock, you need a better lock. He has a checklist. Probably that
policeman has never broken into any houses, you hope not, at least not
for a living. He has a list of things he knows that are potentially
problems and will tell you what to fix. That\'s checklist testing. It\'s
valuable. Open ended testing is like having a professional burglar come
and see if he can steal your TV. It\'s a different approach. The good
thing about this is even if you do everything right on the checklist
approach, sometimes there\'s vulnerability it won\'t address. The bad
thing is the quality of open ended evaluation is really heavily driven
by the quality of your evaluator, somebody skilled will find flaws that
just a normal person wouldn\'t find, and that\'s the interesting thing
here. There have been some examples of open ended voting systems, the
stuff, at the top is conceptual analysis, Harris book from 1934, it
talks about some very broad threats to voting systems, the NIST voting
threats workshop, and these are not exactly what we are talking about,
just an idea where you look at voting systems in a broad sense as to how
you attack them. More specifically stuff, and we only have a few
examples that are public and widely discussed is Hopkins report, SIC
report, public attacks that are done partly for publicity and partly to
show a vulnerability, and probably the best example is the robber
report. I won\'t try to read this whole things, this is a quote from the
report. It is probably the most professionally done of the open ended
analysis I could find. Talk about a red team exercise, and high level,
the goal of a red team exercise is to set up a system in the, in an
environment where it looks like the one it will be used one, and see if
you can find flaws in it, see if you can attack it, and idea is you can
attack this system, evaluation team can try a lot of different attacks,
they are not con strained, and if one attack fails, they don\'t lose any
points. Okay. And the hope is you\'ll discover a lot, kind of a lot of
potential vulnerabilities, so that the interesting thing with that
report, they found a bunch of practical flaws on a particular voting
system, and once you knew about the flaws, I think they were pretty easy
to fix, so you can see an example, something where you have this
analysis and you could mitigate some of these risks and just fix the
problem \-- mitigate these procedures. And it seemed like a valuable
thing you could look at this, and come up with a list of potential
attacks, and you could actually fix the problems, and the next person to
look at those to may be more hostile doesn\'t find the easy attacks. So
the high level idea here is something that, instead of the big attack
world, people will call low hanging fruit. If you want to make the
voting system stronger, you want to pick the low hanging fruit. You want
to find the weaknesses in the voting system that is easy to find. Now,
in a real world computer system, something as complicated as the voting
system, you\'ll never get rid of all the bugs, all the potential
attacks. What you can do is make the attacks harder. You can close the
easy vulnerability, maybe somebody be a gene \-- make somebody be a
genius to break in. That\'s the goal you are trying to accomplish here.
So one of the places where we make voting systems stronger is by fiction
the weaknesses, \-- fix of weaknesses. The other is preparing for a test
can improve your design. We are going to have documentation
requirements, and I will talk about this in the submission package that,
if you do the exercise, I think you will understand your voting better,
and more to the point, you\'ve described it to the evaluator so they can
check urologic your lodge \-- your logic. If you know your work is going
to be checked out in this way, you are more careful. The goal is to have
the voting system vendors attack their own system, they will know it
better than the testing lab, and there\'s incentive for them to spends
significant resources to fix the vulnerability. Let\'s just find the
problems at first. That\'s the hope. Okay. It\'s the broad just case for
what \-- justification for what we are trying to accomplish. I don\'t
have huge amounts of details as we go down some layers, there are places
we will have to do more research to see what it\'s going to look like,
but the broad process is there\'s going to be sort of, some sort of an
agreement between the lab and vendor, which means what does qualify as
an attack? And if it\'s not broken, they produce two reports, one is an
internal report for the vendor, make the EAC, and another thing is the
external report for the public, which doesn\'t detail exactly what tools
were used, but says here is what we looked at, here is why we think this
is okay. So, go to a little more detail with this. Kind of at a high
level, rules of embankment amount to what access and resources an
attacker is allowed to assume in doing an attack. Example worry \-- if
you look at the way that staffs are rated, they are rated in the times
of the attack, and something like that probably makes sense for physical
security, right? You are concerned about widespread attack on a voting
system, you probably don\'t care where the attacker has to spend 6 hours
to break into each voting machine without leaving traces, and that\'s
over the course of a month. You might have rules of engagement that say
the attacker is given 15 minutes with hand tools, and if he can open the
back of the thing, get access to the internals voting machine and not
leave scars on the back of the machine, that\'s, that qualifies for an
attack. I don\'t know what the right parameters are for that. That\'s
the sort of thing we go into in the rules of embankment. The mol si
issue is \-- policy issue is how much should be defined by the standard,
and how much is open for negotiation between the vendor and testing lab,
or alternatively, should be evolved by the EAC. Second part of this is
submission pack. There are two parts to this, first is we wants to
request security documentation from the voting system vendor, wants them
to explain basically what are the controls you are using to accomplish
the goals, and why should we believe they are secure? That will make the
evaluation team\'s job easier when they are looking at this system, if
you say, no software could be installed on that voting machine because
of this, this and this, then that gives the team a chance to read that
documentation and see if they fundamentally disagree with it or it\'s
incomplete. So the other part of this is procedures. Now, you can\'t
mandate exactly what procedures will be done in the states or counties,
but the voting system vendor needs to apply the procedures, because that
affects potential attacks. Also, there are specific things that are
dodge, for example, re counts, where it seems it\'s not always clear
what the producers are \-- we like to have that spelled out so that the
evaluation team can look for doing a re counts, normal voting, provision
at ballot, anything like that might be a problem, so you check that
actually does what it\'s supposed to do. That\'s just a simple goal.
Interesting policy issues here, first of all, what procedures should we
be including? I think we have to specify minimal ones. Broader question
that applies to all testing systems, or all testing regimes is how we
insure the accuracy of the submission. This is mission, security
documentation says they are doing things and note really doing it, that
would be bad. Same applies for the systems that are tested, and are they
the ones that are given the field. Then we get into the more interesting
attacks, right? So there are two different categories, full attacks,
which basically probably are mostly done looking at the documentation,
where you say this is how you would violate the whole security of the
election system, fix the election, maybe violate voter privacy, okay?
Any of those things, if you can demonstrate a way to do that, that
should probably fail the system, although if it\'s being done with
documentation, it may be the vendor didn\'t write the documentation
correctly, so you would assume the lab would sends them back a note and
say hey, it looks like attack, and they may say what a success, or fill
in the details. The interesting question is what is, what should the
definition of full attack would be. How much should be negotiable with a
vendor and testing lab and how much should be fixed in the standard. The
kind of interesting bit of this, of our work here is the intermediate
attack goals. The idea is that instead of making you come up with a full
attack on the voting system, if you can violent the security in some
fundamental way, if you could show you could install software on a DRE
without the proper access, that should be enough to fail the system
since that violate the standard. Cause software to run without
authorization, cause a log able event to happen on not have an entry to
corresponds, and I don\'t, the precise list is something we have to
develop, but the idea here is if you can get to any one of these
intermediate steps in a bigger attack, that should be enough that the
system fails. Fails there are 3 reasons for wanting to fail the system
when you have an intermediate attack rather than making you actually
spell out a full attack, so not just get the \-- but show how you would
fix the whole election. The reasons for wanting to do this. Intermediate
compromising attack goal, that means you violated a goal. That means you
failed. The second thing is we would like to encourage defense in depth,
a lot of real world attacks, you look at the difference between the
attacks in the lab and in the real world, often, it\'s hard to get from
step 1 to step 2 to step 3 to get the full attack to work, and one way
of making that harder is to make sure you just don\'t say well, it\'s
okay to have step 1 and 2 be easy, but step 3 has to be hard. You say
all 3 steps have to be hard. That\'s the idea of defense in depth. You
have to get past the lock, alarm and the dog, not just one. The last
thing there is to save the evaluation team time and resource. If you
look, \-- also, that they spent a lot of their time working on how to
get from the vulnerability that should have been enough to fail a system
to the actual attack. And I'd like to see that time spent on finding
other vulnerabilities to patch rather than on proving their case, right?
So an interesting question that comes up, is how you decide whether the
system passes or fails, okay? Now, the assumption is you are going to
have unambiguous pass and unambiguous fail. If the value tapes team has
\-- evaluation team has 20 attacks, it will fail. There are probably
going to be gray areas, and I think the goal of the standard here needs
to be to try to minute might of ambiguity in the gray areas. And the
policy issue is whether the lab should decide the pass/fail itself, or
write a report and then provide that to say the AC or somebody, you
know, produced by the AC and have them do the decision to pass or fail,
and I talked a little bit about that in the document. The last bit of
this is the final report, and this kind of addresses something that I
think JR talked about earlier. That you have this internal report to the
vendor, which hopefully the goal here is to help the vendor make the
system better and make the next version of the system better so, this
will include, if it passes, you might have things as an evaluation team
to testimony the vendor, this seemed like an attack, but it didn\'t go
anywhere. The goal here is to let

Convince themselves this, that this evaluation was done in a meaningful
way, that\'s going to need to specify the rules of engagement and
procedures, so if you know that the assumption here, where you hand
recounted every hundredth DRE or 20th one, if that was the assumption of
the procedures, you need to know that\'s where, what you assumed to do
the evaluation, and if that\'s not the case in the real world, maybe the
evaluation doesn\'t apply. We want to list what was located at and how,
I gave a summary of that in like what the example, a paragraph or two
per immediate attack goal, we tried a crowbar, a lock pick, tried a hack
saw, nothing worked. That would be the thing, maybe more formal
language. External report is supposed to tell you enough you can
convince yourself this evaluation was done correctly. So, interesting
\-- policy issues involve resource and money issues. So the question is
how much money is available for the open ended testing? And then, of
course, there\'s a whole bunch ever things going into the testing
budget, open ended testing can be expensive. When I tried to budget this
out myself, I couldn\'t see doing the evaluation for under about
\$100,000, and I think you run higher than that. I figure about
\$100,000 to do proper evaluation like this. You will spend time just
ramping up to understand the system

Might be less, might be a little more, but that\'s the orders of
magnitude. And I think that speaks to the issue that Paul and Britt were
talking about earlier about the large number of tests for different
states. You would wants one really well funded test that went into a lot
of depth, unless 30 or 4 tests, and nothing doing a full evaluation. \--
30 or 40 tests.

And there are some interesting \-- interesting issues, this is true in
all kinds of testing systems, so the lab is probably paid by the vendor,
so we need to you\'d lab accreditation, use lab accreditation. The big
question, I know I'm out of time, so I'll get out. Big question is
whether it is a feasible approach. There aren\'t a lot of examples of
this operationally, and our current plan is to go very slowly in this,
to do a lot more research but we write any binding standards, and to, or
whatever is guidelines, and also to try to start out with the idea of
open ended testing being done on the parts that are easy to do.
Automated scans of vulnerabilities. Some of the parts of the standard
can\'t be value wait await a minute any way, but the open ended
evaluation, and hopefully, as we gain operation at experience, we can
increase the resources on open ended testing, and that\'s it.

Thank you, John. We\'ll proceed with the last presentation by John Wack.
I was watching a TV show, and it talk about how you can take some
submersibles to the bottom of the ocean, where the plates join together,
and you have a high pressure environment with vents of air, and new life
forms that don\'t obey the laws of nature, and I thought that in a way
is like VVPAT. It\'s where the rubber hits the road. You have new
technology that has really been invented, tried to follow the existing
standards, but there weren\'t standards at that time for VVPAT, and new
election procedures have to be invented. It introduces all sorts of new
legal issues with ballot and so forth, it\'s an extremely interesting
area, but it\'s a work in progress, so, what I'm going to give to you
today is essentially a quick update, I'll just explain as I go along.
Let\'s just started right off with, just a little bit of an overview of
what is going on with VVPAT. I tried to find out how many voters would
be using VVPAT system during the year 2006, and I basically ran out of
luck. I finally came up to five plus states will be using VVPAT systems,
but I don\'t know how many voters, I think we will find o, I'd say, over
the months of June, July, August, that we\'ll get a lot of feedback from
elections where VVPAT systems have been used. The previous time I think
was in Nevada and will basically be able to see the results of using
them on a more widespread basis. It will be very interesting to monitor
how well the systems perform for audit purposes, and usability,
usability not only for the voters, but for the election officials
themselves, so it will be interesting time, I think, and this gets into
basically what I want to talk about, which is broadening the VVPAT
requirements that are in the VVSG 2005. When we came up with this time
line, came up with the chunk-able modules that could be swapped, and
VVPAT was the first one, it brought to head some issues in that we
didn\'t think that the research was really complete that we could
provide a complete finish to VVPAT module that we could swap in, and we
wanted to be like doctors and above all do no harm, but we wanted to at
the same time accommodate what we thought were some legitimate comments
received during the VVSG comment resolution period. And in particular,
we received some comments from the vendor of an electronic ballot
marking device, who mentioned basically that such devices do produce a
very nice piece of paper that is a ballot that can be used as the
official ballot of record, and it is voter verified because a voter
handleds, looks at it, and in mi cases can take it and deposit it into
an up can device where it can be scanned in, and there for, you have two
records there. What i'll get into is, I'm going the wrong direction here
intersection doing a bit of an update, we did some research, talked with
some vendors, and came to the understanding that it would be better to
not con train the existing requirements to DRE\'s being the only types
of voting systems that can in effect be part of a VVPAT system, that if
we opened it up to other types of voting systems that essentially do
ends up producing a voter awd trail, we would \-- audit trail, and we
wouldn\'t be changing the requirements in any big way. So I will just
note, maybe beat it into the ground that it\'s important to focus on
what VVPAT can be. It\'s an audit trail, but a paper audit trail, and
it\'s a voter verifiable paper trail, so we believe that using an
electronic ballot marker device with an out scan system in effect
produces a voter federal paper audit trail. Now, to broaden the
requirements so that they would allow the different types of approaches,
not specifically, I should not, not specifically to broaden the
requirements for only electronic ballot marker device but to broaden the
requirements in general to allow different approaches, it essentially
requires that in some requirements we have to proceed them with for DRE
systems, and a quick example on the screen. They produce an electronic
record, optical scan systems currently do not, but for the purposes of
providing a voter verified paper audit trail that can be used in an
audit of the election counts, it\'s not specifically necessary that
electronic records for each ballot be present. Along with this, we noted
difficulties in the press with auditing some VVPAT systems, and we need
to at some point in the introductory material basically describe the
results of our thread analyses, threat analyses, and we aren\'t there
yet, but we can say that we believe that there has to be a basis of
auditing for voting systems, that no matter hour secure our requirements
are, they have to depend on the fact that some sort of audit will occur,
and in this particular slide here, I just want to point out that with
VVPAT, you have two records, a paper and electronic, or a paper and the
machine totals, and obviously, the paper needs to be used in an audit of
the machine totals, and/or of the electronic records. A lot of the VVPAT
systems produce not only a paper record, but a bar code, and the bar
code is basically supposed to encode what is in the paper record, and
essentially one can take a bar code scanner and scan that in and you\'ve
got it in memory, and you can more easily manipulate, but the fact is
that is a third record, and it\'s not a voter verifiable paper audit
trail record. The voter does not verify that bar code, so if you are
going to use those in an audit, it\'s imperative that basically the bar
codes themselves be audited to in junior should do match up with a paper
record \-- insure. So it has to be a two-stretch audit, \-- two step,
and states that decide to take this approach essentially have to take
this into mind, and it does call into question what software you are
going to be using to perform this audit. And whether that software has
been in suspected along with the voter system code, there are issues
there. Where will we be going with VVPAT in the final version? Earlier
in the morning, I talked about, well, I basically said in many ways,
there isn\'t a lot new in the VVSG, biggest contribution we are making
is we are specifying the requirements well, looking at them to test,
making the document easy to use, trying to simplify. The existing VVPAT
requirements are monolithic, they basically need to be distributed more.
There are accuracy related requirements, reliability related
requirement, workmanship related, usability, accessibility mixed in with
VVPAT, and those belong in other parts of the document. So that will
change. We need to do more research in the area of electronic paper
record formats, and we definitely need to explore more of the issue of
use ability for election officials as well as voters with VVPAT. So I'll
leave with you this \-- oops, going in the wrong direction again. Open
areas that we need to look at as these requirements evolve. We will talk
about these issues in some of the STS tell comes, whether bar codes are
a good idea. They exist because basically if you have a relatively small
paper spool, you can scan in the bar code quickly, and it\'s easier to
do that than to actually read the paper spool. But again, it introduces
\-- again, it introduce complexity, it\'s a third record, you don\'t
know what is in the bar code, so is it in general a good idea? Is this
something the standards ought to depend upon? I've talked about more
study needing to be done in the area of usability, and ease of auditing,
I want to highlight that voter verified paper audit system is the voter
in a sense being able to compare to records and prove that the voting
system is working correctly, but depends highly, the other part it
depends on the ability of the records to be audited, and if the records
can\'t be audited easily, then it\'s essentially not worth doing, so we
need to make sure in the requirements that we specify good usability for
election officials when it comes to the VVPAT records. Another area that
really goes in many areas of the VVSG, but specifically right now for
VVPAT is the some sort of common format for electronic records, and
we\'ve talked about going in the direction of EML, Lexmark-up language,
but we recognize in there were a common format, and records eventually
had things in them such as digital signatures, identifications of
machines and things like that, it would make auditing more simple in the
long run. And, discussion. So I've tried to make up a little bit of
time, and I've gone over material quickly, but I think we are at the
discussion area now, and I think it\'s discuss area for all 3 of our
presentations.

Let me just point out that we believe these preliminary reports of the
Security & Transparency Subcommittee respond to resolutions 1205, 1405,
1505, 16, 18, 21, 22, 23, 35 and 3905. So a lot of resolutions are being
addressed through this work. And unless there are supplement Al
directions or questions, technical support and relied work product will
be continue to be developed consistent with this preliminary report. So
are there any questions, further directions or questions? Mr. Berger?

John, thank you for the report. Just have one question. The VVPAT is
intended to be a solution to a problem. Just would like your, you to
reflect on are the standards written such that if a better conclusion
come along, it can be qualified?

Well, when you say better solution, would you be thinking of, you know,
some solution that didn\'t necessarily use paper? Basically, it\'s going
back to almost a security, what\'s the threat that we are worried about
and how is this the solution if somebody comes up with a better
mousetrap.

That\'s the tough one to answer. We have been pushing the concept of, we
called it IDV, independent dual verification, and that work is evolving
right now, we\'ve had a lot of very active discussions, trying to boil
that down into what do we really need for records, for a record of a
voting system to exist that is independently verifiable and can be used
in recounts and audits, and basically prove that the machine is
functioning correctly. We don\'t have specific answering for that at
this point, we are still going down that path. I want the VVPAT
requirements though to basically be a subset \-- I didn\'t turnoff my
cell phone. I apologize. That\'s very nasty of me. \[laughter\] the
VVPAT requirements should be a subset of the I.V. requirements, so we do
not wants them written in any way that precludes any other approaches
that perhaps are more flexible, when I made the comment about paper, I
didn\'t mean to be funny or facetious, but paper has been noted as being
difficult to handle, and if there are other ways of doing it that don\'t
necessarily involve paper, we would want to definitely explore those
ways.

John, I'm sorry, maybe I missed something in one of our meetings, but
didn\'t we change the term here to voter verifiable?

Yes, we did, and I noticed that about a minute before I came up on stage
and was hoping nobody else would. Did is not voter verified, it is
potentially voter verifiable. Number of voters that verify VVPAT
records, we don\'t know, but I would say one in five.

That will be corrected with all the material is positioned on the web
\-- posted on the web.

Do \-- yes?

John gale from Nebraska, in listening to your comments and looking at
your notes, I don\'t see that you\'ve tied the EBM in this particular
presentation, as, in other words, like an auto mark ballot as voter
verifiable \-- I guess that\'s good from my points of view, I think they
are two distinct products, but is that contained? Did I miss something?

It\'s my contention that, a honor mark or another similar sort of system
does essentially produce a ballot, and if the voter picks up the ballot
and can inspect the ballot and put it into an out scan or a tabulate
tore system, that in essence is a voter verified paper audit trail, you
end up with electronic machine totals, and you ends up with a piece of
paper, the piece of paper can be the ballot of record or it can be a
paper spool, but in essence, you ends up with two records, and voter has
verified one, and that record can be used in recounts, and in high
quality audits, so we contends that in essence, you are creating a voter
verifiable audit trail.

I guess I see them as so distinct, I don\'t think the courts of law have
resolved that issue of what is a ballot, but I think it\'s very clear
with the mark, in a is the official cast valid, and with a voter
verifiable paper trail \-- verifiable paper trail, the official is the
electronic, and this is just a piece of paper that maybe is used or not,
so I hate to have a system that has a very clear product, paper ballot
that is tabulated, and suddenly put in this mass of confusion and fog
that resolves around the VVPAT, so to combine they are you are
prejudging some things that the law will decide, and if we build some
assumptions here, assuming a clarity that is not going to be there when
the courts handle it, I think it is jumping ahead of the game. I think
they should be maintain fld as separate systems until a court determines
that there were the same as a matter of policy, which hasn\'t been
decided. You are jumping out ahead of the courts and making a
determination that I don\'t think will be there

Can I ask a follow up?

Do you consider an up scan system, a human marked paper ballot that has
been scanned to also be in a similar category?

I was hoping not to get into that.

I'm sorry.

It\'s \-- I don\'t have a good answer for you there. I mean that\'s,
we\'ve been working on the accuracy requirements, or David has in that
particular area, and doing active research. But its probably see is that
if you have hand marked or manually marked paper ballots, you have
something that is potentially ambiguous and basically, \-- well, when
they are scanned, most likely scanned accurately, but not always,
depending on marginal marks, and you know these things, so can it be
used to create an unambiguous audit trail that if you gave it to 3 sets
of election judges, they would come up with the same conclusion? I
don\'t know the answer to that yet. And I'll get back to that I felt we
were safe broadening it but not yet its hand marked ballot.

I want to respond to the secretary\'s comment. I think one interesting
question for this committee is sorts of our rules of engagement. We have
voting systems which as pieces of equipment produce multiple records,
some maybe paper, some electronic, some paper marked by people and so
on, and the question that you raise, distinction you raise is which is
the ballot of record. We are rieg standards up to date, at least in the
security committee, we have knots paid attention to that distinction,
because states do vary on this. We care about things like the
correspondence between the paper and electronic and so on, but if a
vendor was to submit a system for certification, then under your
interpretation, you would have to specify which is the voter the ballot
of record produced by this machine, and that usage modes, and to my
understanding, we\'ve not have vendors submitting voting systems where
they submit this particular records with the paper records as to be the
ballot of record, and only certified in that usage mode, and in we wants
to get \-- if we wants to get into that, that would be interest, bs the
understanding of our task is not to take those considerations into
accounts

I'd like to give a thought that we really don\'t \-- actual usage of
these things, what we call the \-- we really don\'t fully understand
thing, it would be interesting to do some tests. I would reckon that if
I was on a machine and selecting based upon the screen, and then I got a
prints-out which was the vote of records that if that printout did \-- I
would think a good percentage of people were not looking at what they
were casting, and what, only they they looked at was on the screen. They
were looking at the vote of record.

There\'s been some studies along those lines \>\>\>

What he reported in the i.e. EE committee, is 5% will look at the
record.

Did he not do a controlled study on whether people actually verified the
record. He observed during an election to see if he thought people were
checking that paper record.

A good test would be to have a case where we printout something
different and see if anybody catches it.

I think the committee is focused on correspondence as the doctor said,
you can assess whether the same information comes out of two difference
information channels, and perhaps that\'s something for further
discussion in the future.

Mr. Chairman, John gale. I think it\'s critically important, I think
it\'s a fundamental thing that we have to decide because a manually
marked ballot in the minds of every voter is an official ballot, maybe
they were not so sure about electronic ballot and maybe that\'s why they
need this paper trail, but in Nebraska, we will end up with two ballots,
man knew marked ballot, a ballot marked through the equipment that
becomes an official ballot as well, but we are going to call them two
different things. The manually marked ballot, everybody says that\'s
fine, that doesn\'t have to be called verified paper trail, but the one
that comes through the auto mark is going to have to be called a
verified paper trail that may or may not be considered a ballot by a
court of law, it seems they are the same thing, you have a manually
marked ball \-- manually marked ballot, and a machine that does it, and
they are the official \-- but yet, the official ballot under the DRE
standards is, as I understand, is the electronic ballot. And the other
is only for use in the events much a court contest in the election. \>\>

One complicating thing, I don\'t know which state specifically, but some
states in the statutes basically say that in the events of a recount,
the paper, the VVPAT paper spool will be the official record and since I
don\'t think that\'s happened yet, it will be interesting to observe
what happens at that point but I think in the research we are doing in
I.V., independence verification systems, we haven\'t made the
distinction of the, well, if there is a paper trail or some other record
produced of that being specifically a ballot of record or some ancillary
audit report. We haven\'t gone that far into the policy areas, I hear
what you are saying, but we haven\'t addressed that issue.

It brings up a point when our canvassing board sits, we are certifying
the election based on the certification of our county officials who are
certifying paper ballots, and they are manual or auto marked, but they
are paper. That\'s what we certify. You do not certify as far as I
November the paper audit trail from DRE\'s. They certify the electronic
vote, unless there is a contest. That\'s the direction I understand many
states are going, but it is a matter of state law that I think that
confusion or that oversight has to be resolved so that we are not trying
to answer the political policy issues through some equipment guidelines.

May I call on the commissioner to see whether DAC is, this is something
you want to comment on? Something that this committee should be
concerned about? Would you like to think about that and maybe respond at
a later time?

We would defer to think about it and get back to you. (inaudible).

Just wanted to probably comment \-- let me ask you, that number that I
quoted, do you that I that\'s wildly off the, do you think that\'s
wildly off the mark?

5% of people look at a printed record and verify that\'s how they voted
on it.

I haven\'t the slightest idea, I haven\'t done the research, I think it
would depend on the instructions they were given. It would depend on the
presentation of the material, how and when it was presented, and depend
on the state election laws in which it was happening. I think that\'s,
do people check their bank records? I don\'t know. I think he does, but
I don\'t think there\'s any rule of thumb about that.

I \--

The other question, does it matter it only a few people check it?

I guess I would offer this observation. In the popular mind, at least,
this is viewed as a remedy to a potential threat if our research shows
that it\'s not as effective a remedy or protection as might be afforded.
We \-- that might indicate some direction.

Doctor?

I think there\'s a misperception that everybody needs to check the \--
in other words for it to be an effective deterrent. You have good proof
the paper don\'t agree to what they voted. So even if the number was
lower than commonly thought was necessary, it could be effective as a
deterrent.

Okay. Do I hear a motion to adopt this preliminary report?

I move to adopt.

I second.

We have a motion and a second. Any other comments? If not, those all in
favor, show of adopting the report, say aye. Any opposed?

NAY. One vote opposed. The report is documented. By majority vote I
think Mr. Harding is waiting, and at this time, I would like to open the
floor to the introduction of any regs loses, and we \-- resolutions.

Dr. Harding: Thank you, Mr. Chairman. Before I (inaudible) this thing we
call \-- and in that regard, it makes me think ever the access boards
and 16 years we\'ve had the \-- I would like for America, even though we
have verifiable standards that are precise, they are implemented
differently. This hotel right here, case in points. Just spent \$5.7
million upgrading stuff and 20% of it on disability things, but there\'s
a half of a dozen things in my room that do not comply with the
standards. Now. Does that interfere with me using the room? Well, not me
in general, but another member of my community would very much have a
problem whether there isn\'t even a toilet paper dispenser. It makes me
want me to then move a motion to increase the interaction, I don\'t want
to call it the outreach, but the interaction by the EAC and specifically
the TGDC with the disabled community in the development of the BBS 2007.
I would like that motion to reads that we move the subcommittee chairs
to consult with the commissioners to develop an action or strategy plan
to more involve the disabled community with the relevant issues for the
BBSG 2007 requirements that are being considered, and that\'s this
interaction of events, perhaps include public hearings or special kinds
of involvements regarding the voting requirements as we work on issues,
and specifically address voting issues.

Does that help?

JR, I think you\'ll have to reads \-- do you have this written down?

I have it written down, but \-- Allen, do you have it?

No. Can you go slowly? We could probably write it down.

I would like to move that the TGDC subcommittee chairs work in
consultation with the EAC commissioners and the chair of the TGDC \--

Hold on a minute.

Sorry.

What I have is that you move that the TGDC subcommittee chairs work in
consultation with the EAC, and go from there

And the chair of the TGDC \--

Go from TGDC.

To develop an interaction strategy to involve the disabled community in
the review of its relevant \-- let\'s go to, to involve the disabled
community and take it from there.

In the review of relevant BVS 2007 requirements being considered at this
point

Take it from 2007 requirements.

Relevant \--

Being considered.

Being considered by the TGDC.

That\'s a given.

Okay. The last part then is its interaction/outreach event could include
public hearings or other special inclusion activities or voting
requirements that include the cognitive disabled voter.

Outreach activities could include \-- and go from there

Public hearings and other unique events

Other unique events.

That specifically address voting requirements for the disabled.

Specifically address voting requirements for the disabled.

And specifically the cognitive disability voter. Or the voter with
cognitive disability issues is the politically way to say \>\>\>

Especially \-- after disabled? Take it from there

Especially those with cognitive disabilities.

What we have here, and, we can modify it, move the, you move that the
subcommittee chairs work in consultation \--

Let\'s mods few that, it\'s \-- modify that. Say that move that TGDC
chairs and the subcommittee chairs, I think that\'s what he meant.

Yeah. So starts there. And sub committee chairs to work in consultation
with the EAC, and take out the one after that to develop. Right. So move
that the TGDC chair and subcommittee chairs to work in consultation with
EAC to develop an interactive \-- I think to \--

Interactive strategy. I didn\'t want to say outreach, because outreach
is such a loose word.

It\'s not an interactive strategy. Strategy to involvement of disabled
community.

There you go. Thank you, Whitney.

So take outs the \-- to develop a strategy to involve the disabled
community and review the relevant TGDC 2007 requirements that are being
considered by the TGDC outreach activities could include public hearings
and other unique events that specifically address voting requirements
for the disabled, and especially those with cognitive disabilities.

That\'s afirm tif.

Is that, does that capture?

It does, Mr. Chairman. Thanks.

Okay.

I'll second.

We have a motion and second. Any comments? Questions?

I'd have a question. It seems like the same thing could be said of
vendors, for example, in terms of promoting their earlier involvement.
It seems like there is a procedure and orders to these things where the
staff develops recommendations as we\'ve been hearing today, we develop
resolutions and policy, and eventually, there are public hearings will
all relevant and interested partners and groups get to testify and
submit written commentary and testimony. If we keep moving that earlier
and earlier, it seems like it makes us over burdened with participation
before we develop something that people can look at and I'm not
objecting to your thoughts, but it seems like the public hearing part of
it is the logical part but earlier involvement, how do you decide,
let\'s say we are talking about vendors, how do you decide which vendors
will be culled earlier, and \-- consulted earlier, or those with
cognitive disabilities, are we able toss identify organizations and
groups that would be representative enough to be consulted without other
people objecting if we consult early? Does that make sense?

Yeah. Mr. Secretary, you hit the nail on the head with the latter part
of your question \-- (inaudible). The answer is yes. The second
components to that is as we are did the work for the \'05 there was some
parts of the community who felt that we owed this community special
outreach \-- instead of saying this is what we have to do. You are going
to take it and like it, and that to me I think it\'s a strategy to bring
in a population that disenfranchises from this process and I'm
attempting to \-- (inaudible) to dealing with the first part, are there
more specific things you feel we should be doing to accomplish involving
the disabled community?

Well, thinking along that line, for example, we had a big debate on the
(inaudible)

I'd like to come up, slightly different direction, when we started this
committee, a thing we did in September of 2000 something, four,
probably, was held public hearings that were designed to bring out
issues in advance of beginning the serious work on developing the
standard, and I wasn\'t able to attend the other sub committees days,
but those hearings were really useful for us because we were able to
look ahead and say what are the issues we see coming up? And find
experts in that community to bring research to bring their work to sort
of put on the plate for us. I'm \-- jack, if I could finish, JR, if I
could finish. I look forward as what\'s on the human factors and it\'s a
full slate, on the other hand, the last slide that doctor Laskowski had
some stuff coming up, but there is work where perhaps what we need to be
doing is doing a public here where we do some in gathering of things,
not just from a specific disabilities community, but from around a
number of communities that are concerned about types of equipment, phone
voting is one that came up in our area, it was mentioned by commissioner
Davidson, we\'ve been thinking about tactile voting, tactile ballot
markers, not electronic ballot, because with some states rolling back to
paper, large number of people using paper, one question is can we help
people vote whose disabilities maybe of a type where that would be an
effective solution. And maybe that\'s something that we could in fact do
as the TGDC to bring some of that material into our thinking. We had a
team, we had a big mix of academic researchers, advocates, vendors.

I believ commissioner Davidson would like to comment.

JR, are you aware that we did in the human factors area for because of
the disabilities make a lot of them should \-- in the 2005 standards?

I knew we did a great number of them, I just didn\'t know how many, and
then what was the criteria for excluding the others. That is part of
where I was going.

I think maybe if you knew how many of them had been changed, because in
working with John and mark, and different ones, we did change those
before we adopted them in December so I wonder, before, some of the
discussion going on, maybe it would be important for to you see how many
of them had been changed.

I would be willing to withdraw the motion, Mr. Chairman, if we could
perhaps follow up with what a colleague was saying in terms of just
using the human factors and generally doing outreach to the broader
community \-- outreach, as we are playing with these different issues so
we are not getting hijacked on the back ends.

Whitney, this is Whitney. I know we did an analysis for the
subcommittee, and I can\'t remember whether it was verbal or did a
matrix of what the changes were. Is that something we could distribute
to the whole TGDC? I can\'t remember whether it was in a finished form.

We have a write-up, and John and I were trying to remember who we
distributed it to. I think it went to HSP only.

Maybe now to the TGDC.

Did you hear that, JR?

Yes, I did, and if we could get that distributed as well as the shoulds
and shalls.

It is.

That would be fantastic. And perhaps we could at least get back to the
community, and I would withdraw the motion then.

Thank you. Motion is withdrawn. Any other resolutions, motions?

I want to return to the issue of statewide voter registration systems
briefly. And say some remarks and propose a motion. The question I have
is whether it\'s within the scope of this committee or not, and if we
look at the language HAVA.

If that\'s an open question, can we get the opinion of council on it?

If you wish. My resolution was to clarify that by appealing tots EAC
itself. We have a lot on our plate, too. And however you want to resolve
this, is fine. I was proposing we seek it from the EAC directly.
Language, it says to support the development of voluntary voting system
guidelines, including, and part A, says including the computer networks,
computer data storages, and voting systems, including the list
requirement on section 303-A

Can we, I would rather get a ruling from EAC. Do we need the resolution
or can we \--

If you are just asking us, we are willing to have our counsel look at
it. We will follow up with that.

Any other motions? Not hearing any.

Meeting adjourned.

Before that, we need to decide on a date for the next full committee
meeting. I think proposed date is for early December, December time
frame, to review the progress of the work task assigned. I believe a \--
you\'ve been all provided a sheet and which provides you with two
choices, within the same have that sheet with your presences to here, or
by e-mail.

Couldn\'t we do a show of hands right now?

We don\'t have everybody here.

Well, we\'re only missing two people, and they are on the phone.

I think there was a suggestion that I and some others made to expand
these two at least 2 days.

I wants to make sure everybody agreed.

Why don\'t we have a show of hands now just to get a feel, but this
would not be a final decision and you that need to check your schedule,
\-- not a final decision, and check on major event, then we can \-- but
dates are are for 4-5, or 7-8.

7-8 is Thursday/Friday.

And we know how you feel about Friday.

Mr. Chairman, are we going to deal with a day and a half, two day versus
the one day first? Is that going to effect our decision?

My feeling is that a lot will be accomplished between now Andes, and we
will have a lots of material, and even though we may sends them to send
them to you, we will \-- I mean, I think today\'s proceedings is a
perfect example, you know. These are important issues, people want to
discuss them, and we certainly don\'t want to \-- I think we want to
listen to the concerns and issues. So my suggestion is \-- since we are
not doing these every quarter, if we are going to have a meeting in
December, I would suggest that we count on two full days if we finish a
little early, that\'s fine, but I think we should make the decision for
the dates with that level of commitment in mind.

I hate to be difficult, but may I suggest perhaps the fifth and sixth

Can\'t do it on the sixth. No room available on the sixth.

How about another location? How about in Atlanta or somewhere?

Matter of availability of the chair.

I guess the reason I suggested that is we would be traveling on Monday
rather than on the weekend, and we would not be trying to get out \--

Let me suggest something else. Ooh. It\'s going to require preparation.
Did you check the week before?

I did.

And Tom said the week before is out for the EAC.

Because the week before I guess is not good for EAC, and the week after,
we have our own visiting committee, so both the director and myself will
be tied up.

Mr. Chairman? I don\'t know about you other election officials, but our
canvassing board meets 30 days after the election as do most canvassing
boards, and election officials are tied up in the canvassing process,
you may not have any election officials here in those two days, I'm
pretty sure from my staff\'s comments that I'll be at my election board
meeting for two days.

Which two days?

Monday and Tuesday. Is that true?

Actually we would certify ours ten days after our election, so I would
be okay.

I mean, the other possibility is to go later and because the week before
\-- thanks give, and week before, that\'s closer to the elections, so
there is a possibility on 19 and 20th.

You get into the religious holidays.

That\'s what I was going to say. That\'s chose to the holidays. \--
close to the holidays.

I'm the only one with the problem it sounds like. I thought maybe all
election officials might, so I'll withdraw my comment about that date.

Okay. Let\'s see, the choices are fourth and fifth, and then \--

That\'s the one you choose.

Who is in favor of fourth and fifth? That is Monday and Tuesday. Only
one vote. How about you, JR, and Miss Turner Buie? Are you on the phone?

I'm here, but I didn\'t hear the month. I keep.

December 4 and 5, 2006.

I'm flexible. I'll go whatever the group wants.

I didn\'t hear your vote.

Fourth and fifth is fine with me.

And those who are in favor of seventh and Friday? Pi guess we have two
votes for that, so it looks like we\'ll be working towards fourth and
fifth and, you know, unless there\'s some other major issues, we will
probably go with that date, and those that can\'t be here perhaps, you
know, can call by, connect by phone. We\'ll, yes?

If we are planning for the fourth and fifth, would it be possible to
consider starting say at 1:00 on Monday to allows those on the east
coast to fly in?

I was thinking where we can start Tuesday, extending into Wednesday
morning, but the problem is, this is our awards ceremony, so this hall
will be decorated and everything, so I'm sure we won\'t be able to get
in Wednesday morning, but I think we can start Monday noon so you can
fly in \--

And maybe plan staying late Tuesday. That\'s a possibility. If you wants
to have a working session \-- we\'ll look at those, I think we have an
idea of how most people feel about this. And about Friday so \-- let me
close this session by expressing my appreciation for your participation
today. And we look forward to continuing our work with you. I want to
thank the NIST staff for their efforts to make this a successful session
and we will stay \--making this a success, and there are several
suggestions that will take into account regarding increased interaction
among the TGDC members and the subcommittee activities. So with that, I
thank all of you, and I adjourn this meeting. Thank you very much.

(MEETING ADJOURNED)