NARRATOR: This is Part 2 of the Security Testing Requirements Training
Modules for the next Voluntary Voting System Guidelines Document. Parts
1 and 2 are presented by Dr. Nelson Hastings of NIST's Information
Technology Laboratory. This final module reviews Open Ended
Vulnerability Testing Requirements in Part 3, Chapter 5 of the next
VVSG. The presentation includes question and answer sessions with the
Election Assistance Commission's Board of Advisors and Standards Board.

NELSON HASTINGS: Finally, we will talk a little bit about open-ended
vulnerability testing. What it is, what will test labs actually have to
do in relationship to performing open-ended vulnerability testing and
why is open-ended vulnerability testing been added and how does it help.

Open-ended vulnerability testing is an attempt to bypass the security of
the system and try to discover critical flaws. Critical flaws are
defined as being able to change the outcome of an election, interfere
with the voter's ability to cast ballots or have their votes counted or
just a general compromise of the security of the system.

OEVT is not, open-ended vulnerability testing is not a way to prove that
a system is secure. It is not bounded by a pre-determined test plan
because open-ended vulnerability testing is going to change given the
specific architecture of that specific system that's being tested.

The opened-end vulnerability test team will figure out how the system
works, identify vulnerabilities that are actual and potential
vulnerabilities in that system and they will attempt to break in using a
variety of different approaches.

An important thing to note is that the specific findings of the
different labs may differ. Test labs may test different aspects of the
system in a different order. The way the guidelines are written now is
once a lab finds a flaw, at that point, after finding that flaw, the can
stop testing at least in the open-ended vulnerability type testing.

There also has to be a consistent framework for discussing -- okay,
Matt.

MATT MASTERSON: Matt Masterson of the EAC. You say that labs can test
aspects of the system in different order. That doesn't mean, I assume,
that they are going to test systems differently so that the same
manufacturers can come back and say, my system was tested more
rigorously than another system was. Is that correct? I mean, there's not
going to be a differential in the intensity of testing or -- do you see
what I'm saying. That's a huge problem for us. If one manufacturer
complains that their system has been more stringently ---

NELSON HASTINGS: I understand. So the question from Matt is -- could you
say that again and I'll repeat it for the record.

MATT MASTERSON: The open nature of open-ended vulnerability testing,
whether its going to lead to an unfair level of testing for one
manufacturer to another.

NELSON HASTINGS: The question is, a manufacturer perceiving that
open-ended vulnerability testing has been conducted on them more
rigorously by one lab than another and done unfairly on their system
versus another system by the same lab or a different lab. I'll defer to
Barbara on this one.

BARBARA GUTTMAN: The goal of the OEVT, the way its being set up is there
will be a kind of a minimum, you know, that a certain competence level
that all of the testers have to have. That will ensure, there will be a
baseline of rigor. However, the nature of open-ended vulnerability
testing is that one person might see something different than somebody
else saw. So, if Don is testing system A and Larry is testing system B,
lets presume they have equal competence. They are both experienced
election administrators. They know what to look for when things go
wrong. They are actually different people and Don may see something with
system A that's different than what Larry sees with system B.

There will be a minimum level of rigor established, but that doesn't
mean the test will always be the same. Actually Don and Larry may
actually find the same thing, but Don might find problem A first and
Larry might find problem B first, just how it is they go about doing the
test. Even if we gave them all an endless amount of time they might
actually find the same things but that's too expensive. Donna, do you
want to add onto that?

DONNA JOHNSON: Donna Johnson from NIST. Part of the idea also behind the
open-ended vulnerability testing is to be able to look at the individual
solutions that are provided so system A may not even have the same issue
that system B has for it to be identified. So, you may actually look in
one direction for system A and a different direction for system B. I
think you have to keep that in mind as well.

DOUG LEWIS: Just because we spent ten years doing this I will say to you
this is going to be more difficult than you know at this moment in
keeping each of the labs that is approved as a testing laboratory on the
same page running the same kinds of tests in order to assure that you
don't have exactly the situation that was presented to you.

Labs interpret their latitude differently and their responsibilities
differently unless you specify exactly what it is they are going to be
doing. You are going to end up with the allegations of vendors that
their unit was tested more rigorously than that of their competitors and
you are not going to be able to defend yourselves very well until you
work out the details to make sure that each of the labs is doing the
same kind of test.

NELSON HASTINGS: I think that's a very good point that Doug has brought
up there.

There needs to be a consistent framework for discussing critical faults
that are found in the context of specific implementations and the
correspondence between plausible threat scenarios.

In section 5.4 - open-ended vulnerability testing covers the scope and
priorities of open-ended vulnerability testing, the resources and level
of effort to be done on open-ended vulnerability testing, rules of
engagement, failure criteria, reporting requirements and how test labs
respond to the results of open-ended vulnerability testing.

Open-ended vulnerability testing will encompass voting system
manufacturer supplied procedures. It will focus, again, on major flaws
and pass/fail testing will be based on security models as implemented to
address plausible threat scenarios.

The team resources and level of effort. The team will be made up of both
security and election management experts. A minimum of twelve staff
weeks should be put towards this effort. The team will be given a voting
device and its technical data package and its user documentation and the
test team will be given any other test data that has been created to use
as input into their testing.

MATT MASTERSON: Would you explain what twelve staff means because that
seems like a long time but, I don't know, having been on the calls, I
understand that its not actually twelve weeks of testing.

NELSON HASTINGS: Right. Right.

MATT MASTERSON: So, could you just explain that?

NELSON HASTINGS: Twelve staff weeks. The question is, can you explain
the twelve weeks of testing?

What does it mean for twelve staff weeks? In the guidelines I think it
says right now that on the testing team there should be three security
experts and at least one election management expert on the team. So
that's four people, okay. So, four people work for one week, that would
be four staff weeks. So, twelve staff weeks would be three weeks of
effort if you look at it from a linear standpoint. Does that answer your
question Matt?

MATT MASTERSON: Yes, thank you.

NELSON HASTINGS: Rules of engagement. The test team must examine the
system within the process model and plausible threats in which it was
designed to operate. They must also be given a description of the system
as it is to be implemented. The test team must be given a description of
how significant threats are addressed.

Failure criteria. Reasons why a laboratory would recommend a fail
include a direct violation of a guidelines requirement that isn't
covered and hasn't been caught, inadequate means to mitigate a
significant known threat, a threat that's known to exist but hasn't been
mitigated or a critical flaw is found. Again, critical flaw meaning
something that can change the outcome of an election, prevent somebody
from voting and that kind of thing.

Reporting requirements. The team must include in their report the threat
scenarios that they considered. They also need to identify the threat
scenarios that they identified but did not actually --

MALE SPEAKER 1: Threat scenarios or threat models, are they going to be
the same for everybody (undecipherable) threats would be consistent.

MALE SPEAKER 2: The same threats apply to all.

NELSON HASTINGS: The question is are the threat scenarios going to be
consistent across all the test labs, for all systems?

I believe that the answer is yes. What I think is here there will be a
baseline set of threats that everybody will start with and then from
there the test team may develop their own threat scenarios that they see
as potential threats to the system.

So, it could extend. Now, what do we do with those extensions? The
process says, well, maybe it would be good to get that fed back into the
baselines if the threats that this test lab finds that are beyond the
original baseline threats. There might be some feedback in this process
to extent the baseline threat scenarios.

FEMALE SPEAKER 1: The threats and vulnerabilities of an optical scan
system are very different from those for a DRE. Will someone be
providing a list of what to look for? What the risks are? One would
think that would be something that would change over time as the
sophistication of voting equipment and people who might want to harm it
would grow and change.

NELSON HASTINGS: The question is, is there going to be a list of
threats. I believe a baseline set of those is what could be there. More
questions?

RUSS RAGSDALE: Yes, Russ Ragsdale from Colorado. I think if I was with
the EAC, I 'd really be watching this area. Maybe if I could just step
back and take a more overall look, if I am a testing lab and I've
contracted with a manufacturer to test their system, by its nature
open-ended vulnerability testing can evolve as the testing takes place.
What's my motivation as a testing lab to pursue additional vulnerability
testing if I'm under contract, show me the money, I think is what I'm --

NELSON HASTINGS: Well there is the minimum threshold that they have to
do.

RUSS RAGSDALE: Okay, but that could lead to considerably more testing,
depending on what the minimum threshold showed, could it not? Just by
its nature. Where's the motivation for a testing lab to continue
open-ended testing or to pursue open-ended testing? I don't want to be
too mercenary about this, but it generally boils down to dollars.

NELSON HASTINGS: I'm going to defer to Barbara on this one.

BARBARA GUTTMAN: A lab's motivation to do a good job is, they have
several motivations, but one of them is they need to maintain their
accreditation and one of the advantages this system has over the
previous systems where we had periodic re-accreditation, we now have a
group of professional lab accreditors assessing the lab. If the lab is
found not to be competent, if they have not hired or contracted with
qualified election management experts, qualified security experts, they
will not maintain their accreditation. That is part of the baseline for
the system as a whole. You really have to think there's standards, their
tests and there's lab accreditation.

NELSON HASTINGS: Jim has a question. Hold on, Alan's coming over with
the mike.

JIM DICKSON: I just want to follow up the previous question. You spoke
to the laboratories being under scrutiny for accreditation but as I head
the question, what is the motivation for the lab to go above and beyond.
Could you explain that to me? If the answer is there isn't any, then we
need to know that too.

BARBARA GUTTMAN: If after a lab has done twelve weeks of qualified
testing and is ready to write a report and its clear form the report
that they really need to do another week to explore something that
should be clear from the report. If they have done twelve good weeks,
that's what we are asking for. That means the system is pretty good.

NELSON HASTINGS: Paul, do you have a question or comment.

PAUL MILLER: The testing labs will change for the time that they
actually spend testing. It's a cost basis, is that correct?

NELSON HASTINGS: Say again. I didn't get the first part.

PAUL MILLER: If they spent twelve weeks, they've charged for twelve
weeks.

NELSON HASTINGS: Right.

PAUL MILLER: If they spend thirty weeks, they charge for thirty weeks.
Is that correct?

NELSON HASTINGS: Right.

PAUL MILLER: So, what's, you know, I'm going to flip the question a
minute. What's the motivation for them to stop testing?

NELSON HASTINGS: Barbara do you want to pick that side of the question
up too?

BARBARA GUTTMAN: I'm looking around for Brian but I don't see him.

MALE SPEAKER 3: He's gone.

BARBARA GUTTMAN: He's gone. He's so smart. This is actually addressed as
part of the EAC's oversight of the program. I know this is something
Brian is looking at. It has to do with how the model will be set up or
how vendors negotiate costs with labs. So, the EAC is looking at this. I
don't know if any of the rest of you want to address this because it
depends a lot what your cost model is.

MALE SPEAKER 4: First of all the EAC right now can't get involved in a
cost model with the labs. We don't negotiate that. We are not involved
in any of the contracting between the manufacturers and the labs and we
are not allowed to be. So, we can't do that.

The answer to your questions honestly is we don't know. How are we going
to stop them from going on forever --

BARBARA GUTTMAN: The way it works now is that the vendor contracts with
the lab. If they write a contract that says I'm paying for twelve weeks
depending on how --

MALE SPEAKER 5: It the nature of open-ended vulnerability testing. It
doesn't allow for that kind of contract. That's the problem.

Open-ended vulnerability testing dictates that if they find more don't
they get to keep going?

BARBARA GUTTMAN: No. If they find more they have to stop. This model
says, once you have found a critical flaw, you stop. Stop wasting money.
This system needs to go back to the --

MALE SPEAKER 5: At what point do you stop looking for the critical flaw
is the question.

BARBARA GUTTMAN: Twelve weeks.

MALE SPEAKER 5: So they are going to max it out unless they find that
critical flaw.

BARBARA GUTTMAN: Probably.

MALE SPEAKER 5: Okay.

NELSON HASTINGS: Any more questions or discussion? Jim.

JIM DICKSON: It's Jim. One of the problems that has gone on in the
existing processes is it always takes much longer to do the
certification. You keep saying this is going to be done in twelve weeks.
Where's the hammer to enforce that twelve week completion time?

BARBARA GUTTMAN: The twelve weeks is not a calendar twelve weeks. That
is not addressed in the standard. That is part of the negotiation
between the vendor and the lab for, you know, a vendor may wish to pay
more to get higher in the cue for whatever other work the lab is doing.
These labs actually do more than just one thing or some of them do.
That's part of the negotiation. That's part of a market decision. This
justifies the minimum events and the minimum qualifications of people,
not actually anything about calendar time. That's a good question, Jim.

NELSON HASTINGS: Larry.

LARRY LOMAX: I have a question in just understanding how this would
relate to what the Secretary of State did in California and that study
she conducted or had conducted and called a study. There's assumptions
in that, for instance one of the assumptions that they state in that
study is there is no such thing as a tamper proof seal. They can break
into any tamper proof seal, most within two minutes and all of them
within half an hour and we would have no way of knowing that. If find
that hard to believe but they have criteria like that.

I assume there is some baseline here that would bring some rationality
into this study. They disregard all security efforts and things of that
nature on the part of the election officials.

BARBARA GUTTMAN: I always liked that story because in one of our other
testing programs at NIST the cryptographic module validation program
there is a similar thing about tamper evidence dealing with a
cryptographic module.

We had this huge fight within the lab program because one person found a
faster glue that would reset than the other lab found. You know what,
this was addressed.

At one point they had to, all the labs had to agree on how fast a drying
glue you could use as part of your test protocol. We will have similar
issues like that with this.

If you have enough time there is no such thing at tamper evidence. It's
a timing issue. These rules will need to be established and over time
normalized across the lab when one lab finds the better glue. I will
point out that the lab that found the really good glue is currently of
the STL.

MALE SPEAKER 6: On quick question. 5.4.3.a. Would I be right in saying
that this requires the testing to be done in the context that implements
the procedural security requirements provided by the documentation of
the manufacturer? I can't honestly tell by the language there but it
uses the term "specific implementation of voting system."

NELSON HASTINGS: If you could repeat --

MALE SPEAKER 7: Page 89.

MALE SPEAKER 6: 5.4.3.a talks about the context of testing. It says
"shall be conducted within the context of a process model describing a
specific implementation of the voting system." What does that mean? They
are talking about using the context -- I'm not sure I understand what
that means. I'm trying to get a better grasp of that.

NELSON HASTINGS: The example that's been given to me is that -- well,
how do I answer this?

I think what you said is correct. You can't just look at the system, you
have to look at the system and test it in this process including
everything, not just the system in a vacuum. You just don't look at the
system as its sitting out in the desert or something like that. You have
to have it in the context of which it's going to be used.

MALE SPEAKER 8: That means that all of the security features in place.

NELSON HASTINGS: Right, including procedural ones as well.

WENDY NOREN: I guess the way I read that is that the manufacturer's
documentation on how to use this, sets that system up and all the things
they say have to be done and then do the testing based on that criteria
using the assumption that everybody will follow those recommendations.

NELSON HASTINGS: I believe the documentation related to that, one of the
requirements of the documentation is to say if you deviate, if a person
deviates from that or a jurisdiction, what are the risks involved so
that other compensating controls could be placed. Brit.

BRIT WILLIAMS: I have two things. One has to do with what the team has
access to and there's a long list of things that they can have but it
says including but not limited to. Source code is not in that list, is
that covered under the but not limited to or will these teams not have
access to source code?

NELSON HASTINGS: There's a requirement that says the team has access to
the technical data package and user documentation.

BRIT WILLIAMS: Well, the source code is in the technical data package.

NELSON HASTINGS: Right.

BRIT WILLIAMS: Why, if you are operating in an election environment,
this is a black box test. Why would they have access to source codes?
You don't have access to source code in an election environment.

JOHN WACK: One quick example is for reporting. They do, if they come up
to an issue that essentially means it's a severe error, for reporting I
think we had discussed that it would be useful to have at hand the
source code to at least try to diagnose somewhat what's going on. I
think, Barbara you had something to add as well too.

BARBARA GUTTMAN: The OEVT team is not limited to black box testing. They
are open-ended so they can go back and forth between looking at the code
and looking how the system is really fielded.

BRIT WILLIAMS: That's what I suspected and I've got a real problem with
that.

JOHN WACK: Could you repeat that Brit?

BRIT WILLIAMS: I said I've got a real problem with that kind of
open-ended testing because if there is no restrictions at all on these
test team members and what they can do, depending on the personalities
on the test team this can get completely out of hand and the first thing
you know, you are reading about your system, the vulnerability of your
system in the New York Times.

NELSON HASTINGS: Let me answer the question a little bit differently
than John and Barbara had. My understanding for the rationale for
including the source code is to find possible soft spots in the software
that could then be shown to be used. You would still have to exploit, if
you look at the software and say, oh, look there's a buffer overflow
here, you still have to be able to exploit that in open-ended
vulnerability testing way, is my understanding.

Its not just that here's a buffer overflow.

JOHN WACK: And agreeing with what Nelson said, in some spots in the
document in the description field for some of the requirements it will
say, you know, testing this is fair game for OEVT.

One area, I can't recall the specific requirement but in the part on
casting where it was dealing with e-poll books and I think in general it
had to do with writing credential information to a smart card and then
the DRE or to a token I should say and then the DRE checking. You really
have to examine the source code to make sure that that's being done
correctly. That sort of requirement also in the description field said
this is fair game for OEVT. I think there are other instances too. I
can't recall specifically where that's called out.

NELSON HASTINGS: We'll go back to Brit here.

BRIT WILLIAMS: My other question has to do with the composition of the
team.

JOHN WACK: This is a question on the composition of the team?

BRIT WILLIAMS: Yes. Looking specifically at 5.4.2.c and 5.4.2.d which
describes the composition of the team, it says it will have to have one
member with six or more years of experience in the area of software
engineering. I know how you qualify that, their degree is in software
engineering.

At least one member with experience in the area of information security.
Their degree is with majors in information security.

At least one member with six or more years in the area of penetration
testing. I'm not quite sure how you qualify that.

At least one member with six or more years experience in the area of
voting systems security. How many of those people are there in the
United States?

NELSON HASTINGS: That has been brought up as an issue with that specific
requirement and that specific year limit on six years' experience. We
are looking at that to update that and revise that because --

BRIT WILLIAMS: How do you evaluate, you know, what's the criteria for
being an expert in voting system security? There are no degrees in that.
There's darn little experience in that. How are you going to evaluate
team members for that position? That's a rhetorical question.

My last one is on the eight years experience for an election management
person. Why does an election management person have to have two more
years experience than a computer scientist? There's am aroma to this
that kind of causes the hairs on the back of my neck to stand up and
I'll address it in public, when it comes up for public discussion.

FEMALE SPEAKER 1: I was going to say I'm sure NAVLAP (sic) is looking
for help qualifying people, so, your suggestions for how to make sure we
get good people on these teams would be very appreciated by the NAVLAP
people I know.

NELSON HASTINGS: We're getting to you Jim. Okay, Jim. Go ahead.

JIM DICKSON: Is there a usability expert on the test team?

NELSON HASTINGS: Because the scope of this testing is within the context
of security that there is not a usability expert required for this.

JIM DICKSON: In any of this testing are there requirements for usability
experts to look at the systems before they are certified at any point in
the process?

NELSON HASTINGS: Sharon.

SHARON LASKOWSKI: With respect to security, Jim?

JIM DICKSON: No, usability in the whole process.

SHARON LASKOWSKI: Yes, there's certainly usability experts will be
needed to do usability testing and there are some other, annexability
testing and also expert review to test some of the mother design
guidelines.

JIM DICKSON: Is there an equivalent place in the guidelines where it
lists make up of test teams, it says there needs to be a person with a
degree in usability?

SHARON LASKOWSKI: It's my understanding that the NAVLAP when they do
accreditation of the labs will be looking at the lab having the proper
people to run these tests. So, for the usability test they will need to
be accredited based on their having those experts. That's under the
management --. Its not specifically in the guidelines and I don't
believe there's a spot for doing something like that in the guidelines.

JIM DICKSON: We're talking about something the public has to use here.

JOHN CUGINI: Let me ask the question I think you were almost asking. I'm
going to put words into your mouth. Why isn't there open-ended usability
testing if there is open-ended vulnerability testing?

I think the answer or I think our answer is that we believe the
performance requirements that we outlined earlier that are based on
usability testing serve that role that we can come up with objective
metrics to address usability of the system as a whole. We think we
solved the kind of open-ended, you know, the kind of end-to-end
usability problem in a different way based on statistical techniques
rather than just having a usability expert going in and saying in my
humble opinion this system as a whole is usable or not.

That's just my opinion. That's probably not an official NIST opinion.

JIM DICKSON: That answer addresses usability for the voter. What about
usability for the volunteer polling judge? Is there any place in the
standards where that question is addressed in the certification process?

SHARON LASKOWSKI: Not in the certification process. In the VVSG there
are requirements for poll workers that include ease of use for the set
up, conduct at the poll, and shut down of the machines.

JIM DICKSON: But there is no testing.

SHARON LASKOWSKI: Yes, there will be a test method to verify that, yes.

NELSON HASTINGS: I'm going to go on here. Continuing in the reporting
requirements they need to discuss remaining vulnerabilities, the team
qualifications and each individual's level of effort.

MALE SPEAKER 8: Under the team resources it listed threat analysis,
describing threats mitigated by the voting system as being some that the
OEVT should include but not be limited to the following. This is
material supplied to the OEVT team. Who is creating this threat analysis
describing these threats mitigated by the voting system?

MALE SPEAKER 8: That'S requirement 5.4.2.a, OEVT team resources on page
87. Part III, chapter 5.

NELSON HASTINGS: Barbara is telling me she believes that a vendor

**(END OF AUDIOTAPE 1, SIDE A)**

\* \* \* \* \*

**(START OF AUDIOTAPE 1, SIDE B)**

BARBARA GUTTMAN: flipping through here to make sure that's the one that
you are addressing, but there is some material that will be provided to
all test labs that Alisha is working on even as we speak. There is some
material that vendors provide that address how threats are mitigated in
their system specifically. We need both.

MALE SPEAKER 9: Is the stuff that Alisha is working on currently a
threat matrix or a threat analysis?

FEMALE SPEAKER 3: A process (undecipherable).

MALE SPEAKER 9: Will we be allowed to see that?

FEMALE SPEAKER 3: Of course.

BARBARA GUTTMAN: They are not done. That's why you can't see them.

NELSON HASTINGS: So the laboratories will review the findings of the
open-ended vulnerability testing team in light of all other tests that
the lab has conducted as well. By adding open-ended vulnerability
testing labs may catch unanticipated design and implementation
vulnerabilities and may improve efficiency for testing of certain types
of requirements.

That's the end. Sandy had a question.

SANDY (?): Is there any requirement that the threats they are evaluating
have to be plausible ones? There are lots of threats that one can
imagine that might happen once in a million years and at what level is
there a decision made about, sure, this is a threat. We probably would
bet our pensions it will never happen. Who makes that decision?

NELSON HASTINGS: That's a good question on the plausibility of threats.
To answer your question, I don't know. I don't know who could do that
right now.

NARRATOR: Additional explanatory presentations on the Voluntary Voting
System Guidelines can be accessed from the web site vote.nist.gov.