VVSG Tutorial Narration[^1]

VVSG Overview Part 2

*\[Slide 1\]*

\[NARRATOR:\] The following presentation is Part Two of the next
Voluntary Voting System Guidelines Overview Training Module. There are
four parts to this VVSG training module. Each presentation reviews a
different section of the guidelines document in layman's language. The
overview includes Question and Answer sessions with members of the
Election Assistance Commission's Board of Advisors and Standards Board.

\[MR. WACK:\] Okay, we'll get started again. A couple of things: First I
want to just briefly reply a little bit more to Wendy's comment earlier,
a couple of things I wanted to point out.

*\[Slide 2\]*

I'm looking at page eight of part one, chapter one. You don't have to
turn to it because it's up on the screen, but it talks about
supplemental guidance and that basically says that there are some
sections, they're informative subsections in various spots, chapters six
and seven I think primarily, that basically talk about essentially some
procedures that might support requirements.

But we do not have any requirements for procedures, for election
official's procedures, but we do have some requirements that describe
assumptions for procedures.

*\[Slide 3\]*

The other thing I want to point out is that- it's in chapter two, Test
Reports, 6.1-0, Test Report, Compensatory Procedures. The test lab shall
list any specific election management practices that are required for
the voting system to satisfy the requirements of the VVSG.

And those obviously have to make their way into vendor documentation
that has to be available to users. This partially addresses it, but we
do not know of any specific requirements in the VVSG for procedures.

*\[Slide 4\]*

And earlier I basically said that the VVSG is basically a kind of
unified standard now. There is a foundation to the way requirements are
structured and implemented so that they can be updated and managed in
the future, and a lot of that is laid out for you in the conformance
clause.

*\[Slide 5\]*

As Mark said earlier, the word 'clause'- I actually asked Mark
yesterday, why is it called a conformance clause, and in some earlier
standards, it was essentially a clause.

A lot of chapter two actually goes into foundational aspects of the
VVSG, but there are requirements in there, a relatively small number of
requirements that actually do relate very much to how to show
conformance to the VVSG.

*\[Slide 6\]*

And I've just talked about why it's viewed as a foundational chapter,
and why it's very important to understand things in here.

*\[Slide 7\]*

Mark talked a little bit about language, but I'll repeat it here because
I think sometimes the words normative and informative aren't exactly
obvious, but in the online version of the PDF or if you happen to have a
color copy of it, you'll see that the requirements are in blue and that
is further to differentiate that they are normative. These are the
requirements. This is the language that you must conform to if you're a
vendor. This is the language that if you're a test lab, you must test
to.

There are other parts that are called informative, they inform you. They
don't necessarily lay out exactly what you must conform to, but they're
informative, they support the normative language.

There aren't really exceptions, but I will say there's a little bit of
fuzziness sometimes. I think, in some cases, there may be some tables
that aren't specifically in blue. They aren't part of a requirement
subtext, but a requirement may say you shall implement certain things
according to the settings in table three. So what is in table three is
now normative, that's required. You've got to do what's in table three.

There is a description field, we'll go into that, a description field in
each requirement, and its purpose is to add examples to better inform
you as to the requirement. It is not per se normative. The normative
language is in the requirement subtext in the color blue, but we
certainly wrote it expecting that test labs and vendors would certainly
use it.

*\[Slide 8\]*

I would like to go into the requirement structure and talk a little bit
about this, parent sub-requirements especially.

*\[Slide 9\]*

So at the risk of repeating myself a little bit here, the requirements
have a bunch of different parts to them, and you don't have to remember
the names or anything but something different from previous guidelines,
there is a title. The title was put there basically suggested by the
usability experts we consulted, because it would be a shorthand way of
referring to the requirement in building tables or databases, and we do
intend to build a database of the requirements.

So a shorthand way of referring to the requirement- the requirement
subtext is actually what's in blue. We've talked about that. The
'applies to' field basically says which voting device in essence, which
device class does this requirement apply to and is it normative? Well
yes, really it is. You know if it says this applies to vote capture
device, it applies to vote capture device and that's what it applies to.
So you could say that's normative.

'Test reference' refers to what test approach that's described in part
three of the document, what test approach will be used for this
requirement. Is it normative? Not really. There could be other test
approaches that could be used for the same requirement, but it is what
we believe to be the test approach that will be used for that
requirement.

'Discussion field', optional: You won't see these all the time, and
sources. You know, basically a lot of times, the source has VVSG 2005
and goes back to a specific part. We, as best as we could, tried to link
requirements back to where they came from, and that also adds some
understanding of the requirement.

Will 'source' be there in the final version? That's up to the EAC. We
intended it to be there primarily as helpful to people in reviewing the
document.

*\[Slide 10\]*

Which fields are used where? Well, the document used to be- I think it
was at one point like 830 pages, and we tried to decide what can we do
to reduce the size of the document and make it a little easier to read,
and in some cases it didn't- you know, it was a better idea to get rid
of some of the requirements fields if they could easily be assumed, so
some of those fields are optional.

In the equipment requirements, we don't always have a source. I think a
number of the HFP requirements don't have sources. A number of them
don't have description fields, but they always have 'test reference'
fields and 'applies to' fields.

In the document requirements, we don't use 'test reference' because it's
documentation. You can always assume the testing approach is always
going to be a documentation and design review. That's always going to be
used to test documentation requirements, so we just basically didn't
repeat it all the time.

And in testing related requirements that are in part three, the 'test
reference' is really implied by the requirement itself, where it is in
the chapter, so that was another field that we didn't have to repeat all
the time, and that actually getting rid of those actually reduced the
size of the document a good bit.

*\[Slide 11\]*

'Applies to' field- it's almost always a device class. David is going to
go much more into the distinction between device classes and system
classes, but essentially a device class, I think in the next slide I'll
go into that a little bit more, but it's essentially a device
specification in a sense. There are requirements that apply to certain
types of devices, be they specific devices or general devices, and you
can have a sub-requirement that can basically narrow the scope of a
parent requirement in a sense.

Let's say if you have a top-level requirement, well here I have an
example- tabulator, in other words, a requirement that says all
tabulators shall do this. You can have sub-requirements that further
narrow the scope of the parent and say, well specifically, all precinct
count optical scanners shall do this. So basically that's how that
works.

*\[Slide 12\]*

Parent and sub-requirements, what I alluded to, a number of them are
just single stand-alone requirements, and some actually have
sub-requirements associated with them, and you can easily see how
they're numbered in the document.

They are there for a number of reasons. One of them Mark talked a little
bit about. We tried as best as we could not to do this, but in some
cases, it just was a better idea to do it in terms of readability, to
list the top-level requirement, the parent as a goal requirement and
state kind of generally what we want, because one thing about a goal
requirement is that it can fairly well describe the purpose of the
requirement in a very understandable way, and then you can have
sub-requirements underneath that can get into a lot more specifics about
the requirement. And again, I talked about narrowing the scope of the
parent.

*\[Slide 13\]*

Basically Sandy asked can we go over a couple of examples of this, and
that's not a problem at all.

I will pick on the usability section, because that's just what popped up
there. Okay, let's look at this. This is from part one, chapter three,
the Security, Privacy Accessibility chapter, and I'm looking on your PDF
on page 30 of chapter three. I'm looking down at the bottom 3.2.1.1-D.

And basically there is a parent that's basically saying, report the
metrics for usability of the voting system as measured by a voting
performance protocol.

And so it says okay, you know, report these things, but what are they
specifically? The children go in and actually describe specifically what
specific metrics need to be reported, and in addition, I didn't note
this, but in addition, the children can also have discussion fields,
which go a long way towards making things more understandable. So that's
a simple example there.

If this requirement did not have any children, someone could say, well,
that's not very specific. You know.

*\[Slide 14\]*

\[QUESTIONER:\] So 3.2.1.1-D is the parent and then D.1, D.2.

\[MR. WACK:\] D.2 and D.3 are the children, that's right.

*\[Slide 15\]*

\[QUESTIONER:\] Okay. There are a couple of times- the parent was a
'shall' and the children were all 'shoulds.' And so I guess I would have
to go back through and look at them. And I wondered about the ability to
come up with a 'shall' when the details are 'shoulds.'

\[MR. WACK:\] Any of the NIST people have an example- John?

\[MR. CUGINI:\] An example would be a little beyond this, 3.2.4-C, and
the general requirement is a plain language requirement, and it says,
pretty simple, instructional material for the voter shall conform to
norms and best practices for plain language. So that's pretty general,
you know, what are these so-called norms and best practices and so on.

C.1, C.2, C.3 have the sub-requirements, here's how we want you to use
plain language. Some of those are 'should,' for instance, simple
vocabulary, the system should use common words, I think use of the
passive voice, so some of these are mandatory, but some of these are
guidelines where we felt you couldn't absolutely require that every
single sentence be in the active voice, for instance.

\[MR. SKALL:\] But if you have a 'shall' requirement, something 'shall'
be implemented- something underneath that needs to be implemented:
either by a 'shall' or by a preference among the 'shoulds,' I would
think.

\[MR. WACK:\] And the 'shoulds' are pretty strong guidance in a way to
the vendors.

\[QUESTIONER:\] Basically, if a 'shall' parent is specified and there
are 'should' children, the 'shoulds' are very specific clear guidance on
how the 'shall' shall be met but not necessarily.

*\[Slide 16\]*

Okay, you're looking at just down another page or two, 324-E, Ballot
Design. Voting systems shall provide the capability to design a ballot
with a high level of clarity and comprehensibility.

And then you have a number of 'should nots,' so don't do it this way,
but it shall do something in a certain way, it shall clearly indicate
the maximum number of candidates. Shall be consistent throughout the
ballot.

\[MR. SKALL:\] I think what we're saying is, the low-level 'shalls,'
when there's a high 'shall,' are necessary. You have to do them. That
doesn't necessarily mean the high-level 'shall' is met. The test lab
still has to look at a high-level 'shall' and make sure it is met, so
the low-level 'shalls' are necessary but not sufficient conditions to
satisfy the high level.

\[QUESTIONER:\] Is that the criteria for meeting the parent? I think is
where I'm going with it because there are a couple where there would be
five or six 'shoulds,' one 'shall,' and so what I don't want to see is
the vendor meeting one 'shall' on the child, and that being the criteria
to meet the parent.

\[MR. SKALL:\] I think we are saying it is not. Meeting the 'shall'
doesn't necessarily mean they've fulfilled the whole requirement for
meeting the parent.

\[QUESTIONER:\] If you follow the logic of the answer we just heard, if
you meet that low-level 'shall,' but we don't have anywhere specified
what's required to achieve the high-level 'shall.' We've got these
'should nots,' or 'shall nots' and 'shoulds,' and let's say it meets the
'shall not,' it doesn't do that. It meets the low-level 'shall,' at what
point does it meet, where in the standards here, for instance, have you
indicated it will meet this by achieving X?

\[MR. WACK:\] They are goal requirements, and somebody looking at this
requirement and testing for it is going to have to essentially be
somewhat of a- you know, use expert judgment to a certain extent.

And, for example, if you have the 'shall' up here in 3.2.4-E, the voting
system shall provide the capability, but then clearly down here in E-1
the voting system does visually present single contests spread over two
pages or two columns, I think in the eyes of an expert tester there,
that would be a good indication that the vendor hasn't met the spirit of
the requirement.

\[QUESTIONER:\] Assume you meet the low-level 'shall,' and they also
meet the requirements of a low-level 'shall not.' So you've met these
two independent low levels but there are some 'shoulds.' Where do you
specify which of those children have to be met in order to meet the
parent requirement?

\[MR. WACK:\] In my opinion, again you've got somebody doing the testing
who's an expert in this area and, to a certain extent, they're going to
use their expert judgment in coming up with whether they think the
vendor properly implemented it.

*\[Slide 17\]*

\[QUESTIONER:\] John, I am not sure that we are at the point yet that we
truly have laboratories that are indeed experts on this, and so I would
suggest that we're all sort of feeling our way along in this, and to
suggest that the labs are somehow experts in this, I won't tell you how
long it took us.

I know when NASED ran it, it took us a very long time to continuously
learn things that you didn't know before and to make adjustments as you
went along.

\[MR. WACK:\] Good point. And Brian.

\[QUESTIONER:\] Essentially a follow-up on that: Since you all are
developing or will be developing the test suites, and maybe this is an
unfair question, but how would you write a test suite to tell the lab
how to test to this?

\[MR. WACK:\] John Cugini, you want to-

\[MR. CUGINI:\] Yeah, let me give you my understanding, which other
NISTers may disagree with.

These goal-type requirements like E, 3.2.4-E, are in fact goal-type
requirements, and there is really no test method involving rulers and
oscilloscopes that's going to test that. It really is dependent on
expert judgment, these higher-level requirements.

We're aware that that implies the need for expertise, so when we've been
talking to the test labs, one of the things I think we're going to look
at are requirements for the expert, and in our case, what is the
usability expert, what do you need to be a usability expert such that we
can trust you to evaluate a requirement like this? And we certainly have
in mind putting out certain credentials, you know, not just someone
walks in the room and says, I'm a usability expert. I can tell you
whether this is clear or not. So ultimately we're going to be leaning on
the credentials of the expert. You need that, because you need expert
judgment. You need expert judgment because we think you need sometimes
goal-type requirements. I'd love it if everything was more objective
than that, but right now that's kind of the state of the art, I think.

\[MR. SKALL:\] I think the question here was a little bit more specific.
He specifically asked if the 'shalls' are satisfied and the 'shall nots'
are satisfied- so he's basically saying we have a high-level goal
requirement. Everything underneath it has been satisfied.

It seems to me, and we need to discuss this further, if everything
underneath a high-level requirement has been satisfied, that should be
sufficient, right, as opposed to some things, in which case you would
need selective, you know, sort of, so that's an issue I think for us all
to discuss.

If everything underneath the high-level requirement is satisfied, by
definition does that mean the high-level requirement is satisfied?

Now those of you who were at a TGDC meeting a while back may remember
there was a motion that was approved, and I believe it's somewhere in
there, saying even if all the requirements are satisfied, but the VVSG
implementation in the system does not meet some higher-level goals such
as -- reliability- you could still fail them.

So that would basically support your opinion, but it's so vague that I
think this is an area on which we clearly need more discussion.

\[QUESTIONER:\] Brian, and others here, if this is going to be truly
independent expert opinion in how to do this, and you've got separate
labs, and we ran into this problem Brian, if you will remember before,
that some of the labs used their expert opinion but came out with very
different plans and tests results as a result of it.

You end up with then one lab that fails someone who another lab would
have approved, and so this has got to be tied down a whole lot better
than that unless you're actually ordering the labs to share their test
plans with each other so that they're all doing the same thing.

\[MR. SKALL:\] I think that's a very good point. That clearly leads to
inconsistency among results and could lead to forum shopping or just
accusations of unfairness, and that's not good.

So I think by definition when you have these high-level goal
requirements, which require expert judgment, there is absolutely no
assurance that expert judgment will be uniform, so something either
needs to be done about that or that's going to be the result.

\[MR. WACK:\] There were a couple of things changing at the same time
right now, and I think actually Sharon and Whitney Quesenbery on the
TGDC did work with NVLAP, recognizing that the basic core competencies
of labs needed to include expertise in usability and expertise in
usability testing.

So through adding perhaps to the NVLAP process as well as the
capabilities of labs in certain areas where they may be lacking right
now, we would expect the test labs down the road will be able to do
things that they can't do today and should be able to do a better job in
terms of expert judgment.

The other thing is that the TGDC decided to go this route with these
specific sorts of requirements, but again, this is the beta version, and
it's good we're talking about this because you subsequently have to make
comments, and I think the final version of this may not come out until,
looking at the EAC, approximately even 2009, early 2009, so we have some
time to debate the merits of these, and these are very good points that
are being raised. And Brit, please.

\[QUESTIONER:\] On one hand, we talk about how you've got in the
conformance statement where you've got to conform to all of these
requirements, but now we're saying, well sometimes, I guess, we'll let
all the experts in the room have their own opinions.

If you've got a parent with a set of 'should' subs under it, that parent
has got to address those subs and say you can meet this requirement by
doing three of the following five, or two of the following five, or you
can meet it by doing number one and then any one of three, four, five,
but the parent has got to specify. You know, you can't just throw this
grab bag out there and say here's a requirement, and by the way, mess
with these things. You've got to say what those subs are and how they're
to be put together in order to satisfy the requirement, or otherwise
we're back to the situation Doug was describing with the test labs.

\[MR. WACK:\] Sharon, let me ask you a question since we're talking
about usability. Maybe some things in perhaps Security or Core are more
cut-and-dried in areas of usability and perhaps- do you think that the
TGDC decided in some areas that it wasn't possible to say you shall do
this in terms of usability but you should because we think that's what-

\[MS. LASKOWSKI:\] No, I think in the case of say the plain language,
sometimes you need to violate a rule in order to make things clearer. So
I think those cases usually are cut-and-dried, so I would say that's
typically not the case. Things are pretty cut-and-dried, and you will
get good expert agreement.

\[MR. CUGINI:\] Yeah, I just don't want to leave the impression that the
usability guys are the only guys with parents and sub-requirements.
There are plenty throughout the other sections, so we can find other
examples and pick on other areas if possible within the voting system
architecture, the voting device shall allow the administrator group to
specify password strength. You know, I mean there's lots of play in the
joints in other places as well.

\[MR. WACK:\] But may I ask you, there's specific reasoning you used for
the use of color by the voting systems should agree with common
conventions instead of 'shall,' and you also have the voting performance
protocol, and does that tie into- does the usability performance testing
tie into some of the uses of 'shoulds' here? Why is there a 'should'
there instead of a 'shall'?

\[MR. CUGINI:\] No, I would say the performance protocol is- we're proud
of that because that is a high-level goal-type requirement, and it's
very objective, and you don't need expert judgment.

That's why we think highly of performance requirements because it's not
just somebody's opinion. You're actually running statistical tests. So I
would take that as, you know, a counter example to these other goal-type
requirements. Requirement X is clean and consistent and so on. Yes,
Wendy.

\[QUESTIONER:\] Not to be terribly- but I can understand where, in some
cases, it may have come up that it wanted to be a 'shall,' but for
example, say there are certain state laws that require names in all caps
or something like that, and so you couldn't make it a 'shall' and put --
see, I am intuiting what you all were doing.

But there are- because I know my own state law requires names in all
caps, which is stupid when usability-wise, it shouldn't be all caps, so
that may be a reason to make it a 'should,' but I don't know.

\[MR. WACK:\] I guess the thing I was getting at, and my question to
John and Sharon was, the use of color by the voting system should agree
with common conventions. I believe you're saying it should do this, but
there may be other conventions that work as well, and we don't want to
restrict things to just these common conventions. I think that's another
way of looking at it.

\[QUESTIONER:\] Well, a couple of observations. One is that it appears
to me that some of the 'shoulds' that we're talking about, really we're
expecting them to be 'shalls,' except that there are occasions in which
the rule needs to be broken, and I'm wondering if that isn't a helpful
way to look at some of those.

There are some of the 'shoulds' in here that I picked up right away here
in this usability section that probably shouldn't be in the VVSG at all,
and let me give you one concrete example of that.

The comment about that the contest should not be in more than one column
or more than one page. An excellent requirement that should always be
what we strive for, but if you've got 60 candidates like the California
race did, that's a requirement that can't be met by any system. And so I
think we need to look at that requirement in terms of how it really is a
requirement of the voting system or is it a recommended best practice
for election administrators?

\[MR. SKALL:\] There is a parallel situation in another domain we're
working in. In the healthcare arena, we're writing standards, and there
are many of these same caveats, state law, legislation, and we have,
believe it or not, you'd love to hear this, John, something called
conditional 'shall.'

And basically there are three conditions enumerated. One is conflict
with state law. I forget what the other two are, but they are
administrative things like that, so essentially you must conform unless
one of these conditions goes into effect. And so something like this may
be an approach, but that's something we can certainly take a look at.

*\[Slide 18\]*

\[NARRATOR:\] Additional explanatory presentations on the Voluntary
Voting System Guidelines can be accessed from the Web site:
vote.nist.gov.

[^1]: ^\*^ Certain commercial entities, equipment, or materials may be
    identified in this presentation in order to describe an experimental
    procedure or concept adequately. Such identification is not intended
    to imply recommendation or endorsement by the National Institute of
    Standards and Technology, nor is it intended to imply that the
    entities, materials, or equipment are necessarily the best available
    for the purpose.