VVSG Tutorial Narration[^1]

Standards 101

*\[Slide 1\]*

\[MR. SKALL:\] Welcome, I'm Mark Skall, and I guess for the first few
minutes, I'll just talk about what we're going to be doing today.

The EAC called us, I guess, a few months ago and asked us to do this
training for voting officials, which we agree is very important. We know
that you all need to really understand the VVSG. So we will tell you
everything we know, but I'm going to talk about some of my experience in
standards, developing standards, developing conformance, and relate it
to the VVSG. So I've entitled it Standards 101.

*\[Slide 2\]*

So the objectives of this session are to establish a common
understanding of concepts and terminology. We're going to talk about
standards, requirements, and conformance. We're going to talk about
motivation for the new VVSG and set the stage for the rest of the
presentations on the VVSG.

*\[Slide 3\]*

So as I'm sure you know, we've been focusing our work over the last few
years in three areas: core requirements, security, and human factors.

And as I speak during this presentation, there are some words I'm going
to use synonymously. The term 'implementation' is kind of a standards
geek word. It basically is, you have a standard and then that standard
gets turned into a product. That product, in standards, we call an
implementation. It's often times synonymous with the term 'system,' and
in the VVSG, that's what we call a voting system. So that's equivalent,
implementation, system, and voting system.

The person or persons who develop that implementation are called
implementers or developers, and what we call them in the VSSG are
manufacturers.

Now the last line, standards specification and guideline, really are not
synonymous. A specification is a detailed enumeration of requirements.
When that specification becomes publicly available and shared among
various constituencies and blessed by a standards-making body, it's
usually called a standard.

A guideline is a little bit different. A guideline is typically a
document that has recommendations, not requirements. It's kind of a
softer document.

The VVSG, Voluntary Voting System Guidelines, has the term G, guideline.
In all shape and form, however, that is a standard, that's a
specification. So sometimes I use standard, specification, or guideline
to talk about that particular document, but really it is a standard, and
I'll get into that in a few more minutes.

*\[Slide 4\]*

So the outline is what is a standard, what does it mean to conform to
standards, and what's the difference between conformance and
certification. I'll talk a little bit about conformance testing, and
then summarize and end with the improvements that we've made to the
previous sets of standards.

*\[Slide 5\]*

So what is a standard? Again, VVSG, Voluntary Voting System Guidelines,
voluntary means the use is not mandated by law or regulation. If you
decide to claim conformance, however, you must adhere to its
requirements. So the fact it is voluntary is one thing, but if you
conform to it, you are obliged to meet all the requirements.

A standard is established by consensus or authority and prescribes
technical requirements to be fulfilled by a product, process, or
service, and a requirement is criteria, characteristic, behavior, or
functionality that a system must do or have.

*\[Slide 6\]*

So good standards are really a key because what we really want are good
implementations, good systems, and the only way to get that is to move
from detailed requirements in a specification or a standard into the
actual product.

So the goal is correct, reliable software or correct reliable systems,
and the requirements are captured in the standard, and again those
requirements need to be translated into the voting system.

So in order for that to happen, that standard needs to be clear, it
needs to be precise, it needs to be unambiguous, it needs to be
complete, and it needs to be testable.

Now those of us who communicate through the English language understand
how inexact the English language can be. Oftentimes I say one thing, and
it's interpreted one way. Someone interprets it a certain way, another
person interprets it a third way.

We need these specifications to be precise. We need to have no failure
of communication. In an ideal world, that specification wouldn't even be
written in English. It would be written in some fancy mathematical
notation.

And I know there's been concern about the readability of the VVSG. The
other alternative, and this happens in many other forums, is you
actually get a specification in pure mathematical notation. That hasn't
always worked because it's not very readable, but there are actually
standards written in what we call a formal specification or math.

Oftentimes people designing mission-critical systems like aerospace,
airplanes, will develop specifications and formal specifications. Of
course here we're doing it in English, but again that English has to be
precise and we also know it has to be readable.

*\[Slide 7\]*

So just an example about imprecision in English, the sentence, "The girl
touched the cat with a feather." So I could mean the girl has a feather
touching the cat, or some other people may interpret it as the girl is
touching the cat, which has the feather.

And these examples have nothing whatsoever to do with specifications,
but they abound in the English language, so it is tricky and you have to
be very careful when you use words in English to define a very, very
detailed specification.

*\[Slide 8\]*

So what makes a good standard? One that gets used, used correctly,
implemented consistently. One that defines what and who needs to
implement the standard, and in this case, it's voting systems and test
labs. VSTLs are the people that need to implement the standard.

A good standard distinguishes between normative and informative text.
Normative is necessary information that you need in order to conform to
the specification; informative is everything else.

So for instance in the VVSG, there are requirement sections labeled as
requirements, and then there are discussion fields. Discussion is
informative. Everywhere that there is a requirement, that says
requirements, and in our conformance clause, it distinguishes and tells
you what is normative, what is informative.

A good standard tells you what needs to be implemented, what is
mandatory and what is optional, and typically the way this is done is
through key words, and that's what we use in the VVSG.

There are three primary key words. The key words are in caps in the
VVSG.

So "shall" is mandatory. If there is a "shall" requirement, it has to be
implemented. "Should" means the requirement is optional, but it's
recommended. "May" means it's optional and permitted; there's no
recommendation one way or the other.

A good standard is modular. So, for instance, we have different sections
talking about different parts of the VVSG. You don't want to mix things
together.

A good standard is adaptable as things change. We do not want standards
changing every year or two. A moving target is no good for implementers;
it's no good for users. It's no good for voting officials, so we want a
standard that's adaptable.

*\[Slide 9\]*

One of the ways we've done this in the VVSG is through the innovation
class, so that new technology can be built into the standard without
starting all over and developing a new standard, and one that is
technology- and design-independent.

Now technology-independent means that requirements are not tied to a
specific technology. So, for instance, we don't require that voting
systems be built on any specific platform.

Design-independent means requirements tell developers what to build, not
how to build it, just like a good manager tells their underlings what to
do, not how to do it. Same idea with design-independent, you would
rather tell people what to do, allow innovation, than tell them how to
do it.

*\[Slide 10\]*

So there are three types of requirements in general in a standard. There
are functional requirements specifying that the object is capable of
performing a certain action, and you see an example of that.

Performance requirements specify not only that the object is capable of
performing the action but also set a benchmark for how well it performs,
so typically you must meet or exceed that benchmark.

And a design standard specifies something about the static structure of
the object. For instance, any control buttons on the voting system must
be at least one inch apart.

Now that sort of violates the design independence principle that I just
mentioned. So, in general, we try to concentrate on functional
performance requirements. We do have a few design requirements in the
VVSG, primarily in usability, but the intent is to do as little of that
as possible, and Sharon and John, I'm sure, will talk about that.

*\[Slide 11\]*

So are standards enough? And clearly, clearly they're not enough.
Standards are a piece of paper. When this VVSG is finished, it's a piece
of paper, and if other stuff doesn't happen, then you're just left with
a piece of paper.

So I'm here, I work for NIST, and it's a big S for "Standards," but I'm
telling you standards are worthless unless they're implemented. Again,
it's just a piece of paper. If you don't have a product, you don't have
a voting system. They're worthless and they're useless unless they're
implemented correctly.

So how do you get it to be implemented correctly, and that's where
conformance and testing come in.

*\[Slide 12\]*

So this next diagram shows a triangle with a three-legged stool, is the
way that I like to describe this, where you have the standard or the
specification, again I'm using those terms synonymously, at one point of
the triangle, the voting system which is the thing that needs to be
developed from the specification at another point, and conformance tests
at another point of the triangle.

So you then run the conformance test against the voting system that
purports to conform to this specification at the top, and unfortunately
you only have two possible things that can happen, you either find an
error somewhere, you find an error in one of the functional
requirements, you find an error in the security point, usability,
accessibility. You find at least one error, and you know for sure that
that voting system does not conform to that standard.

The other choice is you find no errors. You go through the whole testing
regimen, and you find absolutely no errors.

Unfortunately, you really don't know anything then. You know either that
the system conforms perfectly, and it does all the requirements exactly
the way they're supposed to, or perhaps your tests were not
comprehensive enough to find the error. Testing can never prove an
implementation is correct. That's the state of the art in computer
science.

*\[Slide 13\]*

Okay, I'm going to talk about a conformance clause. All good standards
should have a conformance clause, and by the way, the term 'clause' is
really misleading because our conformance clause is about what, 50
pages.

Now some conformance clauses are very small; I'll get to that in a
second. Others, unfortunately, like ours, are long.

So conformance clauses address what needs to conform, how to conform and
claim conformance. Typically they talk about subdivision of a standard,
so if a standard is divided into modules, some are divided into nested
levels, a conformance clause will describe how these standards are
subdivided.

Conformance clause talks about different ways a specification is allowed
to conform differently. So everything is not monolithic. You don't have
one way to conform in voting, you have many different systems, DREs, op
scans, others. So there are many different ways you can conform
obviously to the VVSG.

*\[Slide 14\]*

I'm going to define some terminology. This typically comes out of ISO
directives and other documents that we found.

Conformance again is the fulfillment of a product, process, or service
of specified requirements. These requirements are defined in the
conformance clause or elsewhere in the specification.

The reason some conformance clauses are short and some are long is that
a conformance clause could be as short as talking about the subdivision
of specification and then saying the requirements are defined elsewhere
in the specification. It's a pointer to all the other requirements. So
conformance clauses get into requirements in detail.

Conformance clauses are a section of the specification that states all
the requirements, all criteria that must be satisfied to claim
conformance.

*\[Slide 15\]*

Conformance testing is a way to determine directly or indirectly that
relevant requirements are fulfilled, and even without certification,
even without processes, conformance testing is always useful.

You'll have a standard on the marketplace, and you will see invariably
people say they conform to this standard. It doesn't matter what domain
we're talking about.

Conformance tests help buyers increase their confidence that something
actually conforms. If you pass tests, you don't know for sure it
conforms, but it increased your level of confidence. And sellers help
substantiate their claims if they've been tested.

Typically conformance tests are performed by test labs to determine if
the voting system conforms to the VVSG.

*\[Slide 16\]*

Conformity assessment is a process necessary to perform conformance
testing in accordance with prescribed procedures and a test suite. It
ensures that testing can be repeatable and reproducible.

And certification is acknowledgment, typically by a certification body
or certification authority, in this case the EAC, that a conformity
assessment was completed, and that the criteria established for issuing
certificates were met.

So typically the EAC will look at a test report, but they'll have
additional criteria as well which certification typically has. They may
be looking at conflicts of interest and other things.

So certification usually looks at first conformity assessment, looks at
the outcomes of testing, and then has additional criteria as well.

*\[Slide 17\]*

One of the things I've just talked about in conformance testing, I sort
of want to reemphasize. It may seem obvious, but one can only test for
requirements in the standard. That's what conformance testing is.

You have a set of requirements. You can't say we'd like to add this
other requirement and test for it. I mean you can, you can test for it,
but that's not conformance testing. You can't fail an implementation for
failure to conform to a standard unless that requirement is in the
standard.

The analogous case is you can only test for the requirements in the
standards as they are stated, and if I had a dime for every time someone
came to me and said, I wrote a requirement to read as it does in A, but
I really mean B.

Well, it doesn't matter what you meant. What it says on the piece of
paper is what you test for. If there's a problem, if there's an
inconsistency, you need to go back and change the standard. You can't
test for what you meant; you can only test for what the standard
actually says.

*\[Slide 18\]*

Conformance testing is also in a way in the VVSG. The VVSG includes
testing requirements for test labs. It not only has requirements for
voting systems, it also has requirements for test labs.

The VVSG indicates general testing procedures. We have a list of sort of
generic testing procedures in the VVSG like inspection, functional
testing, vulnerability testing. Every time we have a requirement, we
talk about the testing procedure that should be used to test that
requirement. That's in the VVSG.

There are also documentation requirements in the VVSG, not only for
voting systems but also for test labs as well. So test labs must have
test plans, pretesting, they must have test reports, so there are
requirements in the VVSG for how tests labs do conformance testing.

There are also procedural requirements in the VVSG, various procedures
for hardware setup, OEVT, open-ended vulnerability testing has
procedures. So there are requirements in the VVSG for all of these.

The VVSG, however, does not contain the actual test suite. Right now the
actual tests are developed by test labs. They've been developed by test
labs throughout the history of standards in voting.

The problem with that is that although those tests may be very good, we
don't get to see them because they're proprietary, they're trade secrets
if you will of the test labs, so no one can look at them to see how good
they are, and perhaps just as importantly, we know they're not uniform,
because different test labs develop their own test suites, and it would
be very unlikely for those to be exactly identical.

So the problem is if you get tested by one test lab, you're being tested
by a different test suite. What we are doing at NIST is developing a
comprehensive test suite, we are releasing it incrementally over the
next two or three years, and this test suite will be available to all
the test labs.

All the test labs, we believe, will work with our procedures, will try
to ensure that they use that same set of tests, so that everyone will
know it's a comprehensive test suite that tests for the new VVSG, and
that test suite will be consistent and uniformly used by all test labs.

*\[Slide 19\]*

Okay, now this is a really interesting diagram, and it talks about the
things I've just discussed as a series of building blocks, and if you
look at the bottom, there's the standard. After that, you get
conformance testing, then conformity assessment, then certification.

The interesting thing is everything beneath is a prerequisite to do
things above, so unless you have a standard, you can't do conformance
testing. Unless you have the standard and conformance testing, you can't
have conformity assessment. Unless you have all three, you can't have
certification.

The other interesting point about this sort of set of building blocks
is, in the standards world, you could stop at any place. In voting,
we're going all the way up to certification, but there are many
standards that exist just at the bottom level, just the standard.

There are standards out there, a tremendous amount don't have any
conformance tests, no conformity assessment, no certification.

There are others where there are just standards and conformance tests
but no certification. We do that at NIST in many areas.

For instance, the World Wide Web standards that you see today that allow
you to use your browsers correctly, we worked with the W3C to develop a
standard. We developed the conformance test.

Those conformance tests are out there. They are used voluntarily by the
implementers, by Netscape, by Microsoft, and in fact, they're used by
every person that develops browsers for the World Wide Web, because they
are free resources. Why wouldn't anyone want to use them?

So there are many ways you can develop this process. We in the voting
world go all the way to the top: standard, conformance testing,
conformity assessment, and certification.

*\[Slide 20\]*

I grouped the improvements to previous standards into five areas. Again,
we've defined what it means for voting systems to conform. We've created
precise and testable requirements. We've created performance benchmarks,
which didn't exist before. We addressed new technological advances and
added security, accessibility, and usability requirements.

*\[Slide 21\]*

Again, the conformance clause, normative versus informative;
conformance, by the way, and I think it is stated in the conformance
clause, is 100 percent. It's kind of like being pregnant. You either are
or you're not. You're not partially pregnant, and you're not partially
conformant.

The conformance clause defines classes. These are groupings of
requirements by voting devices, and I won't get into more detail, and
I'm sure you have questions about them, and the detailed questions will
be answered by the experts in classes.

There's an implementation statement. All that is, is a statement
requiring the voting system to talk about what optionality it supports.
That's a requirement of the conformance clause.

The next thing is extensions. Extensions in the standards parlance are
additional functionality above and beyond what's required in the
standard, and we allow, as most standards do, extensions.

The idea is you promote advancement and innovation when you allow
extensions, and this, I believe, may also be helpful. States may have
additional requirements above and beyond the VVSG, so if you as a state
develop a voting system with additional requirements, as long as you
don't violate what's in the VVSG, that system will still conform to the
VVSG.

Software independence is a new concept that I won't get into in detail
because John will, but it is defined in the conformance clause.

*\[Slide 22\]*

Again, improvements through previous standards: so we have tried to
specify precise testable requirements. The intent and the objective is
all requirements should have only one interpretation. Everyone should
understand what is meant. That's the goal.

Now we have many hundreds of pages and probably somewhere in there,
although this was our goal, it may not be perfectly precise, and if it
isn't, we try to find that out and correct it, but the goal again is
precision and no ambiguity.

The goal is to make every requirement testable, the ability to determine
that that requirement has been met. We feel if the requirement can't be
tested, then it's really not a meaningful requirement, with the possible
exception of these goal requirements.

*\[Slide 23\]*

Now, this was a late edition to my slide set. It was pointed out that we
do have a few requirements in the VVSG that are what we would call goal
requirements, sort of higher level for the most part, nonspecifically
testable requirements, and John, do you have the slide up with the
examples of some of those?

So just to summarize, there are just a few of these in the VVSG, and I
think different people in the TGDC may have wanted these because they
want to have sort of a goal for implementers, even though they know it's
not testable, to encourage certain things.

*\[Slide 24\]*

So I think we have a couple of those in the accessibility section. I
think there is something that says systems shall be end-to-end
accessible, which isn't very specific.

*\[Slide 25\]*

We have a couple of goal requirements in OEVT and integratability.

*\[Slide 26\]*

So those are kind of vague, and they're really exceptions to the rule,
and they are really a trade-off of having this high-level sort of
essentially non-testable requirement, but that's what was voted, but for
the most part, these are exceptions to the rule.

*\[Slide 27\]*

One of the things we've done that hadn't been done in the previous
standards is benchmark requirements both in usability and in
reliability, and accuracy testing.

There are many new technological advances that we had to account for in
these sets of voluntary voting system guidelines, both in the 2005 one
and the new one.

*\[Slide 28\]*

VVPAT, we had to deal with wireless and how it affects the voting
scenario, electronic poll books, EBMs and digital signatures, so these
are all accounted for in the new VVSG.

*\[Slide 29\]*

And just sort of as a summary slide, there are many, many new areas that
didn't exist before, many new chapters on security in these different
areas, and many new expanded human factors sections.

So that's my introduction. Before we get into John Wack's presentation,
questions or comments?

*\[Slide 30\]*

\[QUESTIONER:\] Are all the goal requirements clearly marked in the
discussion part that they are goal requirements?

\[MR. CUGINI:\] The question over there was, are the goal requirements
clearly marked, and the answer is no, they're not.

My point I wanted to make, I want to disagree with Mark a little bit.
You sort of say the goal requirements aren't testable and leave it at
that. I think that's a little severe.

You can test them, but you typically need to test them with expert
judgment. I think it's fair to say they're testable, but the test method
is more subjective than the test method for other, more specific
requirements, but it's not as if they won't be tested at all, so I think
they're testable.

\[MR. SKALL:\] It depends on the requirement.

\[MR. CUGINI:\] Well, the one I have in mind is testable. Maybe some
aren't either.

\[MR. SKALL:\] I think the one that you guys did is testable. The
integratable one, I don't know how you would test that systems shall be
integratable.

\[MR. CUGINI:\] Yeah. At least the testability is more problematic than
for a more specific requirement, leave it at that.

\[MR. SKALL:\] But that's a good point. I think I would be very welcome
for public comment to talk about maybe making sure those goal
requirements are certainly marked as such and something in the
discussion field highlighting that. That's a good point.

*\[Slide 31\]*

\[NARRATOR:\] Additional explanatory presentations on the voluntary
voting system guidelines can be accessed from the Web site:
vote.nist.gov.

[^1]: ^\*^ Certain commercial entities, equipment, or materials may be
    identified in this presentation in order to describe an experimental
    procedure or concept adequately. Such identification is not intended
    to imply recommendation or endorsement by the National Institute of
    Standards and Technology, nor is it intended to imply that the
    entities, materials, or equipment are necessarily the best available
    for the purpose.