VVSG Tutorial Narration

Core Requirements Part 4[^1]\
*\[Slide 1\]*

\[NARRATOR:\] This is Part 4 of the Next Voluntary Voting System
Guidelines Core Requirements Training Modules. Parts 1 through 4 are
presented by Dr. David Flater of the National Institute of Standards and
Technology's Information Technology Laboratory. These modules present an
overview of the material in Part 3 of the Guideline recommendations
including questions and answers from the members of the Election
Assistance Commission's Board of Advisors and Standards Boards. *\[Slide
2\]*

\[MR. FLATER:\] Section 6.4 is Workmanship with the subsections:
Software engineering practices; Quality assurance and configuration
management; General build quality; Durability and maintainability;
Temperature and humidity; Equipment transportation and storage.*\
\[Slide 3\]*

Software engineering practices is where we talk about coding
conventions, and it has many subsections. *\[Slide 4\]*

And the executive summary of this is that manufacturers are expected to
use current best practices for software engineering. There are some
words like published and credible, which have definitions following them
so that there is not going to be a subjective argument about what is
published and credible. In an ideal world, there would be a committee
somewhere that, on a yearly basis, would review the state of the art in
software engineering and say okay, this is what represents the state of
the art, and we would expect new voting systems submitted to be done in
this way, but failing that, we do have some specifications as best we
can do in the VVSG for what a published and credible set of coding
conventions looks like. Now there's also something in there which I'll
just refer to as the three-year rule. It essentially says for a new
voting system being submitted for certification for the first time, they
have to show that the software engineering practices they used were
current at least within the three years prior to their request for
certification.

Having done that, however, if the same system is brought back for
recertification with incremental changes later, it's not going to be a
moving target. You conform to the best practices when you built the
system, and the state of the art is going to continue to morph over time
and evolve. The maintenance task is different from the development task.
When you're in maintenance, your goal is to be consistent with what you
did before, so that's the three-year rule. Three years as of the first
certification.*\
\[Slide 5\]*

\[QUESTIONER:\] Did this come from the discussion in Denver at the
meeting the EAC put together where all of the different elements were
involved? Some of the vendor software supervisors, Diebold specifically,
were alleging that some of the coding conventions and existing standards
have basically tied their hands. So is this addressing that at all in
terms of their flexibility?

\[MR. FLATER:\] The question was: Did this come from/ address concerns
raised at a meeting in Denver in which it was asserted that coding
standards that were embedded within the 2002 and 2005 VSS were tying the
hands of the vendors unnecessarily? I was not at that meeting in Denver,
but I can tell you, this does address that concern, or the changes to
the section do address that concern, and I'll be going into more detail
to the extent possible. The changes can be summarized as follows: 80
percent of the prescriptive coding rules that had been incorporated in
the 2002 and 2005 specifications were best practice at a particular time
in history, but from the standpoint of here and now, looking at current
best practices, current practices, don't seem to jive with the state of
the art. That 80 percent has been taken out, and now if you look
specifically at what the old guidelines said, what the language really
said was, vendors are expected to use best practices, but as a sort of
fallback position, here's a coding standard you can use. And what I'm
led to understand is that somehow or other, reality on the ground
became, this is the coding standard you must use, and that's
unfortunate.

\[QUESTIONER:\] David, that's not true. The vendors may be taking that
position, but we relaxed those standards years ago. As published, they
were taken as a minimal guideline. There has never been a prohibition on
using state-of-the-art software engineering practices.

\[MR. FLATER:\] Okay, Brit's comment was that, in fact, that section of
the guidelines was treated as a recommendation or informative in 1990.

\[QUESTIONER:\] And that's basically, as I understand it, what you are
saying here. We'll take a benchmark of three years ago, but we are not
going to prohibit you from advances that are considered current state of
the art.

\[MR. FLATER:\] Yes. To try to repeat what Brit said, what we're saying
here is we're setting a benchmark of three years ago, but we're not
going to prohibit you from updating. That is true. What has changed here
is that any ambiguity about the status of what was actually written in
the guidelines has been removed. There is not going to be a
misunderstanding about, well, one, character variable names, are they
banned? Or do X&Y coordinates have to be called X under bar, y under bar
coordinates or something? That level of requirement has been removed.
What's been retained and expanded and beefed up is the 20 percent of the
rules that were in there having to do with software integrity, so those
prescriptive rules that are practices that lead to high-integrity
software have been retained and beefed up. This should be consistent
with best practices. It's sort of a specialization of best practices for
high-integrity software.*\
\[Slide 6\]*

In addition, what we might call worst practices, things that just have
been maligned for many years as a very bad idea, like self-modifying
code, those are still prohibited. Defensive programming is required.*\
\[Slide 7\]*

And we have a question.

\[QUESTIONER:\] Who defines these? Are they going to be defined in your
test criteria? Can you constantly update what is the best practice; what
is the worst practice? Who is going to be responsible for identifying
those items for the testing labs?

\[MR. FLATER:\] The question was: Who's responsible for defining what is
'best practice'? With the exception of that 20 percent integrity rules
that I talked about which are written into the specs still, the best
practice in general in terms of software engineering code conventions,
what we have are a couple of requirements that say it has to be
published, and there's a definition of what it means to be published in
gruesome detail, and it has to be credible, and credible being the
tricky one. There was language in the old specification that said
published, reviewed, industry-accepted, that was really no better, but
credible has a definition that essentially says, you can establish that
your coding conventions are credible if you can point to three other
organizations that are not voting equipment vendors that have used these
same conventions in the past three years. It is a relatively low bar,
but it is an objectively evaluatable bar. So, as I described earlier,
yes, it would be great if we had some kind of commission that reviewed,
on a yearly basis, what the current best practice is, and published a
list that said okay, you should use one of the following, but in the
absence of a big extra commitment that we don't have the authority to
make, this is the best we can do.

\[QUESTIONER:\] Okay, and then worst practices? Who defines them? I mean
they aren't always obvious. Suddenly we find out some things that other
people consider worst practices that we did not know about.

\[MR. FLATER:\] The question was: Who gets to define what worst
practices are? Most of these are things that have carried down through
the history of software engineering as known risk factors for latent
software faults and verifiable code. We can argue about the specific
cases. There really aren't that many of them in there, so we can argue
about specific cases and discuss in the public comment period.

\[QUESTIONER:\] So it's going to be up to the testing lab to determine
them?

\[MR. FLATER:\] Oh, no, no. The question was: Is it up to the testing
lab then? No, there's a list in the document that says self-modifying
code is prohibited and a few others.

\[QUESTIONER:\] On this "best and worst practices," I would foresee that
this would continue kind of the way it has in the past. The EAC has
technical reviewers as part of their certification process. And if a lab
questions whether or not what a vendor is doing is best practice or
worst practice, you submit it to the technical reviewers, and they make
the judgment on it. And that's the end of it. That is kind of the way it
worked in the past. You came to the NASED Voting Systems Board. We
looked at the situation and made a ruling on it. If it was of general
interest, we published a paper on it; otherwise, if it was a specific
issue, we would say if you need this, go ahead with it.

\[MR. FLATER:\] Comment from Brit was, under the NASED process, that
manufacturers essentially would submit requests for interpretation about
specific practices, and then a ruling would be made by NASED, and I mean
the question here: Would these rulings possibly relax and change the
things that are written in the guidelines? And the answer is no. We are
talking specifically about best practices for software engineering.

\[QUESTIONER:\] And we let people operate above that base if they wanted
to. If there was a question as to were they too far above or not above
it at all? We had a mechanism to resolve those kinds of issues. I would
think, Brian, that in your certification process, you will have a
similar type of situation where you have technical reviewers that when
these questions come up, they come in to the EAC, and those technical
reviewers make a recommendation, and then the EAC rules on it.

\[MR.HANCOCK:\] Well, we can certainly do that. But I am hoping that,
and I think that there are some better definitions in here now than
there were before. So there shouldn't be the need for as much of that.

\[QUESTIONER:\] Yeah, we did not have the Software Engineering Institute
at Carnegie Mellon back in the good old days. In the guidelines, you
don't have to try to describe exactly what are the best practices or
worst practices. You have got certain things that are prohibited. Those
are absolute no-nos. But you put in place a mechanism for resolving
questions as they come up.

\[MR. FLATER:\] Okay, Brit made a good point just now that the things
I've been referring to as worst practices are, in fact, not a moving
target. These are things that are simply prohibited and don't require
yearly review. But to try to put the previous conversation on the
record, Brit was discussing the NASED process in which questions about
software engineering practices would be addressed as they came up with
respect to current best practices.

Comment from Brian Hancock was, yeah, we probably still expect to do
some of that, but we hope that by making the guidelines more precise,
we'll reduce that burden, and I believe that burden will be reduced also
because of the removal of many of the possibly conflicting style
guidelines that were in there.*\
\[Slide 8\]*

\[MR. FLATER:\] A question is, please explain defensive programming. I
have subsequent slides that go into more detail. In a nutshell, this is
saying that garbage in, garbage out is not an excuse, that you should,
to the extent possible, detect or prevent improper input, whether it be
from the user or from a faulty piece of code elsewhere in the system.
You should not make assumptions that everyone else is doing their job.
You should do diligence on the input you're given, and if you find
something that's blatantly wrong, you shouldn't just go with it, you
should throw an exception, and this essentially prohibits masking
errors. If in fact there is an integrity problem in the system, then it
should not get swept under the rug, we need to know about it.\
*\[Slide 9\]*

A questions, yes.

\[QUESTIONER 1:\] What I heard you say was, if there are certain things
that are put in that shouldn't be there, there is a mechanism that would
catch it.

\[QUESTIONER 2:\] Except those 20 percent undervotes-

\[QUESTIONER 1:\] But if at the beginning of the system when the
election officials are designing the ballot and doing the input for
what's going to go on the ballot, are there- I think you called them
"defensive systems"- in place to help make sure that you have the most
usable ballot design?

\[MR. FLATER:\] Okay, the question from Jim was, similar to defensive
programming, could there be defensive measures in the system with regard
to ballot design so that common ballot design blunders would be detected
and prevented by the system? And I've simultaneously gotten both Barbara
Guttman and Sharon Laskowski, and Sharon Laskowski is going to comment.

\[MS. LASKOWSKI:\] There are two aspects of ballot design. One is what
is inherent in the system of what it allows, for example, the placement
of instructions in appropriate places that are more usable than in other
places- like instructions at the beginning or instructions in san serif
font that is easier to read. However, the details of the ballot design
for a specific election are up to the election officials, and that is
out of our scope for testing in the lab, because you can't guess what
all the different types of ballots will be across the United States. So
we have tried to do as much as we could within the usability section on
the structure of the ballot. There was not that much we could do with
this particular standard in terms of what details to provide.

\[MR. FLATER:\] So is it a correct summary to say that we've done what
we can in terms of providing usability guidance, but we can't intrude on
the prerogative of jurisdictions to do design their way?

\[MS. LASKOWSKI:\] Correct.

\[MR. FLATER:\] Okay.

\[QUESTIIONER 1:\] Are there some 'shoulds' in there?

\[QUESTIIONER 2:\] I have a question, Mary Herrara, New Mexico. Are you
going to give any recommendations as far as what design is good? I think
this would help the states with ballot counting-

\[MS. LASKOWSKI:\] I believe the EAC has a document that Design for
Democracy put together on ballot design. *\[Slide 10\]*

\[MR. FLATER:\] So the last comment on this slide was that vendors that
use state-of-the-art programming language and standards will have an
easier time meeting the requirements.\
*\[Slide 11\]*

The impact of changes, we resolved this controversy I discussed earlier.
There should be more flexibility for manufacturers. There is some
pressure being applied to migrate the state-of-the-art programming
languages and standards. There is also, because of the beefing up of the
integrity-related requirements, we should get more reliable higher
integrity of software. The costs of this are that if we have legacy code
that isn't up to the state of the art, it may need to be cleaned up and
reinforced to meet the same requirements, and also because there are
less prescriptive rules about variable naming stylistic rules that are
in there, what's left is going to require more experience and judgment
from the test labs, because we're dealing with best practices. *\[Slide
12\]*

The term COTS, or the acronym COTS, standing for commercial
off-the-shelf, because of the way that the entire software field has
moved in recent years, for the purposes of this document, it's most
expedient for us to include widely used open source software under the
umbrella of COTS, if you will, even though it's not necessarily
commercial. There are certain guidelines that apply with regards to
COTS, and certain ones that apply with other kinds of software in the
system, and so we've just lumped all of the general-purpose, widely
available, widely used stuff under the COTS umbrella. Now to clarify
some or to address some borderline cases that have come up over the
years with regards to the false dichotomy of COTS versus non-COTS, some
additional terms have been added.

Application logic is logic from any source that is specific to the
voting system, with the exception of border logic. Border logic is what
in programming jargon is known as glue code. This is stuff that is
produced by programmers to attach the application logic that is
developed by the voting system vendor for the voting system to all the
code that is going to be reused. And finally, there is another
borderline category called third-party logic, which is none of the
above. Anytime you hear the phrase modified COTS, it's not COTS, what it
is, is third-party logic. This also includes source code generated by
the COTS package.\
*\[Slide 13\]*

The last thing I'm going to say about this, is that there is no blanket
testing exemption for COTS software, okay. I hear this phrase COTS
exemption used a good bit. All software in the voting system is subject
to black box testing, period. *\
\[Slide 14\]*

There is a question, yes.

\[QUESTIONER:\] On that issue of COTS, does that mean if a vendor comes
in using WINDOWS 6.0, that the system is certified to that version of
COTS? If they want to use another version of that COTS later, does that
require them to put back in for-

\[MR. FLATER:\] The question was about certification with respect to a
particular version of a COTS package. This is, in fact, a configuration
management concern. The certification is specific to the software
configuration that was tested. If someone wants to use a newer version
of a COTS package, this falls under the category of recertification. In
this case, you're not necessarily going to need to perform a complete
top-to-bottom retesting of everything. What you're going to do is a kind
of regression testing that has to do with what the impact at that
particular change would be. So the cost of recertifying with a newer
version of the COTS package would be expected to be considerably lower,
but it is a recertification, it gets a new number. Yes.

\[QUESTIONER:\] Does this also include small devices such as a mouse or
keyboard or those types of commercial hardware?

\[MR. FLATER:\] The question again, about certification of COTS, in this
case COTS hardware like a mouse keyword, etc. Again, this is all part of
the configuration that is specified by the manufacturer and is
certified. This, again, is in the category of what you might call fidley
little changes that we might want to make to the system. If you change
out any device from the certified configuration, you are technically
breaking the configuration or breaking the EAC certification.

Now states, as a matter of practicality, I mean, states are sovereign
here. If you want to make a rule that says, well, for our purposes,
we're going to consider all Microsoft-compatible mice to be equivalent,
you can do that, but of course, there's no way someone can certify a
configuration they haven't tested. Incompatibilities can happen. So this
again would be a case where ideally you'd have a very quick inexpensive
recertification. Okay, here's another mouse, bing, bang boom, no
problems, that mouse works fine, and we issue a new number for it. Yes.

\[QUESTIONER:\] Well, particularly in the case of say, Microsoft, which
is doing service packs- we are now at service pack 3 in XP- is it
specific to a service pack or is it that all service packs that come
after that are okay if it is still the Microsoft XP?

\[MR. FLATER:\] Question from Doug Lewis was: Do service packs count as
a change? And if we want to go into lots of gruesome detail about this,
it gets to the point when I really ought to let the EAC talk about their
practices here. My understanding is that, that is a change. A change is
a change is a change, and if you change the system, it gets a new
number. That's good configuration management.*\
\[Slide 15\]*

Archivalness, some people don't think archivalness is a word, but it's a
word, this is fairly quick and simple. Records in the voting system
shall last at least 22 months in temperatures up 40 degrees Celsius, 104
degrees Fahrenheit, and humidity up to 85 percent. *\[Slide 16\]*

This dovetails with some other requirements such as in the operations
manual, the manufacturer is expected to specify the care and handling
precautions you must follow for your removal of media and records to
last that statutory 22-month period. In addition, in the testing
standard, there's a requirement for the test labs to review the
specifications, the data sheets, for these media that the vendor has
specified to ensure that they are, in fact, supposed to survive this
long under the conditions that they're being used under.*\
\[Slide 17\]*

This is responsive to complaints about thermal paper going off. You
know, anecdotes are anecdotes, but some of them sound pretty bad. So we
have specified the ambient conditions, and end users should not have to
resort to extreme measures to preserve records for the 22-month period.
And we would expect that there should be additional scrutiny now to
ensure that the materials and media being used will, in fact, last the
22 months that is required.*\
\[Slide 18\]\
*\[NARRATOR:\] Additional explanatory presentations on the Voluntary
Voting System Guidelines can be accessed from the Web site:
vote.nist.gov.

[^1]: ^\*^ Certain commercial entities, equipment, or materials may be
    identified in this presentation in order to describe an experimental
    procedure or concept adequately. Such identification is not intended
    to imply recommendation or endorsement by the National Institute of
    Standards and Technology, nor is it intended to imply that the
    entities, materials, or equipment are necessarily the best available
    for the purpose.