Rev. 2005-03-01

[THE RECOMMENDATIONS IN THIS DOCUMENT ARE ALL TENTATIVE, PRE-DECISIONAL
DRAFTS THAT DO NOT REPRESENT THE CONSENSUS OF NIST OR OF THE
SUBCOMMITTEE.]{.mark}

# Changelog

2005-03-01 Miscellaneous updates reflecting progress since 01-05.
Replaced "testing authority" with "VSTL" (refs to ITAs in context of
current process are still retained). Renamed X-dependencies and added
discussion about why the section is needed. Moved types-of-testing
discussion from Testing Pieces.

2005-01-05 Added bigger disclaimer, removed a possibly sensitive
reference.

2005-01-04 Revised X-dependencies to be aware of 2002VSS Vol. 1 Sec.
9.6.4, which describes an existing but outdated process for
interpretations and continuous improvement.

30. Replaced deprecated "ITA" term with more generic "testing
    authority," except in contexts where the 2002 VSS and existing
    qualification process are clearly intended.

> Tweaked X-dependencies to emphasize "NIST recommends..." and reduce
> the appearance of raising problems without offering solutions.

2004-12-27 First versioned version.

# Pieces for Rationale Document

## 2. Strategy

### a. General

#### Review of existing standards, specifications, and related work

To ensure that previously written requirements would not be overlooked,
NIST reviewed the following resources. The resulting guide to existing
requirements has not been put into publishable form but is being
utilized by project members as they develop new recommendations.

NIST also reviewed sample ballot forms, vote data reports and other
materials from several states.

##### Standards, draft standards, regulations, and guidelines

\[HAVA\] Help America Vote Act of 2002, Public Law 107-252, 2002-10-29.

\[2002VSS\] 2002 Voting Systems Standards, available from
<http://www.fec.gov/pages/vssfinal/vss.html>.

\[P1583/D5.3.1\] IEEE Draft Standard for the Evaluation of Voting
Equipment, draft 5.3.1, 2004-10-08, available from
<http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[CoE 2004-09-30\] Council of Europe, Committee of Ministers to member
states on legal, operational, and technical standards for e-voting,
adopted by the Committee of Ministers on 2004-09-30 at the 898^th^
meeting of the Minister's Deputies, e-mail from Lori Steele, 2004-11-10.

\[EML3\] Election Markup Language v3.0, 2003-02-24, available from
<http://www.oasis-open.org/committees/election/index.shtml>.

[INSERT HUMAN FACTORS & PRIVACY REFERENCES HERE]{.mark}

\[SP 500-256\] Sharon J. Laskowski et al., "Improving the Usability and
Accessibility of Voting Systems and Products," NIST SP 500-256, 2004-05.

\[508\] Section 508 of the Rehabilitation Act: Electronic and
Information Technology Accessibility Standards, 2000-12-21, available
from <http://www.access-board.gov/508.htm>.

\[ADA\] ADA Checklist for Polling Places, 2004-02, available from
<http://www.usdoj.gov/crt/ada/votingchecklist.htm>.

[INSERT SECURITY & TRANSPARENCY REFERENCES HERE]{.mark}

[See also \[2002VSS\] Vol. I Appendix B.]{.mark}

##### Issue lists

\[D5.3.1 Comments 2004-10-19\] Comments for d5-3-1 dated 10-19-2004
revC.xls, available from
<http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 Software Comments 2004-09-01\] Software comments 5.0
(9-01-04).xls, available from
<http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 Security Comments 2004-08-18\] Security extract V5 Comments --
2^nd^ NJ Meeting.xls, available from
<http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 Reliability Accuracy Comments 2004-09-06\] 5.0 Comments Section
5.2 & 6.2 (9-6-04).xls, available from
<http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 Accessibility Comments 2004-08-01\] V5 Ballot Accessibility
Comments -- TG3 (8-1-04) .xls, available from
<http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 Environmental 2004-08-15\] 5.0 Comments Section 5.4 & 6.4.xls,
available from <http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 EMC 2004-08-23\] 5.0 Comments Section 5.5 (8-23-04).xls, available
from <http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 Provisional 2004-09-10\] Gough-Provisional Ballot Comments.xls,
available from <http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 COTS 2004-06-18\] Resolutions for COTS Comments for Draft 5.0 of
IEEE P-1583, <http://www.lipsio.com/COTS/docs/COTS.resolved.html>.

\[5.0 TDP 2004-04-23\] 5.0 p1583 \_TDP-Proposed resolution_Apr04.xls,
available from <http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

\[5.0 Comments 2003-10-16\] Ballot Comment Form 5-0 10-16-2003.xls,
available from <http://grouper.ieee.org/groups/scc38/1583/private/>
(password-protected).

##### Requests for proposals

\[AZ\] "OCR and DRE Voting Equipment -- Statewide," Request for
Proposal, Arizona, 2003. E-mail from Allan Eustis, 2004-10-12.

\[CO-REG\] "Statewide Voter Registration System," Request for Proposals
\# DOS-HAVA-0001, Colorado, 2004-01-16, formerly available from
<http://www.sos.state.co.us/pubs/hava/hava_main.htm> (now gone).

\[CO-IVV\] "Independent Verification and Validation for SCORE Project,"
Request for Proposals \# DOS-HAVA-0002, Colorado, 2004-06-03, formerly
available from <http://www.sos.state.co.us/pubs/hava/hava_main.htm> (now
gone).

\[GA\] Request for Proposal GTA000040, Georgia, 2001. E-mail from Merle
King via Allan Eustis, 2004-10-11.

\[MD\] "Direct Recording Electronic Voting System and Optical Scan
Absentee Voting System for Four Counties," Project Number SBE-2002-01,
Maryland, 2001-07-17, available from
<http://www.elections.state.md.us/citizens/voting_systems/voting_system_procurement.html>.

\[MI\] Invitation To Bid \# 071I4001011, Michigan, 2003, available from
<http://www.michigan.gov/sos/0,1607,7-127-1633_11619_27151-77943--,00.html>.

\[OH-VOT\] "Statewide Voting System(s)," Request For Proposal \#
SOS0428365, Ohio, 2003-05-23, available from
<http://www.sos.state.oh.us/sos/hava/index.html>.

\[OH-REG\] Request For Proposal \# SOS032786279, Ohio, 2003-04-09,
available from <http://www.sos.state.oh.us/sos/hava/index.html>.

\[UT\] "Executive Summary: Voting Equipment Selection Committee Request
for Proposal," Utah. E-mail from Allan Eustis, 2004-10-07.

##### Testimony

\[Coney 2004-09-22\] Lillie Coney, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Conrad 2004-09-22\] Frederick Conrad, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Deutsch 2004-09-21\] Herb Deutsch, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Fischer 2004-09-20\] Eric A. Fischer, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Gaston 2004-09-20\] Charles A. Gaston, testimony to EAC, available
from <http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Golden 2004-09-22\] Diane Cordry Golden, testimony to EAC, available
from <http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Jones 2004-09-20\] Douglas W. Jones, testimony to EAC, available from
[http://www.cs.uiowa.edu/%7Ejones/voting/nist2004.shtml](http://www.cs.uiowa.edu/~jones/voting/nist2004.shtml).

\[Jones 2004-09-23\] Douglas W. Jones, supplemental testimony to EAC,
available from
[http://www.cs.uiowa.edu/%7Ejones/voting/nist2004supp.shtml](http://www.cs.uiowa.edu/~jones/voting/nist2004supp.shtml).

\[King 2004-09\] Merle S. King, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Noren 2004-09\] Wendy S. Noren, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Redish 2004-09-22\] Janice Redish, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Relton 2004-09-21\] Joy Relton, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Saltman 2004-09-20\] Roy G. Saltman, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Shamos 2004-09-20\] Michael I. Shamos, testimony to EAC, available
from <http://vote.nist.gov/PanalistandPublicTestimony.html>.

\[Wallach 2004-09-20\] Dan S. Wallach, testimony to EAC, available from
<http://vote.nist.gov/PanalistandPublicTestimony.html>.

#### Standards architecture

NIST is recommending a reorganization of the Voting Systems Standards to
bring them in line with applicable standards practices that are
abstracted from our years of association with ISO, W3C and other
standards-creating organizations. This includes adding a section to
define the meaning of conformance (called a *conformance clause* in
ISO), identifying testable requirements as *compliance points*, and
defining *profiles,* which allow requirements to vary as needed to
accommodate variations in voting equipment.

Preferably, requirements should specify *what* (the desired
performance), not *how* (a design to accomplish that). For example, a
requirement that reads "single-bit errors shall be detected" is
preferable to one that reads "products shall use memories with parity
bits." Profiles are created to resolve the conflict that occurs when the
*what* depends on the *how*. For example, the unstated assumption that
the voting equipment would have an electronic memory at all requires
placing the preceding example in a profile for electronic voting
equipment.

Design-constraining requirements are controversial because vendors would
like the freedom to provide the desired qualities / performance in
different ways. However, in cases where vendors are unable to determine
for themselves whether or not a given design is conforming, they may
welcome design constraints as a way to avoid repeated failures and
costly retesting of their products. Moreover, in cases where the desired
quality is difficult to define abstractly, an enumeration of conforming
cases may be the only practical alternative, particularly if there is
only one design approach that is ever actually usable in practice. Some
pragmatism will be required.

A vendor who is submitting a system for testing must provide an
*implementation statement* that identifies exactly which profiles the
system is asserted to support. Conformance tests may be catalogued
according to which compliance points they exercise. The set of
conformance tests appropriate to that claim may then be determined
automatically. Upon passing those tests, the system may be qualified for
only the claimed profiles.

Identified compliance points and a profiles mechanism in the VSS will
facilitate traceability from state standards to the VSS. States will
effectively define their own profiles over the VSS, adding compliance
points they deem necessary without excessive repetition and revision of
VSS text.

### d. Testing

#### Purpose

The 2002 Voting Systems Standards define qualification testing as "the
examination and testing of a computerized voting system by an
Independent Test Authority using qualification test standards to
determine if the system complies with the qualification performance and
test standards and with its own specifications. This process occurs
prior to state certification."

The purpose of qualification testing is to provide the states and other
affected stakeholders with some level of assurance that a voting system
is fit for use. States have the option to subject a voting system to
additional scrutiny before purchasing and deploying it; however, most
states require qualification by an ITA as an entry condition.

Even if procedural controls and audit trails ensured that any miscount
would be detected, it could still be catastrophic for a state to have to
rerun a compromised election and to remedy the faulty equipment. It is
in the states' interests for the qualification process to eliminate
voting systems that are not trustworthy before they are purchased and
deployed.

#### Types of test methods

Traditionally, testing methods have been divided into black-box and
white-box test design. Neither method has universal applicability; they
are useful in the testing of different items.

Black-box testing is usually described as focusing on testing functional
requirements, these requirements being defined in an explicit
specification. It treats the item being tested as a "black box," with no
examination being made of the internal structure or workings of the
item. Rather, the nature of black-box testing is to develop and utilize
detailed scenarios, or test cases. These test cases include specific
sets of input to be applied to the item being tested. The output
produced by the given input is then compared to a previously defined set
of expected results.

White-box testing (sometimes called clear-box testing to suggest a more
accurate metaphor) allows one to peek inside the "box," and focuses
specifically on using knowledge of the internals of the item being
tested to guide the testing procedure and the selection of test data.
White-box testing can discover extra non-specified functions that
black-box testing wouldn't know to look for and can exercise data paths
that would not have been exercised by a fixed test suite. Such extras
can only be discovered by inspecting the internals.

Complimentary to any kind of testing is logic verification, in which
formal methods are used to prove that the logic of the system satisfies
certain assertions. When it is impractical to test every case in which a
failure might occur, formal methods can be used to prove the correctness
of the logic generally. However, verification is not a substitute for
testing because there can be faults in a formal proof just as surely as
there can be faults in a system. Used together, testing and verification
can provide a high level of assurance that a system's logic is correct.

#### Repeatability and reproducibility

For qualification of voting systems to be consistent, fair, and
meaningful, it is necessary to control variability in the conformity
assessment system. Testing cannot be an afterthought to a standard: both
the requirements to be tested and the methods by which they are to be
tested must be specified with appropriate precision. The following
hypothetical example illustrates the codependence of requirements and
test methods.

  ------------------------------------ ----------------------------------
  **Example text**                     **Impact on testing**

  The unit shall respond to all user   Vague requirement leaves tester in
  input in a timely fashion.           the position of determining what
                                       is considered "timely," creates
                                       opportunities for inconsistent
                                       evaluation and challenges by
                                       vendors.

  The unit shall respond to all user   Good requirement leading to
  input in 3 seconds or less.          pass-fail verdict. However, the
                                       test method to verify the
                                       requirement is undefined.
                                       Different testing authorities
                                       using different test methods may
                                       get different results. The vendor
                                       could challenge that the set of
                                       user inputs chosen by a VSTL is
                                       atypical of use in practice.

  The VSTL shall measure and report    In conjunction with the good
  the mean response time and worst     requirement, this specified test
  response time over the following set method enables consistent,
  of user inputs, employing the test   informative, and
  ballot form defined in Section XYZ:  difficult-to-challenge results.
  opening the ballot; voting for one   
  randomly selected candidate in each  
  contest; \[...\]. Units with worst   
  response time exceeding 3 seconds    
  shall be disqualified.               
  ------------------------------------ ----------------------------------

NIST is recommending a complete review and revision of requirements in
the Voting Systems Standards to ensure that they are sufficiently
precise to enable meaningful testing and an expansion of the testing
standards to specify test methods for a large subset of those
requirements.

To date (2005-02-28), NIST has produced draft formal definitions for the
terms that appear in vote data reports so that the accuracy of those
terms in actual reports is well-defined; has begun defining abstract
test cases pertaining to the accumulation, counting, and reporting of
votes; and is revising the text of relevant compliance points to improve
their clarity and precision.

Additionally, NIST recommends eliminating the provision in the 2002 VSS
for qualification of voting systems that do not conform to the
requirements.[^1] If there are requirements that are frequently unmet by
qualified systems, these requirements should be reviewed for possible
elimination.

#### Transparency

The public must also be assured that the voting system is fit for use.
This can occur vicariously, through trust in the VSTL and election
officials; indirectly, through verification that the qualification
process was responsibly executed; directly, through election
verification; or through a combination of these.

Recognizing that public records laws are standard practice in many
arenas where public trust and/or safety are at stake, NIST is
recommending that qualification test reports be released to the public
as evidence that the qualification process was responsibly executed. In
the event that release of the entirety of the reports proves to be
problematic, NIST will recommend standards on data to be provided,
called a Public Information Package, that will set out minimum
requirements on the information that must be published.

An ideal election verification mechanism would enable every voter to
become a tester. Every voter could assure himself or herself that his or
her vote was both cast as intended and counted as cast. However, the
voters would need to trust the verification mechanism, or it would
accomplish nothing. It remains to be demonstrated that a mechanism that
enables both cast-as-intended and counted-as-cast verification while
preserving the secrecy of the vote can gain widespread understanding and
trust. A mechanism that enables only cast-as-intended verification is
considerably less complicated to design, and such mechanisms are already
required in several states. The Security and Transparency Subcommittee
is currently drafting recommendations pertaining to Directly Verifiable
(DV) systems and Indirectly Verifiable (IV) systems.

#### Coding conventions and code reviews

Volume 1, Section 4.2 and Volume 2, Section 5.4 of the 2002 Voting
Systems Standards define coding conventions and a source code review to
be conducted by ITAs. Vendors are permitted to use current best
practices in lieu of the coding conventions defined in the VSS; however,
the coding conventions in the VSS are out of date, and if followed,
could do more harm than good.

The coding conventions are a means to the end of facilitating ITA
evaluation of the code's correctness to some level of assurance beyond
that provided by black-box testing. That evaluation is underspecified in
the 2002 VSS, yielding a cart-before-horse situation in which adherence
to the coding conventions could be verified much more rigorously than
the correctness of the software.

NIST will recommend standards for logic verification to be used in
evaluating the correctness of voting system logic, including but not
limited to software implementations. The coding conventions will be
evaluated in light of their applicability to the recommended
verification techniques and revised as appropriate.

To date (2005-02-28), NIST has produced a draft standard for logic
verification and proposed revisions to the coding conventions. Coding
conventions addressing the need for integrity in voting software have
been retained, expanded, and made mandatory, while stylistic conventions
that are made redundant by more recent, publicly available coding
conventions have been removed in favor of the published conventions.
Whether the coding conventions addressing integrity can also be replaced
by recent, publicly available coding conventions for high-integrity
software is yet to be determined.

#### Quality assurance and configuration management

Volume 1, Sections 7 and 8 and Volume 2, Section 7 of the 2002 VSS
require the vendor to follow certain quality assurance and configuration
management practices and require the ITA to conduct several audits and
documentation reviews to ensure that they were followed. These too are a
means to an end, the end being an assurance that the vendor is following
responsible engineering practices.

Since the Voting Systems Standards were first issued, it has become
possible for vendors to be certified under ISO 9000 and/or appraised
under CMMI.[^2] It is not clear whether a separate standard for voting
system vendors, in lieu of requiring ISO 9000 certification, is any
longer necessary or desirable. NIST recommends replacing the affected
text of the Voting Systems Standards with a requirement for ISO 9000
certification of all voting systems vendors to a scope of operations
appropriate to the purpose of developing voting systems.

## 3. Procedural requirements

It is not within the scope of the Voting Systems Standards to impose
requirements on election administration or on the standards process.
However, the goals of this revision to the standards *cannot be
accomplished* unless those goals are also upheld in the affected
processes. In light of that, NIST makes the following observations.

## a. Training

Poll workers are a critical "component" of the voting system. The system
as a whole cannot function properly if poll workers are not enabled to
fulfill their roles properly. In a statement dated 2004-11-08, the
National Association of Secretaries of State called for a national
survey to gather best practices for poll worker recruitment and training
and a study to attempt to simplify the process.

Although the Voting Systems Standards cannot have a direct impact on
poll worker training, they can help indirectly by recommending a higher
level of usability testing for voting equipment. More usable equipment
simplifies training.

## b. Continuous improvement process

As ambiguities are discovered and interpretations are made, the Voting
Systems Standards should be revised to remove the offending ambiguities.
The current process described in Volume I, Section 9.6.4 of the 2002
Voting Systems Standards charges the FEC with this task, but this does
not reflect the changes brought about by HAVA. NIST recommends that a
new standing committee be established for the continuous improvement of
the Voting Systems Standards.

## c. Interpretations and appeals process

It is a virtual certainty that any given specification contains
requirements that can be subject to multiple, equally defensible
interpretations. These are usually discovered during conformity
assessment, when the interpretation of the tester is found to conflict
with the interpretation of the vendor. The availability of a process to
resolve conflicting interpretations of the specification in a timely and
authoritative manner is a vital part of conformity assessment
methodology.

The current process is described in Volume I, Section 9.6.4 of the 2002
Voting Systems Standards. NASED is responsible for issuing
interpretations, but it is unclear whether a vendor or only an ITA can
request an interpretation, or whether such interpretations are binding
on qualifications in progress. NIST recommends that the standing
committee established above also be tasked with issuing interpretations
of the Voting Systems Standards, and that the vendors' right to appeal
be defined clearly.

## d. De-qualification process

No conformity assessment process is perfect. Systems with
non-conformities, even serious ones, can obtain qualification, only to
wreak havoc at the precinct level after they are deployed. When a
serious flaw is discovered in one jurisdiction, other affected
jurisdictions should be informed. At present, however, there is no
process to de-qualify voting systems that are discovered to have serious
problems after qualification has been granted. NIST recommends the
establishment of a de-qualification process in order to reduce the
impact of these events.

## e. Compliance controls

Deployments of equipment and software configurations that are not
identical to any certified configuration have been reported. States
should institute compliance controls to address this problem. NIST can
assist by adding certified voting system software to the National
Software Reference Library (<http://www.nsrl.nist.gov/>) for comparison
with software that is later deployed. The Security and Transparency
Subcommittee is currently working on recommendations for performing that
comparison.

[^1]: Volume 2, Appendix B, Section B.5: "Any uncorrected deficiency
    that does not involve the loss or corruption of voting data shall
    not necessarily be cause for rejection."

[^2]: Capability Maturity Model Integration,
    <http://www.sei.cmu.edu/cmmi/>.