VOLUNTARY VOTING SYSTEM 

GUIDELINES
VOLUNTARY VOTING SYSTEM 

GUIDELINES

Volume II

National Certification Testing Guidelines
5

Voluntary Voting System Guidelines

Table of Contents

Volume I Voting System Performance Guidelines

Overview Voluntary Voting System Guidelines Overview

Section 1 Introduction

Section 2 Functional Requirements

Section 3 Usability and Accessibility Requirements

Section 4 Hardware Requirements

Section 5 Software Requirements

Section 6 Telecommunications Requirements

Section 7 Security Requirements

Section 8 Quality Assurance Requirements

Section 9 Configuration Management Requirements

Appendix A Glossary 

Appendix B References

Appendix C Independent Verification Systems

Appendix D Technical Guidance for Color, Contrast, and Text Size

Volume IINational Certification Testing Guidelines

Overview Voluntary Voting System Guidelines Overview

Section 1 Introduction

Section 2 Description of the Technical Data Package 

Section 3 Functionality Testing

Section 4 Hardware Testing 

Section 5 Software Testing

Section 6 System Integration Testing 

Section 7 Quality Assurance Testing

Appendix A National Certification Test Plan

Appendix B National Certification Test Report

Appendix C National Certification Test Design Criteria
vii

Voluntary Voting System Guidelines Overview

Table of Contents

Voluntary Voting System Guidelines Overview ............................................................. ix

Purpose and Scope of the Guidelines............................................................................... x

Effective Date .................................................................................................................. x

Summary of Changes....................................................................................................... x

Volume I: Voting System Performance Guidelines Summary......................................... xi

Volume II: National Certification Testing Guidelines Summary .................................... xii

Guide to Section Locations.............................................................................................. xiii
Version 1.0

Voluntary Voting System Guidelines

Volume II: National Certification Testing Guidelines

Overview

9

Voluntary Voting System Guidelines Overview

The United States Congress passed the Help America Vote Act of 2002 (HAVA) to

modernize the administration of federal elections, marking the first time in our nation’s

history that the federal government has funded an election reform effort. HAVA provides

federal funding to help the states meet the law’s uniform and non-discretionary

administrative requirements, which include the following new programs and procedures: 1)

provisional voting, 2) voting information, 3) statewide voter registration lists and

identification requirements for first-time registrants, 4) administrative complaint procedures,

and 5) updated and upgraded voting equipment.

HAVA also established the U.S. Election Assistance Commission (EAC) to administer the

federal funding and to provide guidance to the states in their efforts to comply with the

HAVA administrative requirements. Section 202 directs the EAC to adopt voluntary voting

system guidelines, and to provide for the testing, certification, decertification, and

recertification of voting system hardware and software. The purpose of the guidelines is to

provide a set of specifications and requirements against which voting systems can be tested

to determine if they provide all the basic functionality, accessibility, and security capabilities

required of voting systems.

This document, the Voluntary Voting System Guidelines (referred to herein as the Guidelines

and/or VVSG), is the third iteration of national level voting system standards that has been

developed. The Federal Election Commission published the Performance and Test Standards

for Punchcard, Marksense and Direct Recording Electronic Voting Systems in 1990. This

was followed by the Voting Systems Standards in 2002.

As required by HAVA, the EAC formed the Technical Guidelines Development Committee

(TGDC) to develop an initial set of recommendations for the Guidelines. This committee of

15 experts began their work in July 2004 and submitted their recommendations to the EAC in

the 9-month timeline prescribed by HAVA. The TGDC was provided with technical support

by the National Institute for Standards and Technology (NIST), which was given nearly $3

million dollars by the EAC to complete this work.

The EAC reviewed and revised the TGDC recommendations and, as required by HAVA,

published the proposed Guidelines for a 90 day public comment period. The document was

also provided to both the Board of Advisors and the Standards Board for their review and 

comment. During the comment period the EAC conducted 3 public hearings on the 

Guidelines in New York City, Pasadena and Denver. Over 6000 comments were received

from the public and the Boards. Each of these comments was reviewed and considered by the

EAC in consultation with NIST in the development of this final version.
Version 1.0

Voluntary Voting System Guidelines

Volume II: National Certification Testing Guidelines

Overview

1

0

Purpose and Scope of the Guidelines

The purpose of the Voluntary Voting System Guidelines is to provide a set of specifications

and requirements against which voting systems can be tested to determine if they provide all

the basic functionality, accessibility and security capabilities required to ensure the integrity

of voting systems. The VVSG specifies the functional requirements, performance

characteristics, documentation requirements, and test evaluation criteria for the national

certification of voting systems. The VVSG is composed of two volumes: Volume I, Voting

System Performance Guidelines and Volume II, National Certification Testing Guidelines.

Effective Date

The 2005 Voluntary Voting System Guidelines will take effect 24 months after their final

adoption in December 2005 by the EAC. At that time, all new systems submitted for national

certification will be tested for conformance with these guidelines. In addition, if a

modification to a system qualified or certified to a previous standard is submitted for national

certification after this date, every component of the modified system will be tested against the

2005 VVSG. All previous versions of national standards will become obsolete at this time.

This effective date provision does not have any impact on the mandatory January 1, 2006,

deadline for states to comply with the HAVA Section 301 requirements.

Summary of Changes

Volume I of the Guidelines, entitled Voting System Performance Guidelines, includes new

requirements for usability, accessibility, voting system software distribution, generation of

software reference information, validation of software during voting system setup, and the

use of wireless communications. System functional requirements have been revised to

comply with HAVA Section 301 requirements. Environmental criteria have been updated.

This volume also includes requirements for a voter verifiable paper audit trail component for

direct-recording electronic voting systems for use by states that require this feature. In

addition, this volume includes an updated glossary and a conformance clause.

Volume II of the Guidelines, entitled National Certification Testing Guidelines, has been

revised to reflect the new EAC process for national certification of voting systems. This

process was initiated in 2005 and replaces the voting system qualification process conducted

by the National Association of State Election Directors (NASED) since 1994. In addition,

revisions have been made to the testing procedures to reflect new requirements for the

conduct of usability and accessibility testing. Volume II also includes an updated appendix

on procedures for testing system error rates. Terminology in both volumes has been revised

to reflect new terminology introduced by HAVA.
1

1

Version 1.0

Voluntary Voting System Guidelines

Volume II: National Certification Testing Guidelines

Overview

Volume I: Voting System Performance Guidelines Summary

Volume I, the Voting System Performance Guidelines, describes the requirements for the

electronic components of voting systems. It is intended for use by the broadest audience,

including voting system developers, manufacturers and suppliers; voting system testing labs;

state organizations that certify systems prior to procurement; state and local election officials

who procure and deploy voting systems; and public interest organizations that have an

interest in voting systems and voting system standards. It contains the following sections:

Section I describes the purpose and scope of the Voting System Performance 

Guidelines.

Section 2 describes the functional capabilities required of voting systems. This section

has been revised to reflect HAVA Section 301 requirements.

Section 3 describes new standards that make voting systems more usable and accessible

for as many eligible citizens as possible, whatever their physical abilities, language

skills, or experience with technology. This section reflects the HAVA 301 (a)(3)

accessibility requirements.

Sections 4 through 6 describe specific performance standards for election system

hardware, software, telecommunications, and security. Environmental criteria have been

updated in Section 4.

Section 7 describes voting system security requirements and includes new requirements

for voting system software distribution, generation of software reference information,

validation of software during system setup, and the use of wireless. It also includes

requirements for voter verifiable paper audit trail components for direct-recording

electronic voting systems.

Sections 8 and 9 describe requirements for vendor quality assurance and configuration

management practices and the documentation about these practices required for the

EAC certification process.

Appendix A contains a glossary of terms.

Appendix B provides a list of related standards documents incorporated into the

Guidelines by reference, documents used in the preparation of the Guidelines, and

referenced legislation.

Appendix C presents an introductory discussion of independent verification systems as

a potential concept for future voting system security design.

Appendix D contains technical guidance on color, contrast and text size adjustment for

individuals with low vision or color blindness.
Version 1.0

Voluntary Voting System Guidelines

Volume II: National Certification Testing Guidelines

Overview

xii

Volume II: National Certification Testing Guidelines Summary

Volume II, the National Certification Testing Guidelines, is a complementary document to

Volume I. Volume II provides an overview and specific detail of the national certification

testing process, which is performed by independent voting system test labs accredited by the

EAC. It is intended principally for use by vendors: test labs: and election officials who 

certify, procure, and accept voting systems. This volume contains the following sections:

Section 1 describes the purpose of the National Certification Testing Guidelines.

Section 2 provides a description of the Technical Data Package that vendors are

required to submit with their system for certification testing.

Section 3 describes the basic functionality testing requirements.

Sections 4 through 6 define the requirements for hardware, software and system

integration testing. Section 6 has been revised to reflect new requirements for usability

and accessibility testing.

Section 7 describes the required examination of vendor quality assurance and

configuration management practices.

Appendix A provides the requirements for the National Certification Test Plan that is

prepared by the voting system test lab and provided to the EAC for review.

Appendix B describes the scope and content of the National Certification Test Report

which is prepared by the test lab and delivered to the EAC along with a

recommendation for certification.

Appendix C describes the guiding principles used to design the voting system

certification testing process. It also contains a revised section on testing system error

rates.
13

National Certification Testing Guidelines

Guide to Section Locations

Section 1: Introduction................................................................................................... 1

Section 2: Description of the Technical Data Package.................................................. 19

Section 3: Functionality Testing .................................................................................... 51

Section 4: Hardware Testing.......................................................................................... 59

Section 5: Software Testing........................................................................................... 75

Section 6: System Integration Testing ........................................................................... 87

Section 7: Quality Assurance Testing............................................................................ 95

Appendix A: National Certification Test Plan............................................................... A-1

Appendix B: National Certification Test Report ........................................................... B-1

Appendix C: National Certification Test Design Criteria.............................................. C-1
1

1 Introduction

Table of Contents

1 Introduction.................................................................................................. 3

1.1 Overview of the National Certification Testing Guidelines.......... 3

1.2 Overview of the National Certification Testing Process............... 3

1.3 Testing Scope .................................................................................... 4

1.3.1 Test Categories.............................................................................. 4

1.3.1.1 Focus of Functionality Tests..................................................... 5

1.3.1.2 Focus of Hardware Tests........................................................... 5

1.3.1.3 Focus of Software Evaluation................................................... 6

1.3.1.4 Focus of System Integration Tests............................................ 6

1.3.1.5 Focus of Vendor Documentation Examination .......................... 7

1.4 Testing Sequence .............................................................................. 8

1.5 Documentation Submitted by Vendor ............................................ 9

1.6 Voting Equipment Submitted by Vendor ...................................... 9

1.7 Test Applicability.............................................................................. 9

1.7.1 General Applicability .................................................................... 10

1.7.1.1 Hardware................................................................................... 10

1.7.1.2 Software.................................................................................... 11

1.7.2 Modifications to Certified Systems............................................... 11

1.7.2.1 General Requirements for Modifications.................................. 11

1.7.2.2 Basis for Limited Testing Determinations................................ 12

1.8 Certification Test Process................................................................ 12

1.8.1 Pre-test Activities .......................................................................... 13

1.8.1.1 Initiation of Testing .................................................................. 13

1.8.1.2 Pre-test Preparation................................................................... 13

1.8.2 Certification Testing...................................................................... 14

1.8.2.1 National Certification Test Plan ............................................... 14

1.8.2.2 Certification Test Conditions.................................................... 14

1.8.2.3 Certification Test Fixtures......................................................... 15

1.8.2.4 Witness of System Build and Installation................................. 15

1.8.2.5 Certification Test Data Requirements....................................... 16

1.8.2.6 Certification Test Practices....................................................... 16

1.8.3 Post-test Activities......................................................................... 17

1.8.4 Resolution of Testing Issues.......................................................... 18
2

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1 Introduction

1.1 Overview of the National Certification Testing Guidelines

Volume II, National Certification Testing Guidelines, is a complementary document to

Volume I, Voting System Performance Guidelines. Volume I specifies the requirements that a

voting system must conform to in order to be nationally certified as acceptable for use in

federal elections. Volume II describes the testing process that is designed to provide a

documented independent verification by an accredited voting system test lab that a voting

system has been demonstrated to conform to the Volume I requirements and therefore should

receive national certification.

Volume II, National Certification Testing Guidelines, provides the specific detail about the

testing process that is needed for the accredited test labs, voting system vendors and election

officials participating in the system certification process.

Independent Accredited Test Labs: Test labs that are accredited to perform conformance

testing of voting systems will use Volume II to guide the development of test plans, the

testing of systems, and the preparation of test reports and recommendations for granting

national certification. Organizations wishing to become accredited as voting system test labs

can refer to Volume II to understand the requirements and obligations placed on an

accredited voting system test lab.

Voting System Vendors: Voting system vendors will use Volume II to guide the design,

construction, documentation, internal testing, and maintenance of voting systems. They will

also use this document to help define the responsibilities of organizations that support the

system, such as suppliers, testers and consultants.

Election Officials: Election officials will use Volume II to guide their state certification,

procurement, and acceptance processes and requirements. Certification at the state level may

entail system conformance with additional requirements beyond those required for national

certification to comply with state election laws or procedures.

1.2 Overview of the National Certification Testing Process

Certification testing encompasses the examination and testing of software; tests of hardware

under conditions simulating the intended storage, operation, transportation, and maintenance

environments; the inspection and evaluation of system documentation; and operational tests

to validate system performance and functioning under normal and abnormal conditions. The

testing also evaluates the completeness of the vendor’s developmental test program,

including the sufficiency of vendor tests conducted to demonstrate compliance with stated

system design and performance specifications, and the vendor’s documented quality
3

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

assurance and configuration management practices. The tests address individual system

components or elements, as well as the integrated system as a whole.

Beginning in 1994, the National Association of State Election Directors (NASED) began 

accrediting Independent Test Authorities for the purpose of conducting qualification testing

of voting systems. The qualification testing process was originally based on the 1990 voting

system standards and evolved to encompass the new requirements contained in the 2002

version of the standards.

The Help America Vote Act (HAVA) directs the U.S. Election Assistance Commission

(EAC) to provide for the testing, certification, decertification, and recertification of voting

system hardware and software by accredited laboratories. HAVA also introduces different 

terminology for these functions. Under the EAC process, test labs are “accredited” and

voting systems are “certified.” The term “standards” has been replaced with the term

“Guidelines.” As prescribed by HAVA, the EAC process was initially based on the 2002

Voting Systems Standards and will transition to the revised standards issued through the

2005 Voluntary Voting System Guidelines.

1.3 Testing Scope

The national certification testing process is intended to discover vulnerabilities that, should

they appear in actual election use, could result in failure to complete election operations in a

satisfactory manner. There are four focuses that guide the overall process:

 Operational accuracy in the recording and processing of voting data, as measured by

target error rate, for which the maximum acceptable error rate is no more than one in

ten million ballot positions, with a maximum acceptable error rate in the test process

of one in 500,000 ballot positions

 Operational failures or the number of unrecoverable failures under conditions

simulating the intended storage, operation, transportation, and maintenance

environments for voting systems, using an actual time-based period of processing test

ballots

 System performance and function under normal and abnormal conditions

 Completeness and accuracy of the system documentation and configuration

management records to enable purchasing jurisdictions to effectively install, test, and

operate the system

1.3.1 Test Categories

The certification test procedure is presented in several parts:

 Functionality testing

 Hardware testing

 Software evaluation
4

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

 System level integration tests, including audits

 Examination of documented vendor practices for quality assurance and for

configuration management

In practice, there may be concurrent indications of hardware and software function, or failure

to function, during certain examinations and tests. Operating tests of hardware partially

exercise the software as well and therefore supplement software testing. Security tests

exercise hardware, software and communications capabilities. Documentation review

conducted during software qualification supplements the review undertaken for system-level

testing.

Not all systems being tested are required to complete all categories of testing. For example, if

a previously certified system has had hardware modifications, the system may be subject

only to non-operating environmental stress testing of the modified component and system

level integration testing. If a system consisting of general purpose COTS hardware, or one

that was previously certified has had modifications to its software, the system is subject only

to software testing and system level integration tests, not hardware testing. However, in all

cases the system documentation and configuration management records will be examined to

confirm that they completely and accurately reflect the components and component versions

that comprise the voting system.

1.3.1.1 Focus of Functionality Tests

Functionality testing is performed to confirm the functional capabilities of a voting system.

The accredited test lab designs and performs procedures to test a voting system against the

requirements outlined in Volume I, Section 2. In order to best complement the diversity of

the voting systems industry, this part of the testing process is not rigidly defined. Although

there are basic functionality testing requirements, additions or variations in testing are

appropriate depending on the system’s use of specific technologies and configurations, the

system capabilities, and the outcomes of previous testing.

1.3.1.2 Focus of Hardware Tests

Hardware testing begins with non-operating tests that require the use of an environmental test

facility. These are followed by operating tests that are performed partly in an environmental

facility and partly in a standard accredited test laboratory or shop environment.

The non-operating tests are intended to evaluate the ability of the system hardware to 

withstand exposure to the various environmental conditions incidental to voting system

storage, maintenance, and transportation. The procedures are based on test methods

contained in Military Standards (MIL-STD) 810F, modified where appropriate, and include

such tests as: bench handling, vibration, low and high temperature, and humidity.

The operating tests involve running the system for an extended period of time under varying

temperatures and voltages. This period of operation ensures with confidence that the

hardware meets or exceeds the minimum requirements for reliability, data reading, and
5

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

processing accuracy contained in Volume I, Section 4. The procedure emphasizes equipment

operability and data accuracy; it is not an exhaustive evaluation of all system functions.

Moreover, the severity of the test conditions, in most cases, has been reduced from that 

specified in the Military Standards to reflect commercial and industrial practice.

1.3.1.3 Focus of Software Evaluation

The software tests encompass a number of interrelated examinations, involving assessment of

application source code for its compliance with the requirements spelled out in Volume I,

Section 5. Essentially, the accredited test lab will look at programming completeness,

consistency, correctness, modifiability, structure, and traceability, along with its modularity

and construction. The code inspection will be followed by a series of functional tests to

verify the proper performance of all system functions controlled by the software.

The accredited test lab may inspect COTS generated software source code in the preparation

of test plans and conduct some minimal scanning or sampling to check for embedded code or

unauthorized changes. Otherwise, the COTS source code is not subject to the full code

review and testing. For purposes of code analysis, the COTS units shall be treated as

unexpanded macros.

1.3.1.4 Focus of System Integration Tests

The functionality, hardware, and software certification tests supplement a fuller evaluation

performed by the system level integration tests. System level tests focus on these aspects

jointly, throughout the full range of system operations. They include tests of fully integrated

system components, internal and external system interfaces, usability and accessibility, and

security. During this process election management functions, ballot-counting logic, and

system capacity are exercised. The process also includes the Physical Configuration Audit

(PCA) and the Functional Configuration Audit (FCA).

The accredited test lab tests the interface of all system modules and subsystems with each

other against the vendor’s specifications. Some systems use telecommunications capabilities

as defined in Volume 1, Section 6. For those systems that do use such capabilities,

components that are located at the poll site or separate vote counting site are tested for

effective interface, accurate vote transmission, failure detection, and failure recovery. For

voting systems that use telecommunications lines or networks that are not under the control

of the vendor (e.g., public telephone networks), the accredited test lab tests the interface of 

vendor-supplied components with these external components for effective interface, vote

transmission, failure detection, and failure recovery.

The security tests focus on the ability of the system to detect, prevent, log, and recover from

a broad range of security risks as identified in Volume 1, Section 7. The range of risks tested

is determined by the design of the system and potential exposure to risk. Regardless of

system design and risk profile, all systems are tested for effective access control and physical

data security. For systems that use public telecommunications networks, to transmit election
6

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

management data or official election results (such as ballots or tabulated results), security

tests are conducted to ensure that the system provides the necessary identity-proofing,

confidentiality, and integrity of transmitted data. The tests determine if the system is capable

of detecting, logging, preventing, and recovering from types of attacks known at the time the

system is submitted for qualification. The accredited test lab may meet these testing

requirements by confirming the proper implementation of proven commercial security

software.

The interface between the voting system and its users, both voters and election officials, is a

key element of effective system operation and confidence in the system. Guidelines for

usability by individual voters with disabilities have been defined in Volume 1, Section 3.

Voting systems are tested to ensure that an accessible voting station is included in the system

configuration and that its design and operation conforms to these guidelines.

The Physical Configuration Audit (PCA) compares the voting system components submitted

for qualification to the vendor’s technical documentation and confirms that the

documentation submitted meets the requirements of the Guidelines. As part of the PCA, the

accredited test lab also witnesses the build of the executable system to ensure that the

qualified executable release is built from the tested components.

The Functional Configuration Audit (FCA) is an exhaustive verification of every system

function and combination of functions cited in the vendor’s documentation. Through use, the

FCA verifies the accuracy and completeness of the system Technical Data Package (TDP).

The various options of software counting logic that are claimed in the vendor’s

documentation shall be tested during the system-level FCA. Generic test ballots or test entry

data for DRE systems, representing particular sequences of ballot-counting events, will test

the counting logic during this audit.

1.3.1.5 Focus of Vendor Documentation Examination

The accredited test lab reviews the documentation submitted by the vendor for its

completeness and accuracy in describing the system. The accredited test lab also reviews the

documentation to evaluate the extent to which it conforms to the requirements outlined in

Volume 1, Sections 8 and 9 for vendor configuration and quality assurance practices. The

accredited test lab examines the conformance of other documentation and information

provided by the vendor with the vendor’s documented practices for quality assurance and

configuration management.

The Guidelines do not require on-site examination of the vendor’s quality assurance and

configuration management practices during the system development process. However, the

accredited test lab conducts several activities while at the vendor site to witness the system

build that enable assessment of the vendor’s quality assurance and configuration

management practices and conformance with them. These include surveys, interviews with

individuals at all levels of the development team, and examination of selected internal work

products such as system change requests and problem tracking logs.
7

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1.4 Testing Sequence

The overall testing process progresses through several stages involving pre-testing, testing, 

and post-testing activities. National certification testing involves a series of physical tests and

other examinations that are conducted in a particular sequence. The sequence is intended to

maximize overall testing effectiveness, as well as conduct testing in as efficient a manner as

possible. The accredited test lab will follow the general sequence outlined below. Test

anomalies and errors are communicated to the system vendor throughout the process.

a. Initial examination of the system and the technical documentation provided by the

vendor to ensure that all components and documentation needed to conduct testing have

been submitted, and to help determine the scope and level of effort of testing needed

b.Examination of the vendor’s Quality Assurance Program and Configuration

Management Plan

c. Development of a detailed system test plan that reflects the scope and complexity of the

system, and the status of system certification (i.e., initial certification or a recertification to incorporate modifications)

d.Code review for selected software components

e. Witnessing of a system ‘build’ conducted by the vendor to conclusively establish the

system version and components being tested

f. Operational testing of hardware components, including environmental tests, to ensure

that operational performance requirements are achieved

g. Functional and performance testing of hardware components

h. System installation testing and testing of related documentation for system installation

and diagnostic testing

i. Functional and performance testing of software components

j. Functional and performance testing of the integrated system, including testing of the

full scope of system functionality, performance tests for telecommunications and

security; and examination and testing of the System Operations Manual

k.Examination of the system maintenance manual

l. Preparation of the National Certification Test Report

m. Delivery of the National Certification Test Report to the EAC
8

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1.5 Documentation Submitted by Vendor

The vendor shall submit all the documentation necessary for the identification of the full

system configuration submitted for evaluation and for the development of an appropriate test

plan by the accredited test lab for conducting system certification testing. This

documentation collectively is referred to as the Technical Data Package (TDP). The TDP

provides information that defines the voting system design, method of operation, and related

resources. It provides a system overview and documents the system’s functionality,

hardware, software, security, test and verification specifications, operations procedures,

maintenance procedures, and personnel deployment and training requirements. It also

documents the vendor’s configuration management plan and quality assurance program. If

another version of the system was previously certified, the TDP would also include

appropriate system change notes.

1.6 Voting Equipment Submitted by Vendor

Vendors may seek to market a complete voting system or an interoperable component of a

voting system. In all instances, vendors shall submit for testing the specific system

configuration that will be offered to jurisdictions or that comprises the component to be

marketed plus the other components with which the vendor recommends that the component

be used. The system submitted for testing shall meet the following requirements:

a. The hardware submitted for certification testing shall be equivalent, in form and

function, to the actual production version of the hardware units or the COTS

hardware specified for use in the TDP

b.The software submitted for certification testing shall be the exact software that will be

used in production units

c. Engineering or developmental prototypes are not acceptable, unless the vendor can

show that the equipment to be tested is equivalent to standard production units both in

performance and construction

d.Benchmark directory listings shall be submitted for all software/firmware elements

(and associated documentation) included in the vendor’s release as they would

normally be installed upon setup and installation

1.7 Test Applicability

Certification tests are conducted for new systems seeking initial certification as well as for

modified versions of systems that have been certified.
9

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1.7.1 General Applicability

Voting system hardware, software, communications and documentation are examined and

tested to determine suitability for elections use. Examination and testing addresses the broad

range of system functionality and components, including system functionality for pre-voting, 

voting, and post-voting functions. All products custom designed for election use shall be 

tested in accordance with the applicable procedures contained in this section. COTS

hardware, system software and communications components with proven performance in

commercial applications other than elections, however, are exempted from certain portions of

the test as long as such products are not modified for use in a voting system. Compatibility of

these products with other components of the voting system shall be determined through

functional tests integrating these products with the remainder of the system.

1.7.1.1 Hardware

Specifically, the hardware test requirements shall apply in full to all equipment used in a

voting system with the exception of the following:

a. Commercially available models of general purpose information technology

equipment that have been designed to an ANSI or IEEE standard, have a documented

history of successful performance for relevant requirements of the standards, and

have demonstrated compatibility with the voting system components with which they

interface

b. Production models of special purpose information technology equipment that have a

documented history of successful performance under conditions equivalent to election

use for relevant requirements of the standards and that have demonstrated

compatibility with the voting system components with which they interface

c. Any ancillary devices that do not perform ballot definition, election database

maintenance, ballot reading, ballot data processing, or the production of an official

output report; and that do not interact with these system functions (e.g. modems used

to broadcast results to the press, printers used to generate unofficial reports, or CRTs

used to monitor the vote counting process)

This equipment shall be subject to functional and operating tests performed during software

evaluation and system level testing. However, it need not undergo hardware non-operating

tests. If the system is composed entirely of off-the-shelf hardware, then the system also shall

not be subject to the 48-hour environmental chamber segment of the hardware operating

tests.
10

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1.7.1.2 Software

Software certification is applicable to the following:

a. Application programs that control and carry out ballot processing, commencing with

the definition of a ballot, and including processing of the ballot image (either from

physical ballots or electronically activated images), and ending with the system's

access to memory for the generation of output reports

b. Specialized compilers and specialized operating systems associated with ballot

processing

c. Standard compilers and operating systems that have been modified for use in the vote

counting process

Specialized software for ballot preparation, election programming, vote recording, vote

tabulation, vote consolidation and reporting, and audit trail production shall be subjected to

code inspection. Functional testing of all these programs during software evaluation and

system-level testing shall exercise any specially tailored software off-line from the ballot

counting process (e.g. software for preparing ballots and broadcasting results).

1.7.2 Modifications to Certified Systems

Changes introduced after the system has completed certified testing will necessitate further

review.

1.7.2.1 General Requirements for Modifications

The accredited test lab will determine tests necessary to certify the modified system based on

a review of the nature and scope of changes, and other submitted information including the

system documentation, vendor test documentation, configuration management records, and

quality assurance information. Based on this review, the accredited test lab may:

a. Determine that a review of all change documentation against the baseline materials is

sufficient for recommendation for certification

b. Determine that all changes must be retested against the previously certified version.

This will include review of changes to source code, review of all updates to the TDP,

and performance of system level and functional tests

c. Determine that the scope of the changes is substantial and will require a complete

retest of the hardware, software, and/or telecommunications
11

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1.7.2.2 Basis for Limited Testing Determinations

The accredited test lab may determine that a modified system will be subject only to limited

certification testing if the vendor demonstrates that the change does not affect demonstrated

compliance with these Guidelines for:

a. Performance of voting system functions

b. Voting system security and privacy

c. Overall flow of system control

d.The manner in which ballots are defined and interpreted, or voting data are processed

Limited testing is intended to facilitate the correction of defects, the incorporation of

improvements, the enhancement of portability and flexibility, and the integration of votecounting software with other systems and election software.

1.8 Certification Test Process

The certification test process may be performed by one or more accredited test labs that

together perform the full scope of tests required. Where multiple accredited test labs are

involved, testing shall be conducted first for the voting system hardware, firmware, and

related documentation; then for the system software and communications; and finally for the

integrated system as a whole. Voting system hardware and firmware testing may be

performed by one accredited test lab independently of the other testing performed by other

accredited test labs. Testing may be coordinated across accredited test labs so that

hardware/firmware tested by one accredited test lab can be used in the overall system tests

performed by another accredited test lab.

When multiple accredited test labs are being used, the development of the National

Certification Test Plan (see Appendix A) and the National Certification Test Report (see

Appendix B) shall be coordinated by a lead accredited test lab. The lead lab is responsible for

ensuring that all testing has been performed and documented in accordance with the

Guidelines.

Whether one or more accredited test labs are used, the testing generally consists of three

phases:

 Pre-test Activities

 National Certification Testing

 National Certification Report Issuance and Post-test Activities
12

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1.8.1 Pre-test Activities

Pre-test activities include the request for initiation of testing and the pre-test preparation.

1.8.1.1 Initiation of Testing

Certification testing shall be conducted at the request of the vendor, consistent with the 

provision of the Guidelines. The vendor shall:

a. Request the performance of certification testing from among the accredited testing

laboratories

b.Enter into formal agreement with the accredited test lab for the performance of testing

c. Prepare and submit materials required for testing consistent with the requirements of

the Guidelines

Certification testing shall be conducted for the initial version of a voting system as well as for

all subsequent changes to the system prior to release for sale or for installation. As described

in Subsection 1.6.2, the nature and scope of testing for system changes or new versions shall 

be determined by the accredited test lab based on the nature and scope of the modifications to

the system and on the quality of system documentation and configuration management

records submitted by the vendor.

1.8.1.2 Pre-test Preparation

Pre-test preparation encompasses the following activities:

a. The vendor shall prepare and submit a complete TDP to the accredited test lab. The

TDP should consist of the materials described in Section 2

b.The accredited test lab shall perform an initial review of the TDP for completeness

and clarity and request additional information as required

c. The vendor shall provide additional information, if requested by the accredited test

lab

d.The vendor and accredited test lab shall enter into an agreement for the testing to be

performed by the accredited test lab in exchange for payment by the vendor

e. The vendor shall deliver to the accredited test lab all hardware and software needed to

perform testing
13

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1.8.2 Certification Testing

Certification testing encompasses the preparation of a test plan, the establishment of the

appropriate test conditions, the use of appropriate test fixtures, the witness of the system

build and installation, the maintenance of certification test data, and the evaluation of the data

resulting from tests and examinations.

1.8.2.1 National Certification Test Plan

The accredited test lab shall prepare a National Certification Test Plan to define all tests and

procedures required to demonstrate compliance with the Guidelines, including:

Verifying or checking equipment operational status by means of manufacturer operating

procedures

a. Establishing the test environment or the special environment required to perform the

test

b.Initiating and completing operating modes or conditions necessary to evaluate the

specific performance characteristic under test

c. Measuring and recording the value or range of values for the characteristic to be 

tested, demonstrating expected performance levels

d. Verifying, as above, that the equipment is still in normal condition and status after all

required measurements have been obtained

e. Confirming that documentation submitted by the vendor corresponds to the actual

configuration and operation of the system

f. Confirming that documented vendor practices for quality assurance and configuration

management comply with the Guidelines

A recommended outline for the test plan and the details of required testing are contained in

Appendix A.

1.8.2.2 Certification Test Conditions

The accredited test lab may perform the tests in any facility capable of supporting the test

environment. The following practices shall be employed:

a. Preparations for testing, arrangement of equipment, verification of equipment status,

and the execution of procedures shall be witnessed by at least one independent, 

qualified observer in the form of an accredited testing laboratory, which shall certify

that all test and data acquisition requirements have been satisfied
14

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

b.When a test is to be performed at “standard” or “ambient” conditions, this

requirement shall refer to a nominal laboratory or office environment, with a

temperature in the range of 68 to 75 degrees Fahrenheit, and prevailing atmospheric

pressure and relative humidity

c. Otherwise, all tests shall be performed at the required temperature and electrical

supply voltage, regulated within the following tolerances:

i. Temperature +/- 4 degrees F

ii. Electrical supply voltage +/- 2 voltage alternating current

1.8.2.3 Certification Test Fixtures

The accredited test lab may use test fixtures or ancillary devices to facilitate testing. These

fixtures and devices may include arrangements for automating the operation of voting

devices and the acquisition of test data:

a. For systems that use a light source as a means of detecting voter selections, the

generation of a suitable optical signal by an external device is acceptable. For systems

that rely on the physical activation of a switch, a mechanical fixture with suitable

motion generators is acceptable

b.The accredited test lab may use a simulation device, and appropriate software, to

speed up the process of testing and eliminate human error in casting test ballots,

provided that the simulation covers all voting data detection and control paths that are

used in casting an actual ballot. In the event that only partial simulation is achieved,

then an independent method and test procedure shall be used to validate the proper

operation of those portions of the system not tested by the simulator

c. If the vendor provides a means of simulating the casting of ballots, the simulation

device is subject to the same performance, reliability, and quality requirements that

apply to the voting device itself

1.8.2.4 Witness of System Build and Installation

Although most testing is conducted at facilities operated by the accredited test lab, a key

element of voting system testing shall be conducted at either the vendor site or the accredited

test lab site. The accredited test lab responsible for testing voting system software,

telecommunications, and integrated system operation (i.e., system level testing) shall witness

the final system build, encompassing hardware, software and communications, and the

version of associated records and documentation. The system elements witnessed, including

their specific versions, shall become the specific system version that is recommended for

certification.
15

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

1.8.2.5 Certification Test Data Requirements

The following test data practices shall be employed:

a. A test log of the procedure shall be maintained. This log shall identify the system and

equipment by model and serial number

b.Test environment conditions shall be noted

c. All operating steps, the identity and quantity of simulated ballots, annotations of

output reports, the elapsed time for each procedure step, and observations of

equipment performance and, in the case of non-operating hardware tests, the

condition of the equipment shall be recorded

1.8.2.6 Certification Test Practices

The accredited test lab shall conduct the examinations and tests defined in the National

Certification Test Plan such that all applicable tests identified in Volume II, National

Certification Testing Guidelines are executed to determine compliance with the voting

system requirements described in Volume I. The accredited testing laboratory shall evaluate

data resulting from examinations and tests, employing the following practices:

a. If any malfunction or data error is detected that would be classified as a relevant

failure using the criteria in Volume II, National Certification Testing Guidelines, its

occurrence, and the duration of operating time preceding it, shall be recorded for

inclusion in the analysis of data obtained from the test, and the test shall be

interrupted

b.If a malfunction is due to a defect in software, then the test shall be terminated and

system returned to the vendor for correction

c. If the malfunction is other than a software defect, and if corrective action is taken to

restore the equipment to a fully operational condition within 8 hours, then the test

may be resumed at the point of suspension

d.If the test is suspended for an extended period of time, the accredited test lab shall

maintain a record of the procedures that have been satisfactorily completed. When

testing is resumed at a later date, repetition of the successfully completed procedures

may be waived, provided that no design or manufacturing change has been made that

would invalidate the earlier test results

e. Any and all failures that occurred as a result of a deficiency shall be classified as

purged, and test results shall be evaluated as though the failure or failures had not 

occurred, if the:
16

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

i. Vendor submits a design, manufacturing, or packaging change notice to correct

the deficiency, together with test data to verify the adequacy of the change

ii. Examiner of the equipment agrees that the proposed change will correct the

deficiency

iii.Vendor certifies that the change will be incorporated into all existing and future

production units

f. If corrective action cannot be successfully taken as defined above, then the test shall

be terminated, and the equipment shall be rejected

1.8.3 Post-test Activities

Certification report issuance and post-test activities encompass the activities described

below.

a. The accredited test lab may issue interim reports to the vendor, informing the vendor

of the testing status, findings to date, and other information.

b.The accredited test lab shall prepare a National Certification Test Report that

confirms the voting system has passed the required testing. This report shall include

the date testing was completed, the specific system version addressed by the report,

the version numbers of all system elements separately identified with a version

number by the vendor, and the scope of tests conducted. A recommended outline for

the test report is contained in Appendix B.

c. Where a system is tested by multiple accredited test labs, the lead accredited test lab

shall prepare a consolidated National Certification Test Report.

d.The accredited test lab shall deliver the report to the vendor and to the EAC.

e. Upon review and acceptance of the test report, EAC shall issue a Certification

Number for the system to the vendor and to the accredited test lab. The issuance of a

Certification Number indicates that the system has been tested by the accredited test

lab for compliance with the Guidelines.

f. This number applies to the system as a whole only for the configuration and versions

of the system elements tested and identified in the National Certification Test Report.

The Certification Number does not apply to individual system components or

untested configurations.

g. The EAC Certification Number is intended for use by the states and their jurisdictions

to support state and jurisdiction processes concerning voting systems. States and their

jurisdictions shall request National Certification Test Reports based on the EAC
17

Version 1.0

Volume II: National Certification Testing Guidelines

1 Introduction

Certification Number to support their voting system certification and procurement

processes.

1.8.4 Resolution of Testing Issues

Prior to the transition of this function to the EAC, the NASED Voting Systems Board (the

Board) was responsible for resolving questions about the application of the Guidelines in the

testing of voting systems. The EAC will have a process for the accredited test labs, vendors

and election officials to request an interpretation of the Guidelines. The interpretation will be

publicly documented for reference by interested parties. The EAC will periodically assess the

interpretations provided to determine which topics should be reflected in a future version of

the Guidelines.
19

2 Description of the Technical Data Package

Table of Contents

2 Description of the Technical Data Package ................................................. 21

2.1 Scope.................................................................................................. 21

2.1.1 Content and Format ....................................................................... 21

2.1.1.1 Required Content for Initial Certification................................. 22

2.1.1.2 Required Content for System Changes and Re-certification.... 22

2.1.1.3 Format....................................................................................... 23

2.1.2 Other Uses for Documentation ...................................................... 23

2.1.3 Protection of Proprietary Information ........................................... 23

2.2 System Overview .............................................................................. 23

2.2.1 System Description........................................................................ 23

2.2.2 System Performance ...................................................................... 24

2.3 System Functionality Description ................................................... 25

2.4 System Hardware Specification ...................................................... 25

2.4.1 System Hardware Characteristics.................................................. 25

2.4.2 Design and Construction ............................................................... 26

2.5 Software Design and Specification.................................................. 27

2.5.1 Purpose and Scope......................................................................... 27

2.5.2 Applicable Documents................................................................... 27

2.5.3 Software Overview........................................................................ 27

2.5.4 Software Standards and Conventions............................................ 28

2.5.5 Software Operating Environment.................................................. 28

2.5.5.1 Hardware Environment and Constraints................................... 28

2.5.5.2 Software Environment .............................................................. 29

2.5.6 Software Functional Specification ................................................ 29

2.5.6.1 Configurations and Operating Modes....................................... 29

2.5.6.2 Software Functions................................................................... 29

2.5.7 Programming Specifications.......................................................... 30

2.5.7.1 Programming Specifications Overview .................................... 30

2.5.7.2 Programming Specifications Details......................................... 30

2.5.8 System Database............................................................................ 31

2.5.9 Interfaces ....................................................................................... 32

2.5.9.1 Interface Identification.............................................................. 32

2.5.9.2 Interface Description ................................................................ 33

2.5.10 Appendices.................................................................................... 34

2.6 System Security Specification ......................................................... 34

2.6.1 Access Control Policy ................................................................... 35

2.6.2 Access Control Measures .............................................................. 35

2.6.3 Equipment and Data Security........................................................ 35

2.6.4 Software Installation...................................................................... 35

2.6.5 Telecommunications and Data Transmission Security.................. 36

2.6.6 Other Elements of an Effective Security Program ........................ 36
20

2.7 System Test and Verification Specification.................................... 37

2.7.1 Development Test Specifications.................................................. 37

2.7.2 National Certification Test Specifications..................................... 38

2.8 System Operations Procedures ....................................................... 38

2.8.1 Introduction ................................................................................... 38

2.8.2 Operational Environment............................................................... 39

2.8.3 System Installation and Test Specification.................................... 39

2.8.4 Operational Features...................................................................... 39

2.8.5 Operating Procedures..................................................................... 40

2.8.6 Operations Support........................................................................ 41

2.8.7 Appendices .................................................................................... 41

2.9 System Maintenance Manual........................................................... 41

2.9.1 Introduction ................................................................................... 42

2.9.2 Maintenance Procedures................................................................ 42

2.9.2.1 Preventive Maintenance Procedures ......................................... 42

2.9.2.2 Corrective Maintenance Procedures......................................... 43

2.9.3 Maintenance Equipment................................................................ 43

2.9.4 Parts and Materials ........................................................................ 43

2.9.4.1 Common Standards................................................................... 43

2.9.4.2 Paper-based Systems................................................................. 44

2.9.5 Maintenance Facilities and Support .............................................. 44

2.9.6 Appendices.................................................................................... 44

2.10 Personnel Deployment and Training Requirements..................... 45

2.10.1 Personnel ....................................................................................... 45

2.10.2 Training ......................................................................................... 45

2.11 Configuration Management Plan ................................................... 46

2.11.1 Configuration Management Policy................................................ 46

2.11.2 Configuration Identification .......................................................... 46

2.11.3 Baseline and Promotion................................................................. 47

2.11.4 Configuration Control Procedures................................................. 47

2.11.5 Release Process ............................................................................. 47

2.11.6 Configuration Audits..................................................................... 48

2.11.7 Configuration Management Resources ......................................... 48

2.12 Quality Assurance Program............................................................ 48

2.12.1 Quality Assurance Policy .............................................................. 49

2.12.2 Parts and Materials Tests............................................................... 49

2.12.3 Quality Conformance Inspections ................................................. 49

2.12.4 Documentation............................................................................... 49

2.13 System Change Notes....................................................................... 49
21

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2 Description of the Technical Data Package

2.1 Scope

This subsection contains a description of vendor documentation relating to the voting system

that shall be submitted with the system as a precondition of national certification testing.

These items are necessary to define the product and its method of operation; to provide

technical and test data supporting the vendor's claims of the system's functional capabilities

and performance levels; and to document instructions and procedures governing system

operation and field maintenance. Any information relevant to the system evaluation shall be

submitted to include source code, object code, and sample output report formats.

Both formal documentation and notes of the vendor's system development process shall be

submitted for qualification tests. Documentation describing the system development process

permits assessment of the vendor's systematic efforts to develop and test the system and

correct defects. Inspection of this process also enables the design of a more precise test plan.

If the vendor's developmental test data are incomplete, the accredited test lab shall design and

conduct the appropriate tests to cover all elements of the system and to ensure conformance

with all system requirements.

2.1.1 Content and Format

The content of the Technical Data Package (TDP) is intended to provide clear, complete

descriptions of the following information about the system:

Overall system design, including subsystems, modules and the interfaces among them

Specific functional capabilities provided by the system

Performance and design specifications

Design constraints, applicable standards, and compatibility requirements

Personnel, equipment, and facility requirements for system operation, maintenance, and

logistical support

Vendor practices for assuring system quality during the system’s development and

subsequent maintenance

Vendor practices for managing the configuration of the system during development and

for modifications to the system throughout its life cycle
22

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

The vendor shall provide a list of all documents submitted controlling the design,

construction, operation, and maintenance of the system. Documents shall be listed in order of

precedence.

2.1.1.1 Required Content for Initial Certification

At minimum, the TDP shall contain the following documentation:

System configuration overview

System functionality description

System hardware specifications

Software design and specifications

System test and verification specifications 

System security specifications

User/system operations procedures 

System maintenance procedures

Personnel deployment and training requirements

Configuration management plan

Quality assurance program

System change notes

2.1.1.2 Required Content for System Changes and Recertification

For systems seeking re-certification, vendors shall submit System Change Notes as described

in Subsection 2.13, as well as current versions of all documents that have been updated to

reflect system changes.

Vendors may also submit other information relevant to the evaluation of the system, such as

test documentation, and records of the system's performance history, failure analysis and

corrective actions.
23

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.1.1.3 Format

The requirements for formatting the TDP are general in nature; specific format details are of

the vendor’s choosing. The TDP shall include a detailed table of contents for the required

documents, an abstract of each document and a listing of each of the informational sections

and appendices presented. A cross-index shall be provided indicating the portions of the 

documents that are responsive to documentation requirements for any item presented.

2.1.2 Other Uses for Documentation

Although all of the TDP documentation is required for national certification testing, some of

these same items may also be required during the state certification process and local level

acceptance testing. Therefore, it is recommended that the technical documentation required

for certification and acceptance testing be deposited in escrow.

2.1.3 Protection of Proprietary Information

The vendor shall identify all documents, or portions of documents, containing proprietary

information not approved for public release. Any person or accredited test lab receiving

proprietary information shall agree to use it solely for the purpose of analyzing and testing

the system, and shall agree to refrain from otherwise using the proprietary information or

disclosing it to any other person or agency without the prior written consent of the vendor,

unless disclosure is legally compelled.

2.2 System Overview

In the system overview, the vendor shall provide information that enables the accredited test

lab to identify the functional and physical components of the system, how the components

are structured, and the interfaces between them.

2.2.1 System Description

The system description shall include written descriptions, drawings and diagrams that

present:

A description of the functional components (or subsystems) as defined by the vendor

(e.g., environment, election management and control, vote recording, vote

conversion, reporting, and their logical relationships)

A description of the operational environment of the system that provides an overview of

the hardware, software, and communications structure
24

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

A concept of operations that explains each system function, and how the function is

achieved in the design

Descriptions of the functional and physical interfaces between subsystems and

components

Identification of all COTS hardware and software products and communications

services used in the development and/or operation of the voting system, identifying

the name, vendor, and version used for each such component, including:

Operating systems

Database software

Communications routers

Modem drivers

Dial-up networking software

Interfaces among internal components, and interfaces with external systems. For

components that interface with other components for which multiple products may be

used, the TDP shall provide an identification of:

File specifications, data objects, or other means used for information exchange

The public standard used for such file specifications, data objects, or other means

Benchmark directory listings for all software (including firmware elements) and

associated documentation included in the vendor’s release in the order in which each

piece of software would normally be installed upon system setup and installation

2.2.2 System Performance

The vendor shall provide system performance information including:

The performance characteristics of each operating mode and function in terms of

expected and maximum speed, throughput capacity, maximum volume (maximum

number of voting positions and maximum number of ballot styles supported), and

processing frequency

Quality attributes such as reliability, maintainability, availability, usability, and

portability

Provisions for safety, security, privacy, and continuity of operation

Design constraints, applicable standards, and compatibility requirements
25

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.3 System Functionality Description

The vendor shall declare the scope of the system’s functional capabilities, thereby

establishing the performance, design, test, manufacture, and acceptance context for the

system.

The vendor shall provide a listing of the system’s functional processing capabilities,

encompassing capabilities required by the Guidelines and any additional capabilities

provided by the system. This listing shall provide a simple description of each capability.

Detailed specifications shall be provided in other documentation required for the TDP.

The vendor shall organize the presentation of required capabilities in a manner that

corresponds to the structure and sequence of functional capabilities indicated in

Volume I, Section 2. The contents of Volume I, Section 2 may be used as the basis

for a checklist to indicate the specific functions provided and those not provided by

the system

Additional capabilities shall be clearly indicated. They may be presented using the same

structure as that used for required capabilities (i.e., overall system capabilities, prevoting functions, voting functions, post-voting functions), or may be presented in

another format of the vendor’s choosing

Required capabilities that may be bypassed or deactivated during installation or

operation by the user shall be clearly indicated

Additional capabilities that function only when activated during installation or operation

by the user shall be clearly indicated

Additional capabilities that normally are active but may be bypassed or deactivated

during installation or operation by the user shall be clearly indicated

2.4 System Hardware Specification

The vendor shall expand on the system overview by providing detailed specifications of the

hardware components of the system, including specifications of hardware used to support the

telecommunications capabilities of the system, if applicable.

2.4.1 System Hardware Characteristics

The vendor shall provide a detailed discussion of the characteristics of the system, indicating

how the hardware meets individual requirements defined in Volume I, Section 4, including:

Performance characteristics: This discussion addresses basic system performance

attributes and operational scenarios that describe the manner in which system functions
26

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

are invoked, describe environmental capabilities, describe life expectancy, and describe

any other essential aspects of system performance

Physical characteristics: This discussion addresses suitability for intended use,

requirements for transportation and storage, health and safety criteria, security criteria,

and vulnerability to adverse environmental factors

Reliability: This discussion addresses system and component reliability stated in terms

of the system’s operating functions, and identification of items that require special

handling or operation to sustain system reliability

Maintainability: Maintainability represents the ease with which maintenance actions

can be performed based on the design characteristics of equipment and software and the

processes the vendor and election officials have in place for preventing failures and for

reacting to failures. Maintainability includes the ability of equipment and software to

self-diagnose problems and make non-technical election workers aware of a problem.

Maintainability also addresses a range of scheduled and unscheduled events

Environmental conditions: This discussion addresses the ability of the system to

withstand natural environments, and operational constraints in normal and test

environments, including all requirements and restrictions regarding electrical service,

telecommunications services, environmental protection, and any additional facilities or

resources required to install and operate the system

2.4.2 Design and Construction

The vendor shall provide sufficient data, or references to data, to identify unequivocally the

details of the system configuration submitted for testing. The vendor shall provide a list of

materials and components used in the system and a description of their assembly into major

system components and the system as a whole. Paragraphs and diagrams shall be provided

that describe:

Materials, processes, and parts used in the system, their assembly, and the configuration

control measures to ensure compliance with the system specification

The electromagnetic environment generated by the system

Operator and voter safety considerations, and any constraints on system operations or

the use environment

Human factors considerations, including provisions for access by disabled voters
27

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.5 Software Design and Specification

The vendor shall expand on the system overview by providing detailed specifications of the

software components of the system, including software used to support the

telecommunications capabilities of the system, if applicable.

2.5.1 Purpose and Scope

The vendor shall describe the function or functions that are performed by the software

programs that comprise the system, including software used to support the

telecommunications capabilities of the system, if applicable.

2.5.2 Applicable Documents

The vendor shall list all documents controlling the development of the software and its

specifications. Documents shall be listed in order of precedence.

2.5.3 Software Overview

The vendor shall provide an overview of the software that includes the following items:

A description of the software system concept, including specific software design 

objectives, and the logic structure and algorithms used to accomplish these objectives

The general design, operational considerations, and constraints influencing the design of

the software

Identification of all software items, indicating items that were:

Written in-house

Procured and not modified

Procured and modified, including descriptions of the modifications to the software

and to the default configuration options

Additional information for each item that includes:

Item identification

General description

Software requirements performed by the item
28

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

Identification of interfaces with other items that provide data to, or receive data

from, the item

Concept of execution for the item

The vendor shall also include a certification that procured software items were obtained

directly from the manufacturer or a licensed dealer or distributor.

2.5.4 Software Standards and Conventions

The vendor shall provide information that can be used by an accredited test lab or state

certification board to support software analysis and test design. The information shall address

standards and conventions developed internally by the vendor as well as published industry

standards that have been applied by the vendor. The vendor shall provide information that

addresses the following standards and conventions:

Software System development methodology

Software design standards, including internal vendor procedures

Software specification standards, including internal vendor procedures 

Software coding standards, including internal vendor procedures

Testing and verification standards, including internal vendor procedures, that can assist

in determining the program's correctness and ACCEPT/REJECT criteria

Quality assurance standards or other documents that can be used to examine and test the

software. These documents include standards for program flow and control charts,

program documentation, test planning, and test data acquisition and reporting

2.5.5 Software Operating Environment

This section shall describe or make reference to all operating environment factors that

influence the software design.

2.5.5.1 Hardware Environment and Constraints

The vendor shall identify and describe the hardware characteristics that influence the design

of the software, such as:

The logic and arithmetic capability of the processor

Memory read-write characteristics
29

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

External memory device characteristics

Peripheral device interface hardware

Data input/output device protocols

Operator controls, indicators, and displays

2.5.5.2 Software Environment

The vendor shall identify the compilers or assemblers used in the generation of executable

code, and describe the operating system or system monitor.

2.5.6 Software Functional Specification

The vendor shall provide a description of the operating modes of the system and of software

capabilities to perform specific functions.

2.5.6.1 Configurations and Operating Modes

The vendor shall describe all software configurations and operating modes of the system,

such as ballot preparation, election programming, preparation for opening the polling place,

recording votes and/or counting ballots, closing the polling place, and generating reports. For

each software function or operating mode, the vendor shall provide:

A definition of the inputs to the function or mode (with characteristics, tolerances or

acceptable ranges, as applicable)

An explanation of how the inputs are processed

A definition of the outputs produced (again, with characteristics, tolerances, or

acceptable ranges, as applicable)

2.5.6.2 Software Functions

The vendor shall describe the software's capabilities or methods for detecting or handling:

Exception conditions

System failures

Data input/output errors
30

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

Error logging for audit record generation

Production of statistical ballot data

Data quality assessment

Security monitoring and control

2.5.7 Programming Specifications

The vendor shall provide in this section an overview of the software design, its structure, and

implementation algorithms and detailed specifications for individual software modules.

2.5.7.1 Programming Specifications Overview

This overview shall include such items as flowcharts, data flow diagrams, and other

graphical techniques that facilitate understanding of the programming specifications. This

section shall be prepared to facilitate understanding of the internal functioning of the

individual software modules. Implementation of the functions shall be described in terms of

the software architecture, algorithms, and data structures.

2.5.7.2 Programming Specifications Details

The programming specifications shall describe individual software modules and their

component units, if applicable. For each module and unit, the vendor shall provide the

following information:

Module and unit design decisions, if any, such as algorithms used

Any constraints, limitations, or unusual features in the design of the software module or

unit

The programming language used and rationale for its use, if other than the specified

module or unit language

If the software module or unit consists of, or contains, procedural commands (such as

menu selections in a database management system for defining forms and reports, online queries for database access and manipulation, input to a graphical user interface

builder for automated code generation, commands to the operating system, or shell

scripts), a list of the procedural commands and reference to user manuals or other

documents that explain them

If the software module or unit contains, receives, or outputs data, a description of its

inputs, outputs, and other data elements as applicable. (Subsection 2.5.9 describes the
31

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

requirements for documenting system interfaces.) Data local to the software module

or unit shall be described separately from data input to, or output from, the software

module or unit

If the software module or unit contains logic, the logic to be used by the software unit,

including, as applicable:

Conditions in effect within the software module or unit when its execution is

initiated

Conditions under which control is passed to other software modules or units 

Response and response time to each input, including data conversion, renaming,

and data transfer operations

Sequence of operations and dynamically controlled sequencing during the software

module’s or unit’s operation, including:

The method for sequence control

The logic and input conditions of that method, such as timing variations,

priority assignments

Data transfer in and out of memory

The sensing of discrete input signals, and timing relationships between

interrupt operations within the software module or unit

Exception and error handling

If the software module is a database, provide the information described in Subsection

2.5.8

2.5.8 System Database

The vendor shall identify and provide a diagram and narrative description of the system’s

databases, and any external files used for data input or output. The information provided shall

include for each database or external file:

The number of levels of design and the names of those levels (such as conceptual,

internal, logical, and physical)

Design conventions and standards (which may be incorporated by reference) needed to

understand the design

Identification and description of all database entities and how they are implemented

physically (e.g., tables, files)
32

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

Entity relationship diagrams and description of relationships

Details of table, record or file contents (as applicable) to include individual data

elements and their specifications, including:

Names/identifiers

Data type (alphanumeric, integer, etc.)

Size and format (such as length and punctuation of a character string)

Units of measurement (such as meters, dollars, nanoseconds)

Range or enumeration of possible values (such as 0-99)

Accuracy (how correct) and precision (number of significant digits)

Priority, timing, frequency, volume, sequencing, and other constraints, such as

whether the data element may be updated and whether business rules apply

Security and privacy constraints

Sources (setting/sending entities) and recipients (using/receiving entities)

For external files, a description of the procedures for file maintenance, management of

access privileges, and security

2.5.9 Interfaces

The vendor shall identify and provide a complete description of all internal and external

interfaces, using a combination of text and diagrams.

2.5.9.1 Interface Identification

For each interface identified in the system overview, the vendor shall:

Provide a unique identifier assigned to the interface

Identify the interfacing entities (systems, configuration items, users, etc.) by name,

number, version, and documentation references, as applicable

Identify which entities have fixed interface characteristics (and therefore impose

interface requirements on interfacing entities) and which are being developed or

modified (thus having interface requirements imposed on them)
33

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.5.9.2 Interface Description

For each interface identified in the system overview, the vendor shall provide information

that describes:

The type of interface (such as real-time data transfer, storage-and-retrieval of data) to be

implemented

Characteristics of individual data elements that the interfacing entity(ies) will provide,

store, send, access, receive, etc., such as:

Names/identifiers

Data type (alphanumeric, integer, etc.)

Size and format (such as length and punctuation of a character string)

Units of measurement (such as meters, dollars, nanoseconds)

Range or enumeration of possible values (such as 0-99)

Accuracy (how correct) and precision (number of significant digits)

Priority, timing, frequency, volume, sequencing, and other constraints, such as

whether the data element may be updated and whether business rules apply

Security and privacy constraints

Sources (setting/sending entities) and recipients (using/receiving entities)

Characteristics of communication methods that the interfacing entity(ies) will use for

the interface, such as:

Communication links/bands/frequencies/media and their characteristics

Message formatting

Flow control (such as sequence numbering and buffer allocation)

Data transfer rate, whether periodic/aperiodic, and interval between transfers

Routing, addressing, and naming conventions

Transmission services, including priority and grade

Safety/security/privacy considerations, such as encryption, user authentication,

compartmentalization, and auditing

Characteristics of protocols the interfacing entity(ies) will use for the interface, such as:
34

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

Priority/layer of the protocol

Packeting, including fragmentation and reassembly, routing, and addressing

Legality checks, error control, and recovery procedures

Synchronization, including connection establishment, maintenance, termination

Status, identification, and any other reporting features

Other characteristics, such as physical compatibility of the interfacing entity(ies) (such

as dimensions, tolerances, loads, voltages and plug compatibility)

2.5.10 Appendices

The vendor may provide descriptive material and data supplementing the various sections of

the body of the Software Specifications. The content and arrangement of appendices shall be

at the discretion of the vendor. Topics recommended for amplification or treatment in

appendix form include:

Glossary: A listing and brief definition of all software module names and variable

names, with reference to their locations in the software structure. Abbreviations,

acronyms, and terms should be included, if they are either uncommon in data

processing and software development or are used in an unorthodox semantic

References: A list of references to all related vendor documents, data, standards, and

technical sources used in software development and testing

Program Analysis: The results of software configuration analysis algorithm analysis

and selection, timing studies, and hardware interface studies that are reflected in the

final software design and coding

2.6 System Security Specification

Vendors shall submit a system security specification that addresses the security requirements

of Volume I, Section 7. This specification shall describe the level of security provided by the

system in terms of the specific security risks addressed by the system, the means by which

each risk is addressed, the process used to test and verify the effective operation of security

capabilities and, for systems that use public telecommunications networks as defined in

Volume I, Section 6, the means used to keep the security capabilities of the system current to

respond to the evolving threats against these systems.

Information provided by the vendor in this section of the TDP may be duplicative of

information required by other sections. Vendors may cross reference to information provided
35

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

in other sections provided that the means used provides a clear mapping to the requirements

of this section.

Information submitted by the vendor shall be used to assist in developing and executing the

system certification test plan. The Security Specification shall contain the sections identified

below.

2.6.1 Access Control Policy

The vendor shall specify the features and capabilities of the access control policy 

recommended to purchasing jurisdictions to provide effective voting system security. The

access control policy shall address the general features and capabilities and individual access

privileges indicated in Volume I, Subsection 7.2.

2.6.2 Access Control Measures

The vendor shall provide a detailed description of all system access control measures and

mandatory procedures designed to permit access to system states in accordance with the

access policy, and to prevent all other types of access to meet the specific requirements of

Volume I, Subsection 7.2.

The vendor also shall define and provide a detailed description of the methods used to

preclude unauthorized access to the access control capabilities of the system itself.

2.6.3 Equipment and Data Security

The vendor shall provide a detailed description of system capabilities and mandatory

procedures for purchasing jurisdictions to prevent disruption of the voting process and

corruption of voting data to meet the specific requirements of Volume I, Subsection 7.3. This

information shall address measures for polling place security and central count location

security.

2.6.4 Software Installation

The vendor shall provide a detailed description of the system capabilities and mandatory

procedures for purchasing jurisdictions to ensure secure software (including firmware)

installation to meet the specific requirements of Volume I, Subsection 7.4. This information

shall address software installation for all system components.
36

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.6.5 Telecommunications and Data Transmission Security

The vendor shall provide a detailed description of the system capabilities and mandatory

procedures for purchasing jurisdictions to ensure secure data transmission to meet the

specific requirements of Volume I, Subsection 7.5:

For all systems, this information shall address access control, and prevention of data

interception

For systems that use public communications networks as defined in Volume I, Section

6, this information shall also include:

i. Capabilities used to provide protection against threats to third party products

and services

ii. Policies and processes used by the vendor to ensure that such protection is 

updated to remain effective over time

iii. Policies and procedures used by the vendor to ensure that current versions of

such capabilities are distributed to user jurisdictions and are installed

effectively by the jurisdiction

iv. A detailed description of the system capabilities and procedures to be employed

by the jurisdiction to diagnose the occurrence of a denial of service attack, to

use an alternate method of voting, to determine when it is appropriate to resume

voting over the network, and to consolidate votes cast using the alternate

method

v. A detailed description of all activities to be performed in setting up the system

for operation that are mandatory to ensure effective system security, including

testing of security before an election

vi. A detailed description of all activities that should be prohibited during system

setup and during the timeframe for voting operations, including both the hours

when polls are open and when polls are closed

2.6.6 Other Elements of an Effective Security Program

The vendor shall provide a detailed description of the following additional procedures

required for use by the purchasing jurisdiction:

Administrative and management controls for the voting system and election

management, including access controls

Internal security procedures, including operating procedures for maintaining the

security of the software for each system function and operating mode
37

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

Adherence to, and enforcement of, operational procedures (e.g., effective password

management)

Physical facilities and arrangements

Organizational responsibilities and personnel screening

This documentation shall be prepared such that these requirements can be integrated by the

jurisdiction into local administrative and operating procedures.

2.7 System Test and Verification Specification

The vendor shall provide test and verification specifications for: 

Development test specifications

National certification test specifications

2.7.1 Development Test Specifications

The vendor shall describe the plans, procedures, and data used during software development

and system integration to verify system logic correctness, data quality, and security. This

description shall include:

Test identification and design, including:

Test structure

Test sequence or progression 

Test conditions

Standard test procedures, including any assumptions or constraints

Special purpose test procedures including any assumptions or constraints

Test data; including the data source, whether it is real or simulated, and how test data

are controlled

Expected test results

Criteria for evaluating test results

Additional details for these requirements are provided by MIL-STD-498, Software Test Plan

and Software Test Description. In the event that test data are not available, the accredited test
38

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

lab shall design test cases and procedures equivalent to those ordinarily used during product

verification.

2.7.2 National Certification Test Specifications

The vendor shall provide specifications for verification and validation of overall software

performance. These specifications shall cover:

Control and data input/output 

Acceptance criteria 

Processing accuracy

Data quality assessment and maintenance

Ballot interpretation logic

Exception handling 

Security

Production of audit trails and statistical data

The specifications shall identify procedures for assessing and demonstrating the suitability of

the software for election use.

2.8 System Operations Procedures

This documentation shall provide all information necessary for system use by all personnel

who support pre-election and election preparation, polling place activities and central

counting activities, as applicable, with regard to all system functions and operations

identified in Subsection 2.3 above. The nature of the instructions for operating personnel will 

depend upon the overall system design and required skill level of system operations support

personnel.

The system operations procedures shall contain all information that is required for the

preparation of detailed system operating procedures, and for operator training, as described

below.

2.8.1 Introduction

The vendor shall provide a summary of system operating functions and modes, in sufficient

detail to permit understanding of the system's capabilities and constraints. The roles of
39

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

operating personnel shall be identified and related to the operating modes of the system.

Decision criteria and conditional operator functions (such as error and failure recovery

actions) shall be described.

The vendor shall also list all reference and supporting documents pertaining to the use of the

system during election operations.

2.8.2 Operational Environment

The vendor shall describe the system environment, and the interface between the user or

operator and the system. The vendor shall identify all facilities, furnishings, fixtures, and

utilities that will be required for equipment operations, including equipment that operates at

the:

Polling place

Central count facility

Other locations

2.8.3 System Installation and Test Specification

The vendor shall provide specifications for validation of system installation, acceptance, and

readiness. These specifications shall address all components of the system and all locations

of installation (e.g., polling place, central count facility), and shall address all elements of

system functionality and operations identified in Subsection 2.3 above, including:

Pre-voting functions 

Voting functions 

Post-voting functions 

General capabilities

These specifications also serve to provide guidance to the procuring agency in developing its

acceptance test plan and procedures according to the agency's contract provisions, and the

election laws of the state.

2.8.4 Operational Features

The vendor shall provide documentation of system operating features that meets the

following requirements:
40

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

A detailed description of all input, output, control, and display features accessible to the

operator or voter

Examples of simulated interactions to facilitate understanding of the system and its

capabilities

Sample data formats and output reports

Illustrate and describe all status indicators and information messages

2.8.5 Operating Procedures

The vendor shall provide documentation of system operating procedures that meets the

following requirements:

Provides a detailed description of procedures required to initiate, control, and verify

proper system operation

Provides procedures that clearly enable the operator to assess the correct flow of system

functions (as evidenced by system-generated status and information messages)

Provides procedures that clearly enable the operator to intervene in system operations to

recover from an abnormal system state

Defines and illustrates the procedures and system prompts for situations where operator

intervention is required to load, initialize, and start the system

Defines and illustrates procedures to enable and control the external interface to the

system operating environment if supporting hardware and software are involved.

Such information also shall be provided for the interaction of the system with other

data processing systems or data interchange protocols

Provides administrative procedures and off-line operator duties (if any) if they relate to

the initiation or termination of system operations, to the assessment of system status,

or to the development of an audit trail

Supports successful ballot and program installation and control by election officials,

provides a detailed work plan or other form of documentation providing a schedule

and steps for the software and ballot installation, which includes a table outlining the

key dates, events and deliverables

Supports diagnostic testing, specifies diagnostic tests that may be employed to identify

problems in the system, verifies the correction of maintenance problems; and isolates

and diagnoses faults from various system states
41

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.8.6 Operations Support

The vendor shall provide documentation of system operating procedures that meets the

following requirements:

Defines the procedures required to support system acquisition, installation, and

readiness testing. These procedures may be provided by reference, if they are

contained either in the system hardware specifications, or in other vendor

documentation

Describes procedures for providing technical support, system maintenance and

correction of defects, and for incorporating hardware upgrades and new software

releases

2.8.7 Appendices

The vendor may provide descriptive material and data supplementing the various sections of

the body of the System Operations Manual. The content and arrangement of appendices shall

be at the discretion of the vendor. Topics recommended for discussion include:

Glossary: A listing and brief definition of all terms that may be unfamiliar to persons

not trained in either voting systems or computer operations

References: A list of references to all vendor documents and to other sources related to

operation of the system

Detailed Examples: Detailed scenarios that outline correct system responses to faulty

operator input; Alternative procedures may be specified depending on the system state

Manufacturer's Recommended Security Procedures: This appendix shall contain

the security procedures that are to be executed by the system operator

2.9 System Maintenance Manual

The system maintenance procedures shall provide information in sufficient detail to support

election workers, information systems personnel, or maintenance personnel in the adjustment

or removal and replacement of components or modules in the field. Technical documentation

needed solely to support the repair of defective components or modules ordinarily done by

the manufacturer or software developer is not required.

Recommended service actions to correct malfunctions or problems shall be discussed, along

with personnel and expertise required to repair and maintain the system; and equipment,

materials, and facilities needed for proper maintenance. This manual shall include the

sections listed below.
42

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.9.1 Introduction

The vendor shall describe the structure and function of the equipment (and related software)

for election preparation, programming, vote recording, tabulation, and reporting in sufficient

detail to provide an overview of the system for maintenance, and for identification of faulty

hardware or software. The description shall include a concept of operations that fully

describes such items as:

The electrical and mechanical functions of the equipment

How the processes of ballot handling and reading are performed (paper-based systems)

How vote selection and casting of the ballot are performed (DRE systems);

How transmission of data over a network is performed (DRE systems, where

applicable)

How data are handled in the processor and memory units

How data output is initiated and controlled

How power is converted or conditioned

How test and diagnostic information is acquired and used

2.9.2 Maintenance Procedures

The vendor shall describe preventive and corrective maintenance procedures for hardware

and software.

2.9.2.1 Preventive Maintenance Procedures

The vendor shall identify and describe:

All required and recommended preventive maintenance tasks, including software tasks

such as software backup, database performance analysis, and database tuning

Number and skill levels of personnel required for each task

Parts, supplies, special maintenance equipment, software tools, or other resources

needed for maintenance

Any maintenance tasks that must be coordinated with the vendor or a third party (such

as coordination that may be needed for off-the-shelf items used in the system)
43

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.9.2.2 Corrective Maintenance Procedures

The vendor shall provide fault detection, fault isolation, correction procedures, and logic

diagrams for all operational abnormalities identified by design analysis and operating

experience.

The vendor shall identify specific procedures to be used in diagnosing and correcting

problems in the system hardware (or user-controlled software). Descriptions shall include:

Steps to replace failed or deficient equipment

Steps to correct deficiencies or faulty operations in software

Modifications that are necessary to coordinate any modified or upgraded software with

other software modules

The number and skill levels of personnel needed to accomplish each procedure

Special maintenance equipment, parts, supplies, or other resources needed to

accomplish each procedure

Any coordination required with the vendor, or other party, for off the shelf items

2.9.3 Maintenance Equipment

The vendor shall identify and describe any special purpose test or maintenance equipment

recommended for fault isolation and diagnostic purposes.

2.9.4 Parts and Materials

Vendors shall provide detailed documentation of parts and materials needed to operate and

maintain the system. Additional requirements apply for paper-based systems.

2.9.4.1 Common Standards

The vendor shall provide a complete list of approved parts and materials needed for

maintenance. This list shall contain sufficient descriptive information to identify all parts by:

Type 

Size

Value or range
44

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

Manufacturer's designation

Individual quantities needed

Sources from which they may be obtained

2.9.4.2 Paper-based Systems

For marking devices manufactured by multiple external sources, the vendor shall provide a

listing of sources and model numbers that are compatible with the system.

The TDP shall specify the required paper stock, size, shape, opacity, color, watermarks, field

layout, orientation, size and style of printing, size and location of punch or mark fields used

for vote response fields and to identify unique ballot formats, placement of alignment marks,

ink for printing, and folding and bleed-through limitations for preparation of ballots that are

compatible with the system.

2.9.5 Maintenance Facilities and Support

The vendor shall identify all facilities, furnishings, fixtures, and utilities that will be required

for equipment maintenance. In addition, vendors shall specify the assumptions made with

regard to any parameters that impact the mean time to repair. These factors shall include at a

minimum:

Recommended number and locations of spare devices or components to be kept on hand

for repair purposes during periods of system operation

Recommended number and locations of qualified maintenance personnel who need to

be available to support repair calls during system operation

Organizational affiliation (i.e., jurisdiction, vendor) of qualified maintenance personnel

2.9.6 Appendices

The vendor may provide descriptive material and data supplementing the various sections of

the body of the System Maintenance Manual. The content and arrangement of appendices

shall be at the discretion of the vendor. Topics recommended for amplification or treatment

in appendices include:

Glossary: A listing and brief definition of all terms that may be unfamiliar to persons

not trained in either voting systems or computer maintenance

References: A list of references to all vendor documents and other sources related to

maintenance of the system
45

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

Detailed Examples: Detailed scenarios that outline correct system responses to every

conceivable faulty operator input; alternative procedures may be specified depending on

the system state

Maintenance and Security Procedures: This appendix shall contain technical

illustrations and schematic representations of electronic circuits unique to the system

2.10 Personnel Deployment and Training Requirements

The vendor shall describe the personnel resources and training required for a jurisdiction to

operate and maintain the system.

2.10.1 Personnel

The vendor shall specify the number of personnel and skill levels required to perform each of

the following functions:

Pre-election or election preparation functions (e.g., entering an election, contest and

candidate information; designing a ballot; generating pre-election reports

System operations for voting system functions performed at the polling place

System operations for voting system functions performed at the central count facility

Preventive maintenance tasks

Diagnosis of faulty hardware or software 

Corrective maintenance tasks

Testing to verify the correction of problems

A description shall be presented of which functions may be carried out by user personnel,

and those that must be performed by vendor personnel.

2.10.2 Training

The vendor shall specify requirements for the orientation and training of the following

personnel:

Poll workers supporting polling place operations

System support personnel involved in election programming
46

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

User system maintenance technicians

Network/system administration personnel (if a network is used)

Information systems personnel

Vendor personnel

2.11 Configuration Management Plan

Vendors shall submit a Configuration Management Plan that addresses the configuration

management requirements of Volume I, Section 9. This plan shall describe all policies,

processes, and procedures employed by the vendor to carry out these requirements.

Information submitted by the vendor shall be used by the accredited test lab to assist in

developing and executing the system certification test plan. This information is particularly

important to support the design of test plans for system modifications. A well-organized,

robust and detailed Configuration Management Plan will enable the accredited test lab to

more readily determine the nature and scope of tests needed to fully test the modifications.

The Configuration Management Plan shall contain the sections identified below.

2.11.1 Configuration Management Policy

The vendor shall provide a description of its organizational policies for configuration

management, addressing the specific requirements of Volume I, Subsection 9.2. These

requirements pertain to:

Scope and nature of configuration management program activities

Breadth of application of vendor’s policy and practices to the voting system

2.11.2 Configuration Identification

The vendor shall provide a description of the procedures and naming conventions used to

address the specific requirements of Volume I, Subsection 9.3. These requirements pertain

to:

Classifying configuration items into categories and subcategories

Uniquely numbering or otherwise identifying configuration items

Naming configuration items
47

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.11.3 Baseline and Promotion

The vendor shall provide a description of the procedures and naming conventions used to

address the specific requirements of Volume I, Subsection 9.4. These requirements

pertain to:

Establishing a particular instance of a system component as the starting baseline

Promoting subsequent instances of a component to baseline throughout the system

development process for the first complete version of the system submitted for testing

Promoting subsequent instances of a component to baseline status as the component is

maintained throughout its life cycle until system retirement (i.e., the system is no

longer sold or maintained)

2.11.4 Configuration Control Procedures

The vendor shall provide a description of the procedures used by the vendor to approve and 

implement changes to a configuration item to prevent unauthorized additions, changes, or

deletions to address the specific requirements of Volume I, Subsection 9.5. These

requirements pertain to:

Developing and maintaining internally developed items

Developing and maintaining third party items

Resolving internally identified defects

Resolving externally identified and reported defects

2.11.5 Release Process

The vendor shall provide a description of the contents of a system release, and the procedures

and related conventions by which the vendor installs, transfers, or migrates the system to

accredited voting system testing laboratories and customers to address the specific

requirements of Volume I, Subsection 9.6. These requirements pertain to:

A first release of the system to an accredited test lab

A subsequent maintenance or upgrade release of a system, or particular components, to

an accredited test lab

The initial delivery and installation of the system to a customer
48

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

A subsequent maintenance or upgrade release of a system, or particular components, to

a customer

2.11.6 Configuration Audits

The vendor shall provide a description of the procedures and related conventions for the two

audits required by Volume I, Subsection 9.7. These requirements pertain to:

Physical configuration audit that verifies the voting system components submitted for

certification testing to the vendor’s technical documentation

Functional configuration audit that verifies the system performs all the functions

described in the system documentation

2.11.7 Configuration Management Resources

The vendor shall provide a description of the procedures and related conventions for

maintaining information about configuration management tools required by Volume I,

Subsection 9.8. These requirements pertain to information regarding:

Specific tools used, current version, and operating environment

Physical location of the tools, including designation of computer directories and files

Procedures and training materials for using the tools

2.12 Quality Assurance Program

Vendors shall submit a Quality Assurance Program that addresses the quality assurance

requirements of Volume I, Section 8. This plan shall describe all policies, processes, and

procedures employed by the vendor to ensure the overall quality of the system for its initial 

development and release and for subsequent modifications and releases. This information is

particularly important to support the design of test plans by the accredited test lab. A wellorganized, robust and detailed Quality Assurance Program will enable the accredited test lab

to more readily determine the nature and scope of tests needed to test the system

appropriately. The Quality Assurance Program shall, at a minimum, address the topics

indicated below.
49

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

2.12.1 Quality Assurance Policy

The vendor shall provide a description of its organizational policies for quality assurance,

including:

Scope and nature of Quality Assurance activities

Breadth of application of vendor’s policy and practices to the voting system

2.12.2 Parts and Materials Tests

The vendor shall provide a description of its practices for parts and materials tests and

examinations that meet the requirements of Volume I, Subsection 8.5.

2.12.3 Quality Conformance Inspections

The vendor shall provide a description of its practices for quality conformance inspections

that meet the requirements of Volume I, Subsection 8.6. For each test performed, the record

of tests provided shall include:

Test location

Test date

Individual who conducted the test

Test outcomes

2.12.4 Documentation

The vendor shall provide a description of its practices for documentation of the system and

system development process that meet the requirements of Volume I, Subsection 8.7.

2.13 System Change Notes

Vendors submitting modifications for a system that has been tested previously and received

national certification shall submit system change notes. These will be used by the accredited

test lab to assist in developing and executing the test plan for the modified system. The

system change notes shall include the following information:

Summary description of the nature and scope of the changes, and reasons for each

change
50

Version 1.0

Volume II: National Certification Testing Guidelines

2 Description of the Technical Data Package

A listing of the specific changes made, citing the specific system configuration items

changed and providing detailed references to the documentation sections changed

The specific sections of the documentation that are changed (or completely revised

documents, if more suitable to address a large number of changes)

Documentation of the test plan and procedures executed by the vendor for testing the

individual changes and the system as a whole, and records of test results
51

3 Functionality Testing

Table of Contents

3 Functionality Testing ................................................................................... 53

3.1 Scope.................................................................................................. 53

3.2 Breadth of Functionality Testing .................................................... 53

3.2.1 Basic Functionality Testing Requirements.................................... 53

3.2.2 Testing to Reflect Technologies.................................................... 54

3.2.3 Testing to Reflect Additional Capabilities.................................... 54

3.2.4 Testing to Reflect Previously Tested Capabilities......................... 54

3.3 General Test Sequence..................................................................... 55

3.3.1 Testing in Parallel with Precinct Count Systems........................... 55

3.3.2 Testing in Parallel with Central Count Systems............................ 56

3.4 Functionality Testing for Accessibility ........................................... 57

3.5 Testing for Systems that Operate on Personal Computers .......... 57
52

Version 1.0

Volume II: National Certification Testing Guidelines

3 Functionality Testing

3 FunctionalityTesting

3.1 Scope

This section contains a description of the testing to be performed to confirm the functional

capabilities of a voting system submitted for national certification. It describes the scope and

basis for functionality testing, outlines the general sequence of tests within the overall test

process, and provides guidance on testing for accessibility.

3.2 Breadth of Functionality Testing

In order to best complement the diversity of the voting systems industry, the certification

testing process is not rigidly defined. Although there are basic functionality testing

requirements, additions or variations in testing are appropriate to the use of specific

technologies and configurations, system capabilities, and the outcomes of previous testing.

3.2.1 Basic Functionality Testing Requirements

The accredited test lab shall design and perform procedures to test a voting system against

the functional requirements outlined in Volume I, Section 2. Test procedures shall be

designed and performed that address:

Overall system capabilities

Pre-voting functions 

Voting functions

Post-voting functions 

System maintenance

Transportation and storage

The specific procedures to be used shall be identified in the National Certification Test Plan

prepared by the accredited test lab. These procedures may replicate testing performed by the

vendor and documented in the vendor’s TDP, but shall not rely on vendor testing as a

substitute for independent functionality testing.

Recognizing variations in system design and the technologies employed by different vendors,

the accredited test lab shall design test procedures that account for such variations and reflect

the system-specific functional capabilities in Volume I, Section 2.
53

Version 1.0

Volume II: National Certification Testing Guidelines

3 Functionality Testing

3.2.2 Testing to Reflect Technologies

Voting systems are not designed according to a standard design template. Instead, system

design reflects the vendor’s selections from a variety of technologies and design

configurations. Such variation is recognized in the definitions of voting systems in Volume

I, Section 1, and serves as the basis for delineating various functional capability

requirements.

Functional capabilities will vary according to the relative complexity of a system and the

manner in which the system integrates various technologies. Therefore, the testing procedure

designed and performed for a particular system shall reflect the specific technologies and

design configurations used by that system.

3.2.3 Testing to Reflect Additional Capabilities

The requirements for voting system functionality provided by Volume I, Section 2 reflect a

minimum set of capabilities. Vendors may, and often do, provide additional capabilities in

systems in order to respond to the requirements of individual states. These additional

capabilities shall be identified by the vendor within the TDP, as described in Volume II,

Section 2. Based on this information, the accredited test lab shall design and perform system

functionality testing for these additional functional capabilities.

3.2.4 Testing to Reflect Previously Tested Capabilities

The required functional capabilities of voting systems defined in Volume I, Section 2 reflect

a broad range of system functionality needed to support the full life cycle of an election,

including post election activities. Many systems submitted for certification are designed to

address this scope, and are to be tested accordingly.

However, some new systems using a combination of new subsystems or system components

interfaced with the components of a previously certified system. For example, a vendor can

submit a voting system certification testing that has a new DRE voting device, but that

integrates the election management component from a previously certified system.

In this situation, the vendor shall identify in the TDP the functional capabilities supported by

new subsystems/components and those supported by subsystems/components taken from a

previously certified system. The vendor shall indicate in its system design documentation

and configuration management records the scope and nature of any modifications made to

the re-used subsystems or components. This will assist the accredited test lab to develop

efficient test procedures that rely in part on the results of testing of the previously certified

subsystems or components.

In this situation the accredited test lab may design and perform a test procedure that draws on

the results of testing performed previously on re-used subsystems or components. However,
54

Version 1.0

Volume II: National Certification Testing Guidelines

3 Functionality Testing

irrespective of previous testing performed, the scope of testing shall include certain

functionality tests:

All functionality performed by new subsystems/modules

All functionality performed by modified subsystems/modules

Functionality that is accomplished using any interfaces to new modules, or that shares

inputs or outputs from new modules

All functionality related to vote tabulation and election results reporting

All functionality related to audit trail maintenance

3.3 General Test Sequence

There is no required sequence for performing the system certification tests. For a system not

previously certified, the accredited test lab may perform tests using generic test ballots, and

schedule the tests in a convenient order, provided that prerequisite conditions for each test

have been satisfied before the test is initiated.

Regardless of the sequence of testing used, the full certification testing process shall include

functionality testing for all system functions of a voting system. Generally, in depth

functionality testing will follow testing of the system hardware and the source code review of

the software. The accredited test lab will usually conduct functionality testing as an integral

element of the system integration testing described in Section 6.

Some functionality tests for the voting functions defined in Volume I, Section 2.may be

performed as an integral part of hardware testing, enabling a more efficient testing process.

Ballots processed and counted during hardware operating tests for precinct count and central 

count systems may serve to satisfy part of the functionality testing, provided that the ballots

were cast using a test procedure that is equivalent to the procedures indicated below.

3.3.1 Testing in Parallel with Precinct Count Systems

For testing voting functions defined in Volume I, Sections 2, the following procedures shall

be performed during the functionality tests of voting equipment and precinct counting

equipment.

The procedure to prepare election programs shall:

Verify resident firmware, if any

Prepare software (including firmware) to simulate all ballot format and logic

options for which the system will be used
55

Version 1.0

Volume II: National Certification Testing Guidelines

3 Functionality Testing

Verify program memory device content

Obtain and design test ballots with formats and voting patterns sufficient to verify

performance of the test election programs

The procedures to program precinct ballot counters shall:

Install program and data memory devices, or verify presence if resident

Verify operational status of hardware as specified in Volume II, Section 4

The procedures to simulate opening of the polls shall:

Perform procedures required to prepare hardware for election operations

Obtain "zero" printout or other evidence that data memory has been cleared

Verify audit log of pre-election operations

Perform procedure required to open the polling place and enable ballot counting

The procedure to simulate counting ballots shall cast test ballots in a number sufficient

to demonstrate proper processing, error handling, and generation of audit data as

specified in Volume I, Sections 2 and 5

The procedure to simulate closing of polls shall:

Perform hardware operations required to disable ballot counting and close the polls

Obtain data reports and verify correctness

Obtain audit log and verify correctness

These procedures need not be performed in the sequence listed, provided the necessary

precondition of each procedure has been met.

3.3.2 Testing in Parallel with Central Count Systems

For testing voting functions defined in Volume I, Sections 2, the following procedures shall

be performed during the functional tests.

The procedure to prepare election programs shall:

Verify resident firmware, if any

Prepare software (including firmware) to simulate all ballot format and logic

options for which the system will be used, and to enable simulation of counting
56

Version 1.0

Volume II: National Certification Testing Guidelines

3 Functionality Testing

ballots from at least 10 polling places or precincts

Verify program memory device content

Procure test ballots with formats, voting patterns, and format identifications

sufficient to verify performance of the test election programs

The procedure to simulate counting ballots shall count test ballots in a number sufficient

to demonstrate proper processing, error handling, and generation of audit data as

specified in Volume I, Sections 2 and 5

The procedure to simulate election reports shall:

Obtain reports at polling places or precinct level

Obtain consolidated reports

Provide query access, if this is a feature of the system

Verify correctness of all reports and queries

Obtain audit log and verify correctness

They need not be performed in the sequence listed, provided the necessary preconditions of

each procedure have been met.

3.4 Functionality Testing for Accessibility

Volume I, Section 4 prescribes the requirements for voting system accessibility to satisfy the

provisions of HAVA 301(a)(4) and 241(b)(5). To demonstrate conformance to these

requirements, vendors shall conduct summative usability tests of accessible voting equipment

with blind and visually impaired individuals and individuals lacking fine motor control. A 

description of the testing performed, the population of test subjects participating, and the

results shall be documented using the Common Industry Format (CIF) by the vendor and

submitted as part of the Technical Data Package. The test labs shall review this information

during the system certification documentation review.

3.5 Testing for Systems that Operate on Personal Computers

For systems intended to use non-standard voting devices, such as a personal computer,

provided by the local jurisdiction, the accredited test lab shall conduct functionality tests

using hardware provided by the vendor that meets the minimum configuration specifications

defined by the vendor.
57

Version 1.0

Volume II: National Certification Testing Guidelines

3 Functionality Testing

Section 4 provides additional information on hardware to be used to conduct functionality

testing of such voting devices, as well as hardware to be used to conduct security testing and

other forms of testing.
59

4 Hardware Testing

Table of Contents

4 Hardware Testing......................................................................................... 61

4.1 Scope.................................................................................................. 61

4.2 Basis of Hardware Testing............................................................... 61

4.2.1 Testing Focus and Applicability.................................................... 61

4.2.2 Hardware Provided by Vendor...................................................... 62

4.3 Test Conditions................................................................................. 62

4.4 Test Log Data Requirements........................................................... 62

4.5 Test Fixtures ..................................................................................... 63

4.6 Non-operating Environmental Tests .............................................. 63

4.6.1 General .......................................................................................... 63

4.6.1.1 Pretest Data............................................................................... 64

4.6.1.2 Preparation for Test .................................................................. 64

4.6.1.3 Mechanical Inspection and Repair............................................ 64

4.6.1.4 Electrical Inspection and Adjustment....................................... 64

4.6.1.5 Operational Status Check ......................................................... 65

4.6.1.6 Failure Criteria.......................................................................... 65

4.6.2 Bench Handling Test ..................................................................... 65

4.6.2.1 Applicability ............................................................................. 65

4.6.2.2 Procedure .................................................................................. 66

4.6.3 Vibration Test................................................................................ 66

4.6.3.1 Applicability ............................................................................. 66

4.6.3.2 Procedure .................................................................................. 66

4.6.4 Low Temperature Test .................................................................. 67

4.6.4.1 Applicability ............................................................................. 67

4.6.4.2 Procedure .................................................................................. 67

4.6.5 High Temperature Test.................................................................. 67

4.6.5.1 Applicability ............................................................................. 67

4.6.5.2 Procedure .................................................................................. 67

4.6.6 Humidity Test................................................................................ 68

4.6.6.1 Applicability ............................................................................. 68

4.6.6.2 Procedure .................................................................................. 68

4.7 Environmental Tests, Operating..................................................... 69

4.7.1 Temperature and Power Variation Tests ....................................... 69

4.7.1.1 Data Accuracy .......................................................................... 70

4.7.2 Maintainability Test....................................................................... 71

4.7.3 Reliability Test .............................................................................. 71

4.7.4 Availability Test ............................................................................ 71

4.8 Other Environmental Tests............................................................. 72
60

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

4 HardwareTesting

4.1 Scope

This section contains a description of the testing to be performed to confirm the proper

functioning of the hardware components of a voting system. It describes the scope and basis

for functionality testing, required test conditions for conducting hardware testing, guidance

for the use of test fixtures, test log data requirements, and test practices for specific nonoperating and operating environmental tests.

4.2 Basis of Hardware Testing

This section addresses the focus and applicability of hardware testing and specifies the

vendor’s obligations to produce hardware to conduct such tests.

4.2.1 Testing Focus and Applicability

The accredited test lab shall design and perform procedures that test the voting system

hardware requirements identified in Volume I, Section 4. Test procedures shall be designed

and performed for both operating and non-operating environmental tests:

Operating environmental tests apply to the entire system, including hardware

components that are used as part of the voting system telecommunications capability

Non-operating tests apply to those elements of the system that are intended for use at

poll site voting locations, such as voting machines and precinct counters. These tests

address environmental conditions that may be encountered by the voting system

hardware at the voting location itself, or while in storage or transit to or from the poll

site

Additionally, compatibility of this equipment with the voting system environment shall be

determined through functional tests integrating the standard product with the remainder of

the system.

All hardware components that are custom-designed for election use shall be tested in

accordance with the applicable procedures contained in this section. Unmodified COTS

hardware will not be subject to all tests. Generally such equipment has been designed to

rigorous industrial standards and has been in wide use, permitting an evaluation of its

performance history. To enable reduced testing of such equipment, vendors shall provide the

manufacturer specifications and evidence that the equipment has been tested to the equivalent

of these Guidelines.
61

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

The specific testing procedures to be used shall be identified in the National Certification

Test Plan prepared by the accredited test lab. These procedures may replicate testing

performed by the vendor and documented in the vendor’s TDP, but shall not rely on vendor

testing as a substitute for hardware testing performed by the accredited test lab.

4.2.2 Hardware Provided by Vendor

The hardware submitted for national certification testing shall be equivalent, in form and

function, to the actual production versions of the hardware units. Engineering or

developmental prototypes are not acceptable unless the vendor can show that the equipment

to be tested is equivalent to standard production units in both performance and construction.

4.3 Test Conditions

Certification tests may be performed in any facility capable of supporting the test

environment. Preparation for testing, arrangement of equipment, verification of equipment

status, and the execution of procedures shall be witnessed by at least one independent,

qualified observer who shall certify that all test and data acquisition requirements have been

satisfied.

When a test is to be performed at “standard” or “ambient” conditions, this requirement shall

refer to a nominal laboratory environment at prevailing atmospheric pressure and relative

humidity.

Otherwise, all tests shall be performed at the required temperature and electrical supply

voltage, regulated within the following tolerances:

Temperature of +/- 4 degrees F

Electrical supply voltage +/- 2 voltage alternating current

4.4 Test Log Data Requirements

The accredited test lab shall maintain a test log of the procedure employed. This log shall

identify the system and equipment by model and serial number. Test environment conditions

shall be noted.

In the event that the accredited test lab deems it necessary to deviate from requirements

pertaining to the test environment, the equipment arrangement and method of operation, the

specified test procedure, or the provision of test instrumentation and facilities, the deviation

shall be recorded in the test log. A discussion of the reasons for the deviation and the effect

of the deviation on the validity of the test procedure shall also be provided.
62

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

4.5 Test Fixtures

The use of test fixtures or ancillary devices to facilitate hardware testing is encouraged.

These fixtures and devices may include arrangements for automating the operation of voting

devices and the acquisition of test data.

The use of a fixture to ensure correctness in casting ballots by hand is recommended. Such a

fixture may consist of a template, with apertures in the desired location, so that selections

may be made rapidly. Such a template will eliminate or greatly minimize errors in activating

test ballot patterns, while reducing the amount of time required to cast a test ballot.

For systems that use a light source as a means of detecting voter selections, the generation of

a suitable optical signal by an external device is acceptable. For systems that rely on the

physical activation of a switch, a mechanical fixture with suitable motion generators is

acceptable.

To speed up the process of testing and to eliminate human error in casting test ballots the

tests may use a simulation device with appropriate software. Such simulation is

recommended if it covers all voting data detection and control paths that are used in casting

an actual ballot. In the event that only partial simulation is achieved, then an independent

method and test procedure must be used to validate the proper operation of those portions of

the system not tested by the simulator.

If the vendor provides a means of simulating the casting of ballots, the simulation device is

subject to the same performance, reliability, and quality requirements that apply to the voting

device itself so as not to contribute errors to the test processes.

4.6 Non-operating Environmental Tests

This section addresses a range of tests for voting machines and precinct counters, as such

devices are stored between elections and are transported between the storage facility and

polling place.

4.6.1 General

Environmental tests of non-operating equipment are intended to simulate exposure to

physical shock and vibration associated with handling and transportation of voting equipment

and precinct counters between a jurisdiction’s storage facility and precinct polling places.

These tests additionally simulate the temperature and humidity conditions that may be

encountered during storage in an uncontrolled warehouse environment or precinct

environment. The procedures and conditions of these tests correspond generally to those of

MIL-STD-810D, “Environmental Test Methods and Engineering Guidelines,” 19 July 1983.

In most cases, the severity of the test conditions has been reduced to reflect commercial,

rather than military, practice.
63

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

Systems exclusively designed with system-level COTS hardware whose configuration has

not been modified in any manner are not subject to this segment of hardware testing. Systems

made up of individual COTS components such as hard drives, motherboards, and monitors

that have been packaged to build a voting machine or other device will be required to

undergo the hardware testing.

Prior to each test, the equipment shall be shown to be operational by means of the procedure

contained in Subsection 4.6.1.5. The equipment may then be prepared as if for actual

transportation or storage, and subjected to appropriate test procedures outlined. After each

procedure has been completed, the equipment status will again be verified as in Subsection

4.6.1.5.

The following requirements for equipment preparation, functional tests, and inspections shall

apply to each of the non-operating test procedures.

4.6.1.1 Pretest Data

The test technician shall verify that the equipment is capable of normal operation. Equipment

identification, environmental conditions, equipment configuration, test instrumentation,

operator tasks, time-of-day or test time, and test results shall be recorded.

4.6.1.2 Preparation for Test

The equipment shall be prepared as for the expected non-operating use, as noted below.

When preparation for transport between the storage site and the polling place is required, the

equipment shall be prepared with any protective enclosures or internal restraints that the

vendor specifies for such transport. When preparation for storage is required, the equipment

shall be prepared using any protective enclosures or internal restraints that the vendor

specifies for storage.

4.6.1.3 Mechanical Inspection and Repair

After the test has been completed, the devices shall be removed from their containers, and

any internal restraints shall be removed. The exterior and interior of the devices shall be

inspected for evidence of mechanical damage, failure, or dislocation of internal components.

Devices shall be adjusted or repaired, if necessary.

4.6.1.4 Electrical Inspection and Adjustment

After completion of the mechanical inspection and repair, routine electrical maintenance and

adjustment may be performed, according to the manufacturer's standard procedure.
64

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

4.6.1.5 Operational Status Check

When all tests, inspections, repairs, and adjustments have been completed, normal operation

shall be verified by conducting an operational status check.

During this process, all equipment shall be operated in a manner and under environmental

conditions that simulate election use to verify the functional status of the system. Prior to the

conduct of each of the environmental hardware non-operating tests, a supplemental test shall

be made to determine that the operational state of the equipment is within acceptable

performance limits.

The following procedures shall be followed to verify the equipment status:

Step 1: Arrange the system for normal operation.

Step 2: Turn on power, and allow the system to reach recommended operating

temperature.

Step 3: Perform any servicing, and make any adjustments necessary, to achieve

operational status.

Step 4: Operate the equipment in all modes, demonstrating all functions and features

that would be used during election operations.

Step 5: Verify that all system functions have been correctly executed.

4.6.1.6 Failure Criteria

Upon completion of each non-operating test, the system hardware shall be subject to

functional testing to verify continued operability. If any portion of the voting machine or

precinct counter hardware fails to remain fully functional, the testing will be suspended until

the failure is identified and corrected by the vendor. The system will then be subject to a

retest.

4.6.2 Bench Handling Test

The bench handling test simulates stresses faced during maintenance and repair of voting

machines and ballot counters.

4.6.2.1 Applicability

All systems and components, regardless of type, shall meet the requirements of this test.

This test is equivalent to the procedure of MIL-STD-810D, Method 516.3, Procedure VI.
65

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

4.6.2.2 Procedure

Step 1: Place each piece of equipment on a level floor or table, as for normal

operation or servicing.

Step 2: Make provision, if necessary, to restrain lateral movement of the equipment or

its supports at one edge of the device. Vertical rotation about that edge shall 

not be restrained.

Step 3: Using that edge as a pivot, raise the opposite edge to an angle of 45 degrees, to

a height of four inches above the surface, or until the point of balance has been

reached, whichever occurs first.

Step 4: Release the elevated edge so that it may drop to the test surface without

restraint.

Step 5: Repeat steps 3 and 4 for a total of six events.

Step 6: Repeat steps 2, 3, and 4 for the other base edges, for a total of 24 drops for

each device.

4.6.3 Vibration Test

The vibration test simulates stresses faced during transport of voting machines and ballot

counters between storage locations and polling places.

4.6.3.1 Applicability

All systems and components, regardless of type, shall meet the requirements of this test. This

test is equivalent to the procedure of MIL-STD-810D, Method 514.3, Category 1- Basic

Transportation, Common Carrier.

4.6.3.2 Procedure

Step 1: Install the test item in its transit or combination case as prepared for transport.

Step 2: Attach instrumentation as required to measure the applied excitation.

Step 3: Mount the equipment on a vibration table with the axis of excitation along the

vertical axis of the equipment.

Step 4: Apply excitation as shown in MIL-STD-810D, Method 514.3-1, “Basic

transportation, common carrier, vertical axis”, with low frequency excitation

cutoff at 10 Hz, for a period of 30 minutes.

Step 5: Repeat steps 2 and 3 for the transverse and longitudinal axes of the equipment

with the excitation profiles shown in Figures 514.3-2 and 514.3-3,

respectively. (Note: The total excitation period equals 90 minutes, with 30

minutes excitation along each axis.)

Step 6: Remove the test item from its transit or combination case and verify its

continued operability.
66

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

4.6.4 Low Temperature Test

The low temperature test simulates stresses faced during storage of voting machines and

ballot counters.

4.6.4.1 Applicability

All systems and components, regardless of type, shall meet the requirements of this test. This

test is equivalent to the procedure of MIL-STD-810D, Method 502.2, Procedure I-Storage.

The minimum temperature shall be -4 degrees F.

4.6.4.2 Procedure

Step 1: Arrange the equipment as for storage. Install it in the test chamber.

Step 2: Lower the internal temperature of the chamber at any convenient rate, but not

so rapidly as to cause condensation in the chamber, and in any case no more

rapidly than 10 degrees F per minute, until an internal temperature of -4

degrees F has been reached.

Step 3: Allow the chamber temperature to stabilize. Maintain this temperature for a

period of 4 hours after stabilization.

Step 4: Allow the internal temperature of the chamber to return to standard laboratory

conditions, at a rate not exceeding 10 degrees F per minute.

Step 5: Allow the internal temperature of the equipment to stabilize at laboratory

conditions before removing it from the chamber.

Step 6: Remove the equipment from the chamber and from its containers, and inspect

the equipment for evidence of damage.

Step 7: Verify continued operability of the equipment.

4.6.5 High Temperature Test

The high temperature test simulates stresses faced during storage of voting machines and

ballot counters.

4.6.5.1 Applicability

All systems and components, regardless of type, shall meet the requirements of this test. This

test is equivalent to the procedure of MIL-STD-810D, Method 501.2, Procedure I-Storage.

The maximum temperature shall be 140 degrees F.

4.6.5.2 Procedure

Step 1: Arrange the equipment as for storage. Install it in the test chamber.
67

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

Step 2: Raise the internal temperature of the chamber at any convenient rate, but in

any case no more rapidly than 10 degrees F per minute, until an internal

temperature of 140 degrees F has been reached.

Step 3: Allow the chamber temperature to stabilize. Maintain this temperature for a

period of 4 hours after stabilization.

Step 4: Allow the internal temperature of the chamber to return to standard laboratory

conditions, at a rate not exceeding 10 degrees F per minute.

Step 5: Allow the internal temperature of the equipment to stabilize at laboratory

conditions before removing it from the chamber.

Step 6: Remove the equipment from the chamber and from its containers, and inspect

the equipment for evidence of damage.

Step 7: Verify continued operability of the equipment.

4.6.6 Humidity Test

The humidity test simulates stresses faced during storage of voting machines and ballot

counters.

4.6.6.1 Applicability

All systems and components regardless of type shall meet the requirements of this test. This

test is similar to the procedure of MIL-STD-810D, Method 507.2, Procedure I-Natural HotHumid. It is intended to evaluate the ability of the equipment to survive exposure to an

uncontrolled temperature and humidity environment during storage. This test lasts for ten

days.

4.6.6.2 Procedure

Step 1: Arrange the equipment as for storage. Install it in the test chamber.

Step 2 Adjust the chamber conditions to those given in MIL-STD-810D Table 507.2-

I, for the time 0000 of the HotHumid cycle (Cycle 1).

Step 3: Perform a 24-hour cycle with the time and temperature-humidity values

specified in Figure 507.2-1, Cycle 1.

Step 4: Repeat Step 2 until 5, 24-hour cycles have been completed.

Step 5: Continue with the test commencing with the conditions specified for time =

0000 hours.

Step 6: At any convenient time in the interval between time = 120 hours and time =

124 hours, place the equipment in an operational configuration, and perform a

complete operational status check as defined in Subsection 4.6.1.5.

Step 7: If the equipment satisfactorily completes the status check, continue with the

sixth 24-hour cycle.

Step 8: Perform 4 additional 24-hour cycles, terminating the test at time = 240 hours.

Step 9: Remove the equipment from the test chamber and inspect it for any evidence

of damage.
68

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

Step 10: Verify continued operability of the equipment.

4.7 Environmental Tests, Operating

This section addresses a range of tests for all voting system equipment, including equipment

for both precinct count and central count systems.

4.7.1 Temperature and Power Variation Tests

This test is similar to the low temperature and high temperature tests of MIL-STD-810-D,

Method 502.2 and Method 501.2, with test conditions that correspond to the requirements of

the performance standards. This procedure tests system operation under various

environmental conditions for at least 163 hours. During 48 hours of this operating time, the

device shall be in a test chamber. For the remaining hours, the equipment shall be operated at

room temperature. The system shall be powered for the entire period of this test; the power

may be disconnected only if necessary for removal of the system from the test chamber.

Operation shall consist of ballot-counting cycles, which vary with system type. An output

report need not be generated after each counting cycle. The interval between reports,

however, should be no more than 4 hours to keep to a practical minimum the time between

the occurrence of a failure or data error and its detection.

Test Ballots per Counting Cycle

Precinct count systems 100 ballots/hour

Central count systems 300 ballots/hour

The recommended pattern of votes is one chosen to facilitate visual recognition of the 

reported totals; this pattern shall exercise all possible voting locations. System features such

as data quality tests, error logging, and audit reports shall be enabled during the test.

Each operating cycle shall consist of processing the number of ballots indicated above.

Step 1: Arrange the equipment in the test chamber. Connect as required and provide

for power, control, and data service through enclosure wall.

Step 2: Set the supply voltage at 117 voltage alternating current.

Step 3: Power the equipment, and perform an operational status check as in Section

4.6.1.5.

Step 4: Set the chamber temperature to 50 degrees F, observing precautions against

thermal shock and condensation.

Step 5: Begin 24 hour cycle.

Step 6: At T=4 hrs, lower the supply voltage to 105 vac. 

Step 7: At T=8 hrs, raise the supply voltage to 129 vac.

Step 8: At T=11:30 hrs, return the supply voltage to 117 vac and return the chamber
69

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

temperature to lab ambient, observing precautions against thermal shock and

condensation.

Step 9: At T=12:00 hrs, raise the chamber temperature to 95 degrees Fahrenheit.

Step 10: Repeat Steps 5 through 8, with temperature at 95 degrees Fahrenheit, complete

at T=24 hrs.

Step 11: Set the chamber temperature at 50 degrees Fahrenheit as in Step 4.

Step 12: Repeat the 24 hour cycle as in Steps 5-10, complete at T=48 hrs.

Step 13: After completing the second 24 hour cycle, disconnect power from the system

and remove it from the chamber if needed.

Step 14: Reconnect the system as in Step 2, and continue testing for the remaining

period of operating time required until the ACCEPT/REJECT criteria of

Subsection 4.7.1.1 have been met.

4.7.1.1 Data Accuracy

As indicated in Volume I, Section 4, data accuracy is defined in terms of ballot position error

rate. This rate applies to the voting functions and supporting equipment that capture, record,

store, consolidate, and report the specific selections, and absence of selections, made by the

voter for each ballot position. Volume I, Subsection 4.1.1 identifies the specific functions to

be tested.

For each processing function, the system shall achieve a target error rate of no more than one

in 10,000,000 ballot positions, with a maximum acceptable error rate in the test process of

one in 500,000 ballot positions. This error rate includes errors from any source while testing

a specific processing function and its related equipment.

This error rate is used to determine the vote position processing volume used to test system

accuracy for each function:

If the system makes one error before counting 26,997 consecutive ballot positions

correctly, it will be rejected. The vendor is then required to improve the system

If the system reads at least 1,549,703 consecutive ballot positions correctly, it will be

accepted

If the system correctly reads more than 26,997 ballot positions but less than 1,549,703

when the first error occurs, the testing will have to be continued until another

1,576,701 consecutive ballot positions are counted without error (a total of 3,126,404

with one error)

Appendix C provides further details of the calculation for this testing volume.
70

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

4.7.2 Maintainability Test

The accredited test lab shall test for maintainability based on the provisions of Volume I,

Section 4 for maintainability, including both physical attributes and additional attributes

regarding the ease of performing maintenance activities. These tests include:

Examining the physical attributes of the system to determine whether significant

impediments exist for the performance of those maintenance activities that are to be

performed by the jurisdiction. These activities shall be identified by the vendor in the

system maintenance procedures portion of the TDP

Performing activities designated as maintenance activities for the jurisdiction in the

TDP, in accordance with the instructions provided by the vendor in the system

maintenance procedures, noting any difficulties encountered

Should significant impediments or difficulties be encountered that are not remedied by the

vendor, the accredited test lab shall include such findings in the certification test results of

the certification test report.

4.7.3 Reliability Test

The accredited test lab shall test for reliability based on the provisions of Volume I, Section 4

for the acceptable Mean Time Between Failure (MBTF). The MBTF shall be measured

during the conduct of other system performance tests specified in this section, and shall be at

least 163 hours. Appendix C provides further details of the calculation for this testing period.

4.7.4 Availability Test

The accredited test lab shall assess the adequacy of system availability based on the

provisions of Volume I, Section 4. As described in this section, availability of voting system

equipment is determined as a function of reliability, and the mean time to repair the system in

the event of failure.

Availability cannot be tested directly before the voting system is deployed in jurisdictions, 

but can be modeled mathematically to predict availability for a defined system configuration.

This model shall be prepared by the vendor, and shall be validated by the accredited testing

laboratory.

The model shall reflect the equipment used for a typical system configuration to perform the

following system functions:

For all paper-based systems:

Recording voter selections (such as by ballot marking)
71

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

Scanning the marks on paper ballots and converting them into digital data

For all DRE systems:

Recording and storing the voter’s ballot selections

For precinct-count systems (paper-based and DRE):

Consolidation of vote selection data from multiple precinct-based systems to

generate jurisdiction-wide vote counts, including storage and reporting of the 

consolidated vote data

For central-count systems (paper-based and DRE):

Consolidation of vote selection data from multiple counting devices to generate

jurisdiction-wide vote counts, including storage and reporting of the 

consolidated vote data

The model shall demonstrate the predicted availability of the equipment that supports each

function. This demonstration shall reflect the equipment reliability, mean time to repair, and

assumptions concerning equipment availability and deployment of maintenance personnel

stated by the vendor in the TDP.

4.8 Other Environmental Tests

This section addresses a range of tests for all voting system equipment, including equipment

for both precinct count and central count systems.

The test for power disturbance disruption shall be conducted in compliance with the test

specified in IEC 61000-4-11 (1994-06).

The test for electromagnetic radiation shall be conducted in compliance with the FCC

Part 15 Class B requirements by testing per ANSI C63.4.

The test for electrostatic disruption shall be conducted in compliance with the test

specified in IEC 61000-4-2 (1995-01).

The test for electromagnetic susceptibility shall be conducted in compliance with the

test specified in IEC 61000-4-3 (1996).

The test for electrical fast transient protection shall be conducted in compliance with the

test specified in IEC 61000-4-4 (1995-01).

The test for lightning surge protection shall be conducted in compliance with the test

specified in IEC 61000-4-5 (1995-02).
72

Version 1.0

Volume II: National Certification Testing Guidelines

4 Hardware Testing

The test for conducted RF immunity shall be conducted in compliance with the test

specified in IEC 61000-4-6 (1996-04).

The test for AC magnetic fields RF immunity shall be conducted in compliance with the

test specified in IEC 61000-4-8 (1993-06).
75

5 Software Testing

Table of Contents

5 Software Testing .......................................................................................... 77

5.1 Scope.................................................................................................. 77

5.2 Basis of Software Testing................................................................. 77

5.3 Initial Review of Documentation..................................................... 78

5.4 Source Code Review......................................................................... 78

5.4.1 Control Constructs......................................................................... 78

5.4.1.1 Replacement Rule ..................................................................... 79

5.4.2 Assessment of Coding Conventions.............................................. 84
76

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

5 Software Testing

5.1 Scope

This section contains a description of the testing to be performed by the accredited test lab to

confirm the proper functioning of the software components of a voting system submitted for

certification testing. It describes the scope and basis for software testing, the initial review of 

documentation to support software testing, and the review of the voting system source code.

Further testing of the voting system software is addressed in the following sections:

Section 3 for specific tests of voting system functionality

Section 6 for testing voting system security and for testing the operation of the voting

system software together with other voting system components

5.2 Basis of Software Testing

The accredited test lab shall design and perform procedures that test the voting system

software requirements identified in Volume I, Section 5. All software components designed

or modified for election use shall be tested in accordance with the applicable procedures

contained in this section.

Unmodified, general purpose COTS non-voting software (e.g., operating systems,

programming language compilers, data base management systems, and Web browsers) is not

subject to the detailed examinations specified in this section. However, the accredited test lab

shall examine such software to confirm the specific version of software being used against

the design specification to confirm that the software has not been modified. Portions of

COTS software that have been modified by the vendor in any manner are subject to review.

Unmodified COTS software is not subject to code examination. However, source code

generated by a COTS package and embedded in software modules for compilation or

interpretation shall be provided in human readable form to the accredited test lab. The

accredited test lab may inspect COTS source code units to determine testing requirements or

to verify the code is unmodified.

The accredited test lab may inspect the COTS generated software source code in preparation

of test plans and to provide some minimal scanning or sampling to check for embedded code

or unauthorized changes. Otherwise, the COTS source code is not subject to the full code

review and testing. For purposes of code analysis, the COTS units shall be treated as

unexpanded macros.
77

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

Compatibility of the voting system software components or subsystems with one another,

and with other components of the voting system environment, shall be determined through

functional tests integrating the voting system software with the remainder of the system.

The specific procedures to be used shall be identified in the National Certification Test Plan

prepared by the accredited test lab. These procedures may replicate testing performed by the

vendor and documented in the vendor’s TDP, but shall not rely on vendor testing as a

substitute for software testing performed by the accredited test lab.

Recognizing the variations in system design and the technologies employed by different

vendors, the accredited test lab shall design test procedures that account for these variations.

5.3 Initial Review of Documentation

Prior to initiating the software review, the accredited test lab shall verify that the

documentation submitted by the vendor in the TDP is sufficient to enable:

Review of the source code

Design and conduct tests at every level of the software structure to verify that the

software meets the vendor's design specifications and the requirements of the

performance guidelines

5.4 Source Code Review

The accredited test lab shall compare the source code to the vendor's software design

documentation to ascertain how completely the software conforms to the vendor's

specifications. Source code inspection shall also assess the extent to which the code adheres

to the requirements in Volume I, Section 5

5.4.1 Control Constructs

Voting system software shall use the control constructs identified in this section as follows:

If the programming language used does not provide these control constructs, the vendor

shall provide them (that is, comparable control structure logic). The constructs shall

be used consistently throughout the code. No other constructs shall be used to control 

program logic and execution

While some programming languages do not create programs as linear processes,

stepping from an initial condition, through changes, to a conclusion, the program 

components nonetheless contain procedures (such as “methods” in object-oriented

languages). Even in these programming languages, the procedures must execute
78

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

through these control constructs (or their equivalents, as defined and provided by the

vendor)

Operator intervention or logic that evaluates received or stored data shall not re-direct 

program control within a program routine. Program control may be re-directed within

a routine by calling subroutines, procedures, and functions, and by interrupt service

routines and exception handlers (due to abnormal error conditions). Do-While (False)

constructs and intentional exceptions (used as GoTos) are prohibited

Conventional constructs that are inherent to the development language are permitted but

must be documented in the code, adjacent to their use.

Illustrations of the following control construct techniques are provided in Figures 1 

through 4.

Fig. 1 Sequence 

Fig. 2 If -Then -Else 

Fig. 3 Do -While 

Fig. 4 Do -Until 

Fig. 5 Case

Fig. 6 General loop, including the special case FOR loop

5.4.1.1 Replacement Rule

In the constructs shown, any ‘process’ may be replaced by a simple statement, a subroutine

or function call, or any of the control constructs. In Fig 4-1 for example, “Process A” may be

a simple statement and “Process B” another Sequence construct.

Using the replacement rule to replace one or both of the processes in the Sequence construct

with other Sequence constructs, a large block of sequential code may be formed. The entire

chain is recognized as a Sequence construct and is sometimes called a BLOCK construct. In

many languages, a Sequence may need to be marked with special symbols or punctuation to

delimit where it starts and where it ends. For example, a “BEGIN” and “END” may be used.

This allows the scope of a Sequence used as “Process C” in the IF-THEN-ELSE (Fig 4-2) to

be recognized as completing the IF-THEN-ELSE rather than part of a higher level Sequence

that included the IF-THEN-ELSE as a component.
79

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

Figures 1-6

Figure 1 SEQUENCE

Control flows from “Process A” to the next in sequence, “Process B”

Figure 2 IF-THEN-ELSE

*In Figure 2, flow of control will skip a process pending the condition of “A."
80

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

Figure 3 DO-WHILE

In Figure 4-3, condition “A” is evaluated. If found to be true, then control is passed to Process “B” and

condition “A” is reevaluated. If condition “A” is found to be false, then control is passed out of the loop. Note

that, if B is a BLOCK, the “DO” may be recognized as the opening symbol. A terminating symbol is needed

from the language used.

Figure 4 DO-UNTIL

Figure 4-4 is similar to a DO-WHILE, except that the test of condition A is performed after “Process B” has

executed and the DO is performed upon a false “A” condition.. If condition “A” is true, control is passed out of

the loop.
81

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

Figure 5 CASE

Control is passed to a Process based on the value of i.
82

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

Figure 6 General LOOP

Optional process A is executed. Condition B is then evaluated. If found to be false, optional process C is

executed and control is passed to process A. Condition B is then evaluated again. If condition B is true, then

control is passed out of the loop.

A special case of the GENERAL LOOP is the FOR loop. The FOR loop is not strictly 

essential, as it can be programmed as a DO-WHILE loop. The FOR loop executes on a

counter. The control FOR statement defines a counter variable or variables, a test for ending

the loop, and a standard method of changing the variable(s) on each pass such as

incrementing or decrementing. For example,

“FOR c = 0; c < 10; c + 1
83

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

DO Process A;”

The counter is initialized to zero, if the counter test is false, the DO process is executed and 

the counter is incremented (or decremented). Once the counter test is true, control exits from

the loop without incrementing the counter. The implementation of the FOR loop in many

languages, however, can be error prone. The use of the FOR loop shall include strictly

enforced coding conventions to avoid common errors such as a loop that never ends.

The GENERAL LOOP should not be used where one of the other loop structures will serve. 

It is error prone and may not be supported in many languages without using GOTOs type

redirections. However, if defined in the language, it may be useful in defining some loops

where the exit needs to occur in the middle. Also, in other languages the GENERAL LOOP

logic can be used to simulate the other control constructs. Like the special case, the use of

the GENERAL LOOP shall require the strict enforcement of coding conventions to avoid

problems.

5.4.2 Assessment of Coding Conventions

The accredited test lab shall test for compliance with the coding conventions specified by the

vendor. If the vendor does not identify an appropriate set of coding conventions in

accordance with the provisions of Volume I, Subsection 5.2.6, the accredited test lab shall

review the code to ensure that it:

Uses uniform calling sequences. All parameters shall either be validated for type and

range on entry into each unit or the unit comments shall explicitly identify the type

and range for the reference of the programmer and tester. Validation may be

performed implicitly by the compiler or explicitly by the programmer

Has the return explicitly defined for callable units such as functions or procedures (do

not drop through by default) for C-based languages and others to which this applies,

and in the case of functions, has the return value explicitly assigned. Where the return

is only expected to return a successful value, the C convention of returning zero shall

be used or the use of another code justified in the comments. If an uncorrected error

occurs so the unit must return without correctly completing its objective, a non-zero

return value shall be given even if there is no expectation of testing the return. An

exception may be made where the return value of the function has a data range

including zero

Does not use macros that contain returns or pass control beyond the next statement

For those languages with unbound arrays, provides controls to prevent writing beyond

the array, string, or buffer boundaries

For those languages with pointers or which provide for specifying absolute memory

locations, provides controls that prevent the pointer or address from being used to
84

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

overwrite executable instructions or to access inappropriate areas where vote counts

or audit records are stored

For those languages supporting case statements, has a default choice explicitly defined

to catch values not included in the case list

Provides controls to prevent any vote counter from overflowing. Assuming the counter

size is large enough such that the value will never be reached is not adequate

Is indented consistently and clearly to indicate logical levels

Excluding code generated by commercial code generators, is written in small and easily

identifiable modules, with no more than 50% of all modules exceeding 60 lines in

length, no more than 5% of all modules exceeding 120 lines in length, and no

modules exceeding 240 lines in length. “Lines” in this context, are defined as

executable statements or flow control statements with suitable formatting and

comments. The reviewer should consider the use of formatting, such as blocking into

readable units, which supports the intent of this requirement where the module itself

exceeds the limits. The vendor shall justify any module lengths exceeding this

standard

Where code generators are used, the source file segments provided by the code

generators should be marked as such with comments defining the logic invoked and,

if possible, a copy of the source code provided to the accredited test lab with the

generated source code replaced with an unexpanded macro call or its equivalent

Has no line of code exceeding 80 columns in width (including comments and tab

expansions) without justification

Contains no more than one executable statement and no more than one flow control

statement for each line of source code

In languages where embedded executable statements are permitted in conditional

expressions, the single embedded statement may be considered a part of the

conditional expression. Any additional executable statements should be split out to

other lines

Avoids mixed-mode operations. If mixed mode usage is necessary, then all uses shall

be identified and clearly explained by comments

Upon exit() at any point, presents a message to the user indicating the reason for the

exit()

Uses separate and consistent formats to distinguish between normal status and error or

exception messages. All messages shall be self-explanatory and shall not require the

operator to perform any look-up to interpret them, except for error messages that

require resolution by a trained technician
85

Version 1.0

Volume II: National Certification Testing Guidelines

5 Software Testing

References variables by fewer than five levels of indirection (i.e., a.b.c.d or a[b].c->d)

Has functions with fewer than six levels of indented scope, counted as follows:

int function()

{

if (a = true)

1 {

if ( b = true )

2 {

3 {

4 {

if ( c = true )

if ( d = true )

while(e > 0 )

5 {

code

}

}

}

}

}

}

Initializes every variable upon declaration where permitted

t. Has all constants other than 0 and 1 defined or enumerated, or shall have a comment

which clearly explains what each constant means in the context of its use. Where “0”

and “1” have multiple meanings in the code unit, even they should be identified.

Example: “0” may be used as FALSE, initializing a counter to zero, or as a special

flag in a non-binary category

Only contains the minimum implementation of the “a = b ? c : d” syntax. Expansions

such as “j=a?(b?c:d):e;” are prohibited

Has all assert() statements coded such that they are absent from a production

compilation. Such coding may be implemented by ifdef()s that remove them from or

include them in the compilation. If implemented, the initial program identification in

setup should identify that assert() is enabled and active as a test version
87

6 System Integration Testing

Table of Contents

6 System Integration Testing .......................................................................... 89

6.1 Scope.................................................................................................. 89

6.2 Basis of Integration Testing............................................................. 89

6.2.1 Testing Breadth ............................................................................. 89

6.2.2 System Baseline for Testing.......................................................... 90

6.2.3 Testing Volume ............................................................................. 90

6.3 Testing Interfaces of System Components..................................... 90

6.4 Security Testing ................................................................................ 91

6.4.1 Access Control............................................................................... 92

6.4.2 Data Interception and Disruption .................................................. 92

6.5 Usability and Accessibility Testing ................................................. 93

6.6 Physical Configuration Audit.......................................................... 93

6.7 Functional Configuration Audit...................................................... 94
88

Version 1.0

Volume II: National Certification Testing Guidelines

6 System Integration Testing

6 System Integration Testing

6.1 Scope

This section contains a description of the testing to be performed by the accredited test lab to

confirm the proper functioning of the fully integrated components of a voting system

submitted for national certification testing. It describes the scope and basis for integration

testing, testing of internal and external system interfaces, testing of security capabilities, and

the configuration audits, including the testing of system documentation.

System level certification tests address the integrated operation of both hardware and

software, along with any telecommunications capabilities. The system level certification tests

shall include the tests (functionality, volume, stress, usability, security, performance, and

recovery) indicated in the National Certification Test Plan, described in Appendix A. These

tests assess the system's response to a range of both normal and abnormal conditions initiated

in an attempt to compromise the system. These tests may be part of the audit of the system's

functional attributes, or may be conducted separately.

The system integration tests include two audits: a Physical Configuration Audit that focuses

on physical attributes of the system, and a Functional Configuration Audit that focuses on the

system’s functional attributes, including attributes that go beyond the specific requirements

of the Standards.

6.2 Basis of Integration Testing

This subsection addresses the basis for integration testing, the system baseline for testing,

and data volumes for testing.

6.2.1 Testing Breadth

The accredited test lab shall design and perform procedures that test the voting system

capabilities for the system as a whole. These procedures follow the testing of the systems

hardware and software, and address voting system requirements defined in Volume I,

Sections 2, 4, 5 and 6.

These procedures shall also address the requirements for testing system functionality

provided in Section 3. Where practical, the accredited test lab will perform coverage

reporting of the software branches executed in the functional testing. The selection of the

baseline test cases will follow an operational profile of the common procedures, sequencing, 

and options among the shared state requirements and those that are specifically recognized

and supported by the vendor. The accredited test lab will use the coverage report to identify

any portions of the source code that were not covered and determine:
89

Version 1.0

Volume II: National Certification Testing Guidelines

6 System Integration Testing

The additional functional tests that are needed

Where more detailed source code review is needed

Both of the above

The specific procedures to be used shall be identified in the National Certification Test Plan.

These procedures may replicate testing performed by the vendor and documented in the

vendor’s TDP, but shall not rely on vendor testing as a substitute for testing performed by the

accredited test lab.

Recognizing variations in system design and the technologies employed by different vendors,

the accredited test lab shall design test procedures that account for these variations.

6.2.2 System Baseline for Testing

The system level certification tests are conducted using the version of the system intended to

be sold by the vendor and delivered to jurisdictions. To ensure that the system version tested

is the correct version, the accredited test lab shall witness the build of the executable version

of the system immediately prior to or as part of, the physical configuration audit.

Additionally, should components of the system be modified or replaced during the testing

process, the accredited test lab shall require the vendor to conduct a new “build” of the

system to ensure that the certified executable release of the system is built from tested

components.

6.2.3 Testing Volume

For all systems, the total number of ballots to be processed by each precinct counting device

during these tests shall reflect the maximum number of active voting positions and the

maximum number of ballot styles that the TDP claims the system can support.

6.3 Testing Interfaces of System Components

The accredited test lab shall design and perform test procedures that test the interfaces of all

system modules and subsystems with each other against the vendor’s specifications. These

tests shall be documented in the National Certification Test Plan, and shall include the full

range of system functionality provided by the vendor’s specifications, including functionality

that exceeds the specific requirements of these Guidelines.

Some voting systems may use components or subsystems from previously tested and

qualified systems, such as ballot preparation. For these scenarios, the accredited test lab shall,

at a minimum:
90

Version 1.0

Volume II: National Certification Testing Guidelines

6 System Integration Testing

Confirm that the version of previously approved components and subsystems is

unchanged

Test all interfaces between previously approved modules/subsystems and all other

system modules and subsystems. Where a component is expected to interface with

several different products, especially from different manufacturers, the vendor shall

provide a public data specification of files or data objects used to exchange

information

Some systems use telecommunications capabilities. For those systems that do use such

capabilities, components that are located at the polling place or separate vote counting

location shall be tested for effective interface, accurate vote transmission, failure detection,

and failure recovery. For voting systems that use telecommunications lines or networks that

are not under the control of the election official (e.g., public telephone networks), the

accredited test lab shall test the interface of vendor-supplied components with these external

components for effective interface, vote transmission, failure detection, and failure recovery.

6.4 Security Testing

The accredited test lab shall design and perform test procedures that test the security

capabilities of the voting system against the requirements defined in Volume I, Section 7.

These procedures shall focus on the ability of the system to detect, prevent, log, and recover

from the broad range of security risks identified. These procedures shall also examine system

capabilities and safeguards claimed by the vendor in the TDP to go beyond these risks. The

range of risks tested is determined by the design of the system and potential exposure to risk.

Regardless of system design and risk profile, all systems shall be tested for effective access

control and physical data security.

For systems that use public telecommunications networks, including the Internet, to transmit

election management data or official election results (such as ballots or tabulated results), the

accredited test lab shall conduct tests to ensure that the system provides the necessary

identity-proofing, confidentiality, and integrity of transmitted data. These tests shall be

designed to confirm that the system is capable of detecting, logging, preventing, and 

recovering from types of attacks known at the time the system is submitted for certification.

The accredited test lab may meet these testing requirements by confirming proper

implementation of proven commercial security software. In this case, the vendor must

provide the published standards and methods used by the U.S. Government to test and accept

this software, or it may provide references to free, publicly available publications of these

standards and methods, such as government web sites.

At its discretion, the accredited test lab may conduct or simulate attacks on the system to

confirm the effectiveness of the system's security capabilities, employing test procedures

approved by the EAC.
91

Version 1.0

Volume II: National Certification Testing Guidelines

6 System Integration Testing

6.4.1 Access Control

The accredited testing laboratory shall conduct tests of system capabilities and review the

access control policies and procedures submitted by the vendor to identify and verify the

access control features implemented as a function of the system. For those access control

features built in as components of the voting system, the accredited test lab shall design tests

to confirm that these security elements work as specified.

Specific activities to be conducted by the accredited test lab shall include:

A review of the vendor’s access control policies, procedures and system capabilities to 

confirm that all requirements of Volume I, Subsection 7.2 have been addressed

completely

Specific tests designed by the accredited test lab to verify the correct operation of all 

documented access control procedures and capabilities, including tests designed to

circumvent controls provided by the vendor. These tests shall include:

Performing the activities that the jurisdiction will perform in specific accordance

with the vendor’s access control policy and procedures to create a secure

system, including procedures for software and firmware installation (as

described in Volume I, Subsection 7.4)

Performing tests intended to bypass or otherwise defeat the resulting security

environment. These tests shall include simulation of attempts to physically

destroy components of the voting system in order to validate the correct

operation of system redundancy and backup capabilities

This review applies to the full scope of system functionality. It includes functionality for

defining the ballot and other pre-voting functions, as well as functions for casting and storing

votes, vote canvassing, vote reporting, and maintenance of the system’s audit trail.

6.4.2 Data Interception and Disruption

For systems that use telecommunications to transmit official voting data, the accredited test

lab shall review, and conduct tests of, the data interception and prevention safeguards 

specified by the vendor in its TDP. The accredited test lab shall evaluate safeguards provided

by the vendor to ensure their proper operation, including the proper response to the detection

of efforts to monitor data or otherwise compromise the system.

For systems that use public communications networks the accredited test lab shall also

review the vendor’s documented procedures for maintaining protection against newly

discovered external threats to the telecommunications network. This review shall assess the

adequacy of such procedures in terms of:

Identification of new threats and their impact
92

Version 1.0

Volume II: National Certification Testing Guidelines

6 System Integration Testing

Development or acquisition of effective countermeasures

System testing to ensure the effectiveness of the countermeasures

Notification of client jurisdictions that use the system of the threat and the actions that

should be taken

Distribution of new system releases or updates to current system users 

Confirmation of proper installation of new system releases

6.5 Usability and Accessibility Testing

The vendor shall design and perform procedures that test the usability and accessibility of the

voting system as defined in Volume I, Section 3. Test procedures shall confirm that:

All voting machines meet the usability requirements specified in Volume I, Subsection

3.1

Voting machines intended for use by voters with disabilities provide the capabilities

required by Volume I, Subsection 3.2

Voting machines intended for use by voters with disabilities operate consistently with

vendor specifications and documentation

6.6 Physical Configuration Audit

The Physical Configuration Audit compares the voting system components submitted for

qualification to the vendor's technical documentation, and shall include the following

activities:

The audit shall establish a configuration baseline of the software and hardware to be

tested. It shall also confirm whether the vendor's documentation is sufficient for the

user to install, validate, operate, and maintain the voting system. MIL-STD-1521 can

be used as a guide when conducting this audit

The test agency shall examine the vendor's source code against the submitted

documentation during the Physical Configuration Audit to verify that the software

conforms to the vendor's specifications. This review shall include an inspection of all

records of the vendor's release control system. If changes have been made to the

baseline version, the accredited test lab shall verify that the vendor's engineering and

test data are for the software version submitted for certification

If the software is to be run on any equipment other than a COTS mainframe data

processing system, minicomputer, or microcomputer, the Physical Configuration

Audit shall also include a review of all drawings, specifications, technical data, and
93

Version 1.0

Volume II: National Certification Testing Guidelines

6 System Integration Testing

test data associated with the system hardware. This examination shall establish the

system hardware baseline associated with the software baseline

To assess the adequacy of user acceptance test procedures and data, vendor documents

containing this information shall be reviewed against the system's functional

specifications. Any discrepancy or inadequacy in the vendor's plan or data shall be

resolved prior to beginning the system integration functional and performance tests

All subsequent changes to the baseline software configuration made during the course

of testing shall be subject to re-examination. All changes to the system hardware that

may produce a change in software operation shall also be subject to re-examination

The vendor shall provide a list of all documentation and data to be audited, cross-referenced

to the contents of the TDP. Vendor technical personnel shall be available to assist in the

performance of the Physical Configuration Audit.

6.7 Functional Configuration Audit

The Functional Configuration Audit encompasses an examination of vendor tests, and the

conduct of additional tests, to verify that the system hardware and software perform all the

functions described in the vendor's documentation submitted for the TDP. It includes a test of

system operations in the sequence in which they would normally be performed, and shall

include the following activities. MIL-STD-1521 may be used as a guide when conducting

this audit:

The accredited test lab shall review the vendor's test procedures and test results to

determine if the vendor's specified functional requirements have been adequately

tested. This examination shall include an assessment of the adequacy of the vendor's

test cases and input data to exercise all system functions, and to detect program logic 

and data processing errors, if such be present

The accredited test lab shall perform or supervise the performance of additional tests to

verify nominal system performance in all operating modes, and to verify on a

sampling basis the vendor's test data reports. If vendor developmental test data is

incomplete, the accredited test lab shall design and conduct all appropriate module

and integrated functional tests. The functional configuration audit may be performed

in the facility either of the accredited test lab or of the vendor, and shall use and

verify the accuracy and completeness of the System Operations, Maintenance, and

Diagnostic Testing Manuals

The vendor shall provide a list of all documentation and data to be audited, cross-referenced

to the contents of the TDP. Vendor technical personnel shall be available to assist in the

performance of the Functional Configuration Audit.
95

7 Quality Assurance Testing

Table of Contents

7 Quality Assurance Testing ........................................................................... 97

7.1 Scope.................................................................................................. 97

7.2 Basis of Examinations...................................................................... 97

7.3 General Examinations Sequence..................................................... 98

7.3.1 Vendor Practices in Parallel with Other Certification Testing...... 98

7.3.2 Functional Configuration Audit and System Integration Testing . 98

7.4 Examination of Configuration Management Practices................. 98

7.4.1 Configuration Management Policy................................................ 98

7.4.2 Configuration Identification .......................................................... 99

7.4.3 Baseline, Promotion, and Demotion Procedures........................... 99

7.4.4 Configuration Control Procedures................................................. 99

7.4.5 Release Process ............................................................................. 99

7.4.6 Configuration Audits..................................................................... 100

7.4.7 Configuration Management Resources ......................................... 100

7.5 Examination of Quality Assurance Practices................................. 100

7.5.1 Quality Assurance Policy .............................................................. 101

7.5.2 Parts and Materials Tests............................................................... 101

7.5.3 Quality Conformance Inspections ................................................. 102

7.5.4 Documentation............................................................................... 102
96

Version 1.0

Volume II: National Certification Testing Guidelines

7 Quality Assurance Testing

7 Quality Assurance Testing

7.1 Scope

This section contains a description of the examination performed by the accredited test lab to

verify conformance with the requirements for configuration management and quality

assurance of voting systems. It describes the scope and basis for the examinations, the

general sequence of the examinations within the overall test process, and provides guidance

on the substantive focus of the examinations.

7.2 Basis of Examinations

The accredited test lab shall design and perform procedures that examine documented vendor

practices for quality assurance and configuration management as addressed by Volume I,

Sections 8 and 9 and Section 2.

Examination procedures shall be designed and performed to ensure:

Conformance with the requirements to provide information on vendor practices required

by these Guidelines

Conformance of system documentation and other information provided by the vendor

with the documented practices for quality assurance and configuration management

The Guidelines do not require on-site examination of the vendor’s quality assurance and

configuration management practices during the system development process. However, the

accredited test lab can conduct several activities while at the vendor site to witness the

system build that enable assessment of the vendor’s quality assurance and configuration

management practices. These include surveys, interviews with individuals at all levels of the

development team, and examination of selected internal work products such as system

change requests and problem tracking logs.

It is recognized that examinations of vendor practices, and determinations of conformance,

entail a significant degree of professional judgment. These guidelines for vendor practices

identify specific areas of focus but heavily rely on the expertise and professional judgment,

of the accredited test lab.

The specific procedures used by the accredited test lab shall be identified in the Qualification

Test Plan. Recognizing variations in vendors’ quality assurance and configuration

management practices and procedures, the accredited test lab shall design examination

procedures that account for these variations.
97

Version 1.0

Volume II: National Certification Testing Guidelines

7 Quality Assurance Testing

7.3 General Examinations Sequence

There is no required sequence for performing the examinations of quality assurance and

configuration management practices. No other testing is dependent on the performance and

results of these examinations. However, examinations pertaining to configuration

management, in particular those pertaining to configuration identification, will generally be

useful in understanding the conventions used to define and document the components of the

system and will assist with other elements of the certification test process.

7.3.1 Vendor Practices in Parallel with Other Certification

Testing

While not required, the accredited test lab is encouraged to initiate the examinations of

quality assurance and configuration management practices early in the overall testing

sequence, and to conduct them in parallel with other testing of the voting system. Conducting

these examinations in parallel is recommended to minimize the overall duration of the testing

process.

7.3.2 Functional Configuration Audit and System Integration

Testing

As described in Volume I, Section 9, the functional configuration audit verifies that the

voting system performs all the functions described in the system documentation. To help

ensure an efficient test process, this audit shall be conducted by the accredited test lab as an

element of the system integration testing that confirms the proper functioning of the system

as a whole.

7.4 Examination of Configuration Management Practices

The examination of configuration management practices shall address the full scope of

requirements described in Volume I, Section 9, and the documentation requirements

described in Section 2. In addition to confirming that all required information has been

submitted, the accredited test lab shall determine the vendor’s conformance with the

documented configuration management practices.

7.4.1 Configuration Management Policy

The accredited test lab shall examine the vendor’s documented configuration management

policy to confirm that it:

Addresses the full scope of the system, including components provided by external

suppliers
98

Version 1.0

Volume II: National Certification Testing Guidelines

7 Quality Assurance Testing

Addresses the full breadth of system documentation

7.4.2 Configuration Identification

The accredited test lab shall examine the vendor’s documented configuration identification

practices policy to confirm that it:

Describes clearly the basis for classifying configuration items into categories and

subcategories, for numbering of configuration items; and for naming of configuration

items

Describes clearly the conventions used to identify the version of the system as a whole

and the versions of any lower level elements (e.g., subsystems, individual elements) if

such lower level version designations are used

7.4.3 Baseline, Promotion, and Demotion Procedures

The accredited test lab shall examine the vendor’s documented baseline, promotion, and

demotion procedures to confirm that they:

Provide a clear, controlled process that promotes components to baseline status when

specific criteria defined by the vendor are met; and

Provide a clear, controlled process for demoting a component from baseline status when

specific criteria defined by the vendor are met.

7.4.4 Configuration Control Procedures

The accredited test lab shall examine the vendor’s configuration control procedures to

confirm that they:

Are capable of providing effective control of internally developed system components

Are capable of providing effective control of components developed or supplied by

third parties

7.4.5 Release Process

The accredited test lab shall examine the vendor’s release process to confirm that it:

Provides clear accountability for moving forward with the release of the initial system

version and subsequent releases

Provides the means for clear identification of the system version being replaced
Version 1.0

Volume II: National Certification Testing Guidelines

7 Quality Assurance Testing

100

Confirms that all required internal vendor tests and audits prior to release have been

completed successfully

Confirms that each system version released to customers has been certified

Confirms that each system release has been received by the customer

Confirms that each system release has been installed successfully by the customer

7.4.6 Configuration Audits

The accredited test lab shall examine the vendor’s configuration audit procedures to confirm

that they:

Are sufficiently broad in scope to address the entire system, including system

documentation

Are conducted with appropriate timing to enable effective control of system versions

Are sufficiently rigorous to confirm that all system documentation prepared and

maintained by the vendor matches the actual system functionality, design, operation,

and maintenance requirements

7.4.7 Configuration Management Resources

The accredited test lab shall examine the configuration management resource information

submitted by the vendor to determine whether sufficient information has been provided to

enable another organization to clearly identify the resources used and acquire them for use.

This examination is intended to ensure that in the event the vendor concludes business

operations, sufficient information has been provided to enable an in-depth audit of the system

should such an audit be required by election officials and/or a law enforcement organization.

7.5 Examination of Quality Assurance Practices

The examination of quality assurance practices shall address the full scope of requirements

described in Volume I, Section 8, and the documentation requirements described in Volume

II, Section 2. The accredited test lab shall confirm that all required information has been

submitted, and assess whether the vendor’s quality assurance program provides for:

Clearly measurable quality standards

An effective testing program throughout the system development life cycle
Version 1.0

Volume II: National Certification Testing Guidelines

7 Quality Assurance Testing

101

Application of the quality assurance program to external providers of system 

components and supplies

Comprehensive monitoring of system performance in the field and diagnosis of system

failures

Effective record keeping of system failures to support analysis of failure patterns and

potential causes

Effective processes for notifying customers of system failures and corrective measures

that need to be taken, and for confirming that such measures are taken

In addition to the general examinations described above, the accredited test lab shall focus on

the specific elements of the vendor’s quality assurance program indicated below.

7.5.1 Quality Assurance Policy

The accredited test lab shall examine the vendor’s quality assurance policy to confirm that it:

Addresses the full scope of the voting system

Clearly designates a senior level individual accountable for implementation and

oversight of quality assurance activities

Clearly designates the individuals, by position within the vendor’s organization, who

are to conduct each quality assurance activity

Provides procedures that determine compliance with, and correct deviations from, the

quality assurance program at a minimum annually

7.5.2 Parts and Materials Tests

The accredited test lab shall examine the vendor’s parts and materials special tests and

examinations to confirm that they:

Identify appropriate criteria that are used to determine the specific system components

for which special tests are required to confirm their suitability for use in a voting

system

Are designed in a manner appropriate to determine suitability

Have been conducted and documented for all applicable parts and materials
Version 1.0

Volume II: National Certification Testing Guidelines

7 Quality Assurance Testing

102

7.5.3 Quality Conformance Inspections

The accredited test lab shall examine the vendor’s quality conformance plans, procedures

and, inspection results to confirm that:

All components have been tested according to the test requirements defined by the

vendor

All components have passed the requisite tests

For each test, the test documentation identifies:

Test location

Test date

Individual who conducted the test

Test outcome

7.5.4 Documentation

The accredited test lab shall examine the vendor’s voting system documentation to confirm

that it meets the content requirements of Volume I, Subsection 8.7, and Section 2, and is

written in a manner suitable for use by purchasing jurisdictions.
A-1

Appendix A: National Certification Test Plan

Table of Contents

A National Certification Test Plan................................................................... A-3

A.1 Scope.................................................................................................. A-3

A.1.1 References ..................................................................................... A-3

A.1.2 Terms and Abbreviations .............................................................. A-4

A.2 Pre-certification Tests...................................................................... A-4

A.3 Materials Required for Testing....................................................... A-4

A.3.1 Software......................................................................................... A-4

A.3.2 Equipment...................................................................................... A-4

A.3.3 Test Materials................................................................................ A-5

A.3.4 Deliverable Materials .................................................................... A-5

A.3.5 Proprietary Data............................................................................. A-5

A.4 Test Specifications............................................................................ A-5

A.4.1 Hardware Configuration and Design............................................. A-6

A.4.2 Software System Functions........................................................... A-6

A.4.3 Test Case Design ........................................................................... A-6

A.4.3.1 Hardware Qualitative Examination Design.............................. A-6

A.4.3.2 Hardware Environmental Test Case Design............................. A-7

A.4.3.3 Software Module Test Case Design and Data .......................... A-7

A.4.3.4 Software Functional Test Case Design..................................... A-8

A.4.3.5 System-level Test Case Design ................................................ A-10

A.5 Test Data ........................................................................................... A-10

A.5.1 Data Recording.............................................................................. A-10 

A.5.2 Test Data Criteria .......................................................................... A-11 

A.5.3 Test Data Reduction ...................................................................... A-11

A.6 Test Procedure and Conditions....................................................... A-11

A.6.1 Facility Requirements.................................................................... A-12

A.6.2 Test Set-up..................................................................................... A-12

A.6.3 Test Sequence................................................................................ A-12 

A.6.4 Test Operations Procedures........................................................... A-12
A-2

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

Appendix A: National Certification Test Plan

A.1 Scope

This Appendix contains a recommended outline for the National Certification Test Plan,

which is to be prepared by the test lab. The primary purpose of the test plan is to document

the test lab's development of the complete or partial certification test. A sample outline is

provided in Figure A-1 at the end of this Appendix.

It is intended that the test lab use this Appendix as a guide in preparing a detailed test plan,

and that the scope and detail of the requirements for certification be tailored to the type of

hardware, and the design and complexity of the software being tested. Required hardware

tests are defined in Section 4, whereas software and system-level tests must be developed

based on the vendor pre-certification tests and information available on the specific

software's physical and functional configuration.

Prior to development of any test plan, the test lab must obtain the Technical Data Package

(TDP) from the vendor submitting the voting system for certification. The TDP contains

information necessary to the development of the test plan, such as the vendor's Hardware

Specifications, Software Specifications, System Operating Manual and System Maintenance

Manual.

It is specified by the Guidelines that voting systems incorporating the vendor's software and

COTS hardware need only be submitted for software and system level tests. Recertification

of systems with modified software or hardware is also anticipated. The test lab shall alter the

test plan outline as required by these situations.

The following discussion describes the individual sections of the recommended National 

Certification Test Plan. The test lab shall include the identification, and a brief description of,

the hardware and software to be tested, and any special considerations that affect the test

design and procedure.

A.1.1 References

The test lab shall list all documents that contain material used in preparing the test plan. This

list shall include specific references to applicable portions of the guidelines, and to the

vendor’s TDP.
A-3

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

A.1.2 Terms and Abbreviations

The test lab shall list and define all terms and phrases relevant to the hardware, the software,

or the test plan.

A.2 Pre-certification Tests

The test lab shall evaluate vendor tests, or other lab tests in determining the scope of testing

required for system certification. Pre-certification test activities may be particularly useful in

designing software functional test cases and tests of system security. The test lab shall

summarize pre-certification test results that support the discussion of the preceding section.

A.3 Materials Required for Testing

The following materials must be provided to the test lab to facilitate testing of the voting

system:

Software 

Equipment

Test materials

Deliverable materials

Proprietary data

A.3.1 Software

The test lab shall list all software required for the performance of hardware, software,

telecommunications, security and system integration tests. If the test environment requires

supporting software such as operating systems, compilers, assemblers, or database managers,

then this software shall also be listed.

A.3.2 Equipment

The test lab shall list all equipment required for the performance of the hardware, software,

telecommunications, security and system integration tests. This list shall include system

hardware, general purpose data processing and communications equipment, and test

instrumentation, as required.
A-4

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

A.3.3 Test Materials

The test lab shall list all test materials required in the performance of the test including, as

applicable, test ballot layout and generation materials, test ballot sheets, test ballot cards and

control cards, standard and optional output data report formats, and any other materials used

to simulate preparation for, and conduct of, elections.

A.3.4 Deliverable Materials

The test lab shall list all documents and materials to be delivered as a part of the system, such

as:

Hardware specification

Software specification

Voter, operator, hardware, and software maintenance manuals

Program listings, facsimile ballots, tapes

Sample output report formats

A.3.5 Proprietary Data

The test lab shall list and describe all documentation and data that are proprietary to the

vendor, and hence are subject to restrictions with respect to test lab use, release, or 

disclosure.

A.4 Test Specifications

The test lab shall cite the pertinent hardware qualitative examinations and quantitative tests

that follow from Volume I, Sections 2, 4, 5, 6, 7 and 8. The test lab shall also describe the

specific test requirements that follow from the design of the software and

telecommunications capabilities under test.

The certification test shall include hardware, software and telecommunications design and

the development and conduct of all tests to demonstrate satisfactory performance.

Environmental, non-operating tests shall be performed in the categories of simulated

environmental conditions specified by the vendor or user requesting the tests. Environmental

operating tests shall be performed under varying temperatures. Other functional tests shall be

conducted in an environment that simulates, as nearly as possible, the intended use

environment.
A-5

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

Test hardware and software shall be identical to that designed to be used together in the

voting system, except that software intended for use with general purpose off-the-shelf

hardware may be tested using any equivalent equipment capable of supporting its operation

and functions.

A.4.1 Hardware Configuration and Design

The test lab shall document the hardware configuration and design in detail sufficient to

identify the specific equipment being tested. This document shall provide a basis for the

specific test design and include a brief description of the intended use of the hardware.

A.4.2 Software System Functions

The test lab shall describe the software functions in sufficient detail to provide a foundation

for selecting the test case designs and conditions described in Section A.4.3. On the basis of

this test case design, the test lab shall prepare a table delineating software functions and how

each shall be tested.

A.4.3 Test Case Design

The test lab shall examine the test case design of the following aspects of the voting system:

Hardware qualitative examination design

Hardware environmental test case design

Software module test case design and data 

Software functional test case design 

System level test case design

A.4.3.1 Hardware Qualitative Examination Design

The test lab shall review the results, submitted by the vendor, of any previous examinations

of the equipment to be tested. The results of these examinations shall be compared to the

performance characteristics specified by Section 2 of the Guidelines concerning the

requirements for:

Overall system capabilities

Pre-voting functions
A-6

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

Voting functions 

Post-voting functions

In the event that a review of the results of previous examinations indicates problem areas, the

test lab shall provide a description of further examinations required prior to conducting the

environmental and system level tests. If no previous examinations have been performed, or

records of these tests are not available, the test agency shall specify the appropriate tests to be

used in the examination.

A.4.3.2 Hardware Environmental Test Case Design

The test lab shall review the documentation, submitted by the vendor, of the results and

design of any previous environmental tests of the equipment submitted for testing. The test

design and results shall be compared to the tests described in Section 1. The test lab shall cite

any additional tests required, based on this review and those tests requested by the vendor or 

the state. The test lab shall also cite any environmental tests that are not to be conducted, and

note the reasons why.

For complete certification, environmental tests shall include the following tests, depending

upon the design and intended use of the hardware:

Non-operating tests, including the: 

Bench handling test 

Vibration test

Low temperature test

High temperature test

Humidity test

Operating tests involving a series of procedures that test system reliability and accuracy

under various temperatures and voltages relevant to election use

A.4.3.3 Software Module Test Case Design and Data

The test lab shall review the vendor's program analysis, documentation, and, if available,

module test case design. The test lab shall evaluate the test cases for each module, with

respect to flow control parameters and data on both entry and exit. All discrepancies between

the Software Specifications and the test case design shall be corrected by the vendor prior to 

initiation of the certification test.
A-7

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

If the vendor's module test case design does not provide conclusive coverage of all program

paths, then the test lab shall perform an independent analysis to assess the frequency and

consequence of error of the untested paths. The test lab shall design additional module test

cases, as required, to provide coverage of all modules containing untested paths with

potential for untrapped errors.

The test lab shall also review the vendor's module test data in order to verify that the

requirements of the Software Specifications have been demonstrated by the data. In the event

that the vendor's module test data are insufficient, the test lab shall provide a description of

additional module tests, prerequisite to the initiation of functional tests.

A.4.3.4 Software Functional Test Case Design

The test lab shall review the vendor's test plans and data to verify that the individual

performance requirements described in Subsection 2.5.3, are reflected in the software.

As a part of this process, the test lab shall review the vendor's functional test case designs.

The test lab shall prepare a detailed matrix of system functions and the test cases that exercise

them. The test lab shall also prepare a test procedure describing all test ballots, operator

procedures, and the data content of output reports. Abnormal input data and operator actions

shall be defined. Test cases shall also be designed to verify that the system is able to handle

and recover from these abnormal conditions.

The vendor's test case design may be evaluated by any standard or special method

appropriate; however, emphasis shall be placed on those functions where the vendor data on

module development reflects significant debugging problems, and on functional tests that

resulted in disproportionately high error rates.

The test lab shall define ACCEPT/REJECT criteria for certification using the Software

Specifications and, if the software runs on special hardware, the associated Hardware

Specifications to determine acceptable ranges of performance.

The test lab shall describe the functional tests to be performed. Depending upon the design

and intended use of the voting system, all or part of the functions listed below shall be tested.

Ballot preparation subsystem

Test operations performed prior to, during, and after processing of ballots, including:

Logic tests to verify interpretation of ballot styles, and recognition of precincts to

be processed

Accuracy tests to verify ballot reading accuracy

Status tests to verify equipment statement and memory contents
A-8

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

Report generation to produce test output data 

Report generation to produce audit data records

Procedures applicable to equipment used in the polling place for:

Opening the polling place and enabling the acceptance of ballots and maintaining a

count of processed ballots

Monitoring equipment status

Verifying equipment response to operator input commands

Generating real-time audit messages

Closing the polling place and disabling the acceptance of ballots

Generating election data reports

Transfer of ballot counting equipment, or a detachable memory module, to a

central counting location

Electronic transmission of election data to a central counting location

Procedures applicable to equipment used in a central counting place:

Initiating the processing of a ballot deck or programmable memory device for one

or more precincts

Monitoring equipment status

Verifying equipment response to operator input commands

Verifying interaction with peripheral equipment, or other data processing systems

Generating real-time audit messages

Generating precinct-level election data reports

Generating summary election data reports

Transfer of a detachable memory module to other processing equipment

Electronic transmission of data to other processing equipment

Producing output data for interrogation by external display devices
A-9

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

A.4.3.5 System-level Test Case Design

The test lab shall provide a description of system tests of both the software and hardware. For 

software, these tests shall be designed according to the stated design objective without

consideration of its functional specification. The test lab shall independently prepare the

system test cases to assess the response of the hardware and software to a range of

conditions, such as:

Volume tests: These tests investigate the system's response to processing more than the

expected number of ballots/voters per precinct, to processing more than the expected

number of precincts, or to any other similar conditions that tend to overload the

system's capacity to process, store, and report data.

Stress tests: These tests investigate the system's response to transient overload

conditions. Polling place devices shall be subjected to ballot processing at the high

volume rates at which the equipment can be operated to evaluate software response to

hardware-generated interrupts and wait states. Central counting systems shall be

subjected to similar overloads, including, for systems that support more than one card

reader, continuous processing through all readers simultaneously.

Usability tests: These tests are designed to exercise characteristics of the software such

as response to input control or text syntax errors, error message content, audit

message content, and other features contained in the software design objectives but

not directly related to a functional specification.

Accessibility tests: The test lab shall review the vendor’s documentation of the

usability and accessibility testing performed during system development.

Security tests: These tests are designed to defeat the security provisions of the system

including modification or disruption of pre-voting, voting, and post voting 

processing; unauthorized access to, deletion, or modification of data, including audit 

trail data; and modification or elimination of security mechanisms.

Performance tests: These tests verify accuracy, processing rate, ballot format handling

capability, and other performance attributes claimed by the vendor.

Recovery tests: These tests verify the ability of the system to recover from hardware

and data errors.

A.5 Test Data

A.5.1 Data Recording

The test lab shall identify all data recording requirements (e.g.; what is to be measured, how

tests and results are to be recorded). The test lab shall also design or approve the design of
A-10

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

forms or other recording media to be employed. The test lab shall supply any special

instrumentation (e.g., pulse measuring device) needed to satisfy the data requirements.

A.5.2 Test Data Criteria

The test lab shall describe the criteria against which test results will be evaluated, such as the

following:

Tolerances: These criteria define the acceptable range for system performance. These

tolerances shall be derived from the applicable hardware performance requirements

contained in Volume I, Section 4

Samples: These criteria define the minimum number of combinations or alternatives of

input and output conditions that can be exercised to constitute an acceptable test of

the parameters involved

Events: These criteria define the maximum number of interrupts, halts or other system

breaks that may occur due to non-test conditions. This count shall not include events

from which recovery occurs automatically or where a relevant status message is

displayed

A.5.3 Test Data Reduction

The test lab shall describe the techniques to be used for processing test data. These

techniques may include manual, semi-automatic, or fully automatic reduction procedures.

However, semi-automatic and automatic procedures must be demonstrated to be capable of

handling the test data accurately and properly. They shall also produce an item-by-item

comparison of the data and the embedded acceptance criteria as output.

A.6 Test Procedure and Conditions

The test lab shall describe the test conditions and procedures for performing the tests. If tests

are not to be performed in random order, this section shall contain the rationale for the

required sequence, and the criteria that must be met, before the sequence can be continued.

This section shall also describe the procedure for setting up the equipment in which the

software will be tested, for system initialization, and for performing the tests. Each of the

following sections that contain a description of a test procedure shall also contain a statement

of the criteria by which readiness and successful completion shall be indicated and measured.
A-11

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

A.6.1 Facility Requirements

The test lab shall describe the space, equipment, instrumentation, utilities, manpower, and

other resources required to support the test program.

A.6.2 Test Set-up

The test lab shall describe the procedure for arranging and connecting the system hardware

with the supporting hardware and telecommunications equipment, if applicable. It shall also

describe the procedure required to initialize the system, and to verify that it is ready to be

tested.

A.6.3 Test Sequence

The test lab shall state any restrictions on the grouping or sequence of tests in this section.

A.6.4 Test Operations Procedures

The test lab shall provide the step-by-step procedures for each test case to be conducted. 

Each step shall be assigned a test step number and this number, along with critical test data

and test procedures information, shall be tabulated onto a test report form for test control and

the recording of test results.

In this section, the test lab shall also identify all test operations personnel, and their

respective duties. In the event that the operator procedure is not defined in the vendor's

operations or user manual, the test lab shall also provide a description of the procedures to be

followed by the test personnel.
A-12

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix A: National Certification Test Plan

Figure 1 Test Plan Outline

1 Introduction

1.1 References

1.2 Terms and Abbreviations

2 Pre-certification Tests

Pre-certification Test Activity

2.2 Pre-certification Test Results

3 Materials Required for Testing

3.1 Software

3.2 Equipment

3.3 Test Materials

3.4 Deliverable Materials 

Proprietary Data

4 Test Specification

4.1 Requirements

4.2 Hardware Configuration and Design

4.3 Software System Functions

4.4 Test Case Design

4.4.1 Hardware Qualitative Examination Design

4.4.2 Hardware Environmental Test Case Design

4.4.3 Software Module Test Case Design and Data

4.4.4 Software Functional Test Case Design and Data

4.4.5 System-level Test Case Design

5 Test Data

5.1 Data Recording

5.2 Test Data Criteria

5.3 Test Data Reduction

6 Test Procedure and Conditions

6.1 Facility Requirements

6.2 Test Set-up

6.3 Test Sequence Test 

Operations Procedures
B-1

Appendix B: National Certification Test Report

Table of Contents

B National Certification Test Report............................................................... B-3

B.1 Scope .................................................................................................. B-3

B.1.1 New Certification Test Report....................................................... B-3

B.1.2 Changes to Previously Certified Test Report ................................ B-3 

B.2 Certification Test Background........................................................ B-4

B.3 System Identification........................................................................ B-4

B.4 System Overview .............................................................................. B-4

B.5 Certification Test Results and Recommendation.......................... B-4

B.6 Appendix - Test Operations and Findings..................................... B-5

B.7 Appendix - Test Data Analysis........................................................ B-5
B-2

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix B: National Certification Test Report

Appendix B: National Certification Test Report

B.1 Scope

This Appendix contains a recommended outline for the National Certification Test Report to

be prepared by the accredited test lab. The test report shall be organized so as to facilitate the

presentation of conclusions and recommendations regarding system acceptability, a summary

of the test operations, a summary of the test results, the test data records, and the analyses

that support the conclusions and recommendations. The content of the report may vary based

on the scope of review conducted.

B.1.1 New Certification Test Report

A full report is prepared for the initial certification testing of a voting system. This document

consists of five main sections: Introduction, Certification Test Background, System

Identification, System Overview, and Certification Test Results.

Detailed information about the test operations and findings, and test data, are included as

appendices to the report.

Sections B.2 through B.7 describe the contents of the individual sections of this report.

B.1.2 Changes to Previously Certified Test Report

This report addresses a wide range of scenarios. After a preliminary review of the submitted

changes, the accredited test lab may determine that:

A review of all change documentation against the baseline materials is sufficient for

recommendation for certification

All changes must be retested against the previously certified baseline

The scope of the changes is substantial enough that a complete retest of the software is

required

The format of this report will vary, based on the type of review that is performed. If only a

review of change documentation against the baseline materials is performed the report is

quite simple. It consists of an Introduction, a Version Description, the Testing Approach,

and a Results Summary. A more extensive report is prepared, for changes that have extensive

impact on the system design and/or operations.
B-3

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix B: National Certification Test Report

B.2 Certification Test Background

This section contains the following information:

General information about the certification test process

A list and definition of all terms and nomenclature peculiar to the hardware, the

software, or the test report

B.3 System Identification

This section gives information about the tested software and supporting hardware, including:

System name and major subsystems (or equivalent)

System version

Test support hardware

Specific documentation provided in the vendor's TDP used to support testing

B.4 System Overview

This section describes the voting system in terms of its overall design structure, technologies

used, processing capacity claimed by the vendor for system components (such as ballot

counters, voting machines, vote consolidation equipment), and mode of operation. It may

also identify other products that interface with the voting system.

B.5 Certification Test Results and Recommendation

This section provides a summary of the results of the testing process, and indicates any

special considerations that affect the conclusions derived from the test results. This summary

includes:

The acceptability of the system design and construction based on the performance of the

system hardware, software and communications, and on the source code inspection

The degree to which the hardware and software meet the vendor's specifications and the

guidelines, and the acceptability of the vendor's technical and user documentation

General findings on the maintainability of the system including, where applicable,

notation of specific maintenance activities that are determined to be difficult to

perform
B-4

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix B: National Certification Test Report

Identification and description of any deficiencies that remain uncorrected after

completion of the certification test and that have caused or are judged to be capable of

causing, the loss or corruption of voting data, providing sufficient detail to support a 

recommendation to reject the system being tested. Similarly, any deficiency in

compliance with the security, accuracy, data retention, and audit requirements are

fully described

A specific recommendation to the EAC for approval or rejection

Of note, any uncorrected deficiency that does not involve the loss or corruption of voting

data shall not necessarily be cause for rejection. Deficiencies of this type may include failure

to fully achieve the levels of performance specified in Volume I or failure to fully implement

formal programs for quality assurance and configuration management described in Volume I,

Sections 8 and 9. The nature of the deficiency is described in detail sufficient to support the

recommendation either to accept or to reject the system. The recommendation is based on

consideration of the probable effect the deficiency will have on safe and efficient system

operation during all phases of election use.

B.6 Appendix – Test Operations and Findings

This appendix provides additional detail about the test results to enable the understanding of

test results and recommendation. This information is organized in a manner that reflects the

Certification Test Plan. Summaries of the results of hardware examinations, operating and

non-operating hardware tests, software module tests, software function tests, and systemlevel tests (including security and telecommunications tests, and the results of the Physical

and Functional Configuration Audits) are provided.

B.7 Appendix - Test Data Analysis

This appendix provides summary records of the test data and the details of the analysis. The

analysis includes a comparison of the vendor's hardware and software specifications to the

test data, together with any mathematical or statistical procedure used for data reduction and

processing.
C-1

Appendix C: National Certification Test Design

Criteria

Table of Contents

C National Certification Test Design Criteria ................................................. C-3

C.1 Scope .................................................................................................. C-3

C.2 Approach to Test Design.................................................................. C-3

C.3 Probability Ratio Sequential Test (PRST)..................................... C-4

C.4 Time-based Failure Testing Criteria .............................................. C-5

C.5 Accuracy Testing Criteria ............................................................... C-7
C-2

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix C: National Certification Test Design Criteria

Appendix C: National Certification Test Design

Criteria

C.1 Scope

This appendix describes the guiding principles used to design the voting system certification

testing process conducted by the accredited test lab.

Certification tests are designed to demonstrate that the system meets or exceeds the

requirements of the Guidelines. The tests are also used to demonstrate compliance with other

levels of performance claimed by the manufacturer.

Certification tests must satisfy two separate and possibly conflicting sets of considerations.

The first is the need to produce enough test data to provide confidence in the validity of the

test and its apparent outcome. The second is the need to achieve a meaningful test at a

reasonable cost, and cost varies with the difficulty of simulating expected real-world

operating conditions and with test duration. It is the test designer's job to achieve an

acceptable balance of these constraints.

The rationale for, and statistical methods of, the test designs required by the Guidelines are

discussed below. Technical descriptions of these designs can be found in any of several

books on testing and statistical analysis.

C.2 Approach to Test Design

The certification tests specified in the Guidelines are primarily concerned with assessing the

magnitude of random errors. They are also, however, capable of detecting bias errors that

would result in the rejection of the system.

Test data typically produce two results. The first is an estimate of the true value of some

system attribute such as speed, error rate, etc. The second is the degree of certainty that the 

estimate is a correct one. The estimate of an attribute's value may or may not be greatly

affected by the duration of the test. Test duration, however, is very important to the degree of

certainty; as the length of the test increases, the level of uncertainty decreases. An efficient

test design will produce enough data over a sufficient period of time to enable an estimate at

the desired level of confidence.

There are several ways to design tests. One approach involves the pre-selection of some test

parameter, such as the number of failures or other detectable factors. The essential element of

this type of design is that the number of observations is independent of their results. The test

may be designed to terminate after 1,000 hours or 10 days, or when 5 failures have been
C-3

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix C: National Certification Test Design Criteria

observed. The number of failures is important because the confidence interval (uncertainty

band) decreases rapidly as the number of failures increases. However, if the system is highly

reliable or very accurate, the length of time required to produce a predetermined number of

failures or errors using this method may be unachievably long.

Another approach is to determine that the actual value of some attribute need not be learned

by testing, provided that the value can be shown to be better than some level. The test would

not be designed to produce an estimate of the true value of the attribute but instead to show,

for example, that reliability is at least 123 hours or the error rate is no greater than one in ten

million characters.

The latter design approach, which was chosen for the Guidelines, uses what is called

Sequential Analysis. Instead of the test duration being fixed, it varies depending on the 

outcome of a series of observations. The test is terminated as soon as a statistically valid

decision can be reached that the factor being tested is at least as good as, or no worse than,

the predetermined target value. A sequential analysis test design called the "Wald Probability

Ratio Test" is used for reliability and accuracy testing.

C.3 Probability Ratio Sequential Test (PRST)

The design of a Probability Ratio Sequential Test (PRST) requires that four parameters be

specified:

H0, the null hypothesis

H1, the alternate hypothesis 

a, the producer's risk

b, the consumer's risk

The Guidelines anticipate using the PRST for testing both time-based and event-based

failures.

This test design provides decision criteria for accepting or rejecting one of two test

hypotheses: the null hypothesis, which is the Nominal Specification Value (NSV), or the

alternate hypothesis, which is the MAV. The MAV could be either the Minimum Acceptable

Value, or the Maximum Acceptable Value, depending upon what is being tested.

Performance may be specified by means of a single value or by two values. When a single

value is specified, it shall be interpreted as an upper or lower single-sided 90 percent

confidence limit. If two values, these shall be interpreted as a two-sided 90 percent

confidence interval, consisting of the NSV and MAV.

In the case of Mean Time Between Failure (MTBF), for example, the null hypothesis is that

the true MTBF is at least as great as the desired value (NSV), while the alternate hypothesis

is that the true value of the MTBF is less than some lower value (Minimum Acceptable

Value). In the case of error rate, the null hypothesis is that the true error rate is less than some

very small desired value (NSV), while the alternate hypothesis is that the true error rate is
C-4

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix C: National Certification Test Design Criteria

greater than some larger value that is the upper limit for acceptable error (Maximum

Acceptable Value).

C.4 Time-based Failure Testing Criteria

The equivalence between a number of events and a time period can be established when the

operating scenarios of a system can be determined with precision. Some of the performance

test criteria of Volume II, Section 4, use this equivalence.

System acceptance or rejection can be determined by observing the number of relevant

failures that occur during equipment operation. The probability ratio for this test is derived

from the exponential probability distribution. This distribution implies a constant hazard rate

for equipment failure that is not dependent on the time of testing or the previous failures. In

that case, two or more systems may be tested simultaneously to accumulate the required

number of test hours, and the validity of the data is not affected by the number of operating

hours on a particular unit of equipment. However, for environmental operating hardware

tests, no unit shall be subjected to less than two complete 24-hour test cycles in a test

chamber as required by Volume II, Subsection 4.7.1.

In this case, the null hypothesis is that the Mean Time Between Failure (MTBF), as defined

in Volume I, Subsection 4.3.3 is at least as great as some value, here the Nominal

Specification Value. The alternate hypothesis is that the MTBF is no better than some value,

here the Minimum Acceptable Value.

For example, a typical system operations scenario for environmental operating hardware tests

will consist of approximately 45 hours of equipment operation. Broken down, this time

allotment involves 30 hours of equipment setup and readiness testing and 15 hours of

elections operations. If the Minimum Acceptable Value is defined as 45 hours, and a test

discrimination ratio of 3 is used (in order to produce an acceptably short expected time of

decision), then the Nominal Specification Value equals 135 hours.

With a value of decision risk equal to 10 percent, there is no more than a 10 percent chance

that a system would be rejected when, in fact, with a true MTBF of at least 135 hours, the

system would be acceptable. It also means that there is no more than a 10 percent chance that

a system would be accepted with a true MTBF lower than 45 hours when it should have been 

rejected.

Therefore,

H0: MTBF = 135 hours 

H1: MTBF = 45 hours 

a = 0.10

b = 0.10.
C-5

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix C: National Certification Test Design Criteria

Under this PRST design, the test is terminated and an ACCEPT decision is reached when the

cumulative number of equipment hours in the second column of the following table has been

reached, and the number of failures is equal to or less than the number shown in the first

column. The test is terminated and a REJECT decision is reached when the number of

failures occurs in less than the number of hours specified in the third column. Here, the

minimum time to accept (on zero failures) is 169 hours. In the event that no decision has

been reached by the times shown in the last table entries, the test is terminated, and the

decision is declared as indicated. Any time that 7 or more failures occur, the test is

terminated and the equipment rejected. If, after 466 hours of operation, the cumulative

failure score is less than 7.0, then the equipment is accepted.

Number of

Failures

Accept if Time

Greater Than

Reject if Time

Less Than

0 169 Continue test

1 243 Continue test

2 317 26

3 392 100

4 466 175

5 466 249

6 466 323

7 N/A (1)

(1) Terminate and REJECT

This test is based on the table of test times of the truncated PRST design V-D in the Military

Handbook MIL-HDBK-781A that is designated for discrimination ratio 3 and a nominal

value of 0.10 for both a and b. The Handbook states that the true producer risk is 0.111 and 

the true consumer risk is 0.109. Using the theoretical formulas for either the untruncated or

truncated tests will lead to different numbers.

The test design will change if given a different set of parameters. Some jurisdictions may

find the Minimum Acceptable Value of 45 hours unacceptable for their needs. In addition, it

may be appropriate to use a different discrimination ratio, or different, Consumer’s and

Producer’s risk. Also, before using tests based on the MTBF, it should be determined

whether time-based testing is appropriate rather than event-based or another form of testing. 

If MTBF-based procedures are chosen, then the appropriateness of the assumption of a

constant hazard rate with exponential failures should in turn be assessed.
C-6

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix C: National Certification Test Design Criteria

C.5 Accuracy Testing Criteria

Some voting system performance attributes are tested by inducing an event or series of

events, and the relative or absolute time intervals between repetitions of the event has no

significance. Although equivalence between a number of events and a time period can be

established when the operating scenarios of a system can be determined with precision,

another type of test is required when such equivalence cannot be established. It uses eventbased failure frequencies to arrive at ACCEPT/REJECT criteria. This test may be performed

simultaneously with time-based tests.

For example, the failure of a device is usually dependent on the processing volume that it is

required to perform. The elapsed time over which a certain number of actuation cycles occur

is, under most circumstances, not important. Another example of such an attribute is the

frequency of errors in reading, recording, and processing vote data.

The error frequency, called “ballot position error rate,” applies to such functions as process 

of detecting the presence or absence of a voting punch or mark, or to the closure of a switch

corresponding to the selection of a candidate.

Certification and acceptance test procedures that accommodate event-based failures are,

therefore, based on a discrete, rather than a continuous probability distribution. A Probability

Ratio Sequential Test using the binomial distribution is recommended. In the case of ballot

position error rate, the calculation for a specific device (and the processing function that

relies on that device) is based on:

HO: Desired error rate = 1 in 10,000,000

H1: Maximum acceptable error rate = 1 in 500,000

a = 0.05

b = 0.05

and the minimum error-free sample size to accept for qualification tests is 1,549,703 votes.

The nature of the problem may be illustrated by the following example, using the criteria

contained in the Guidelines for system error rate. A target for the desired accuracy is

established at a very low error rate. A threshold for the worst error rate that can be accepted is

then fixed at a somewhat higher error rate. Next, the decision risk is chosen, that is, the risk

that the test results may not be a true indicator of either the system's acceptability or

unacceptability. The process is as follows:

The desired accuracy of the voting system, whatever its true error rate (which may be

far better), is established as no more than one error in every ten million characters

(including the null character)
C-7

Version 1.0

Volume II: National Certification Testing Guidelines

Appendix C: National Certification Test Design Criteria

If it can be shown that the system's true error rate does not exceed one in every five

hundred thousand votes counted, it will be considered acceptable. This is more than

accurate enough to declare the winner correctly in almost every election

A decision risk of 5 percent is chosen, to be 95 percent sure that the test data will not

indicate that the system is bad when it is good or good when it is bad

This results in the following decision criteria:

d.If the system makes one error before counting 26,997 consecutive ballot positions

correctly, it will be rejected. The vendor is then required to improve the system

e. If the system reads at least 1,549,703 consecutive ballot positions correctly, it will be

accepted

f. If the system correctly reads more than 26,997 ballot positions but less than 1,549,703

when the first error occurs, the testing will have to be continued until another

1,576,701 consecutive ballot positions are counted without error (a total of 3,126,404

with one error)
EAC Commissioners

Chairman Paul DeGregorio

Vice Chairman Ray Martinez III

Commissioner Donetta Davidson

Commissioner Gracia Hillman

EAC Staff

Tom Wilkey, Executive Director

Juliet Thompson, General Counsel

U.S. Election Assistance Commission

1225 New York Avenue, NW

Suite 1100

Washington, DC 20005

866–747–1471 (toll free)

HAVAinfo@eac.gov

www.eac.gov

U.S. Election Assistance Commission