Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 

Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 1 of 19 

Certification Test Plan 

Report Number 07-V-ESS-035-CTP-01 

Election Systems & Software 

Unity 3.2.0.0 Voting System 

Includes:

AM, EDM, ESSIM, HPM, ERM & the intElect DS200 Scanner

Test Plan Rev 03 

September 26, 2007

Prepared for: 

Vendor Name Election Systems & Software 

Vendor Address 11208 John Galt Boulevard 

Omaha, NE 68137-2364 

Prepared by: 

216 16th St. 

Suite 700 

Denver, CO 80202 

Accredited by the Election Assistance 

Commission (EAC) for Selected Voting 

System Test Methods or Services 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 2 of 19 

Disclaimer

Copyright © 2006-2007 by SysTest Labs Incorporated 

Trademarks 

• SysTest Labs is a registered trademark of SysTest Labs Incorporated. 

• Intel and Pentium are registered trademarks, and 486 is a trademark of Intel Corporation. 

• Microsoft, MS are registered trademarks and Internet Explorer, Windows, MS Windows, Windows NT, Windows 2000, 2003, 

Windows 95, Windows 98, Visual C++, Visual Basic, VBX, ODBC, and MFC are trademarks of Microsoft Corporation. 

• ES&S’s Unity is a trademark of ES&S 

• Hardware Programming ManagerTM is a registered trademark of Election Systems & Software Inc. 

• ES&S Image ManagerTM is a registered trademark of Election Systems & Software Inc. 

• Election Data ManagerTM is a registered trademark of Election Systems & Software Inc. 

• Election Reporting ManagerTM is a registered trademark of Election Systems & Software Inc. 

• Audit ManagerTM is a registered trademark of Election Systems & Software Inc. 

• DS200TM is a registered trademark of Election Systems & Software Inc. 

Other Trademarks

• All other products and company names are used for identification purposes only and may be trademarks of their 

respective owners. 

Revision History

Date Description of Revision Author Revision No. 

05/04/2007 Initial Test Plan D. Watz Rev 00 

5/14/2007 Review of Test Plan M. Santos, J. 

Garcia, D. Valdez 

Rev 01 

5/22/2007 Final Review of Test Plan with changes J. Garcia Rev 01 

6/20/2007 Update EAC comments from Conference Call 6/8/07 K. Swift Rev 02 

9/26/2007 Updates from EAC comments document 9/21/07 D. George Rev 03 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 3 of 19 

TABLE OF CONTENTS 

1 INTRODUCTION.......................................................................................................................................................... 4

1.1 REFERENCES............................................................................................................................................................... 5

1.2 TERMS AND ABBREVIATIONS...................................................................................................................................... 5

2 PRE-CERTIFICATION TESTS................................................................................................................................... 6

2.1 PRE-CERTIFICATION TEST ACTIVITY .......................................................................................................................... 6

2.2 PRE-CERTIFICATION ASSESSMENT RESULTS............................................................................................................... 7

3 MATERIALS REQUIRED FOR TESTING ............................................................................................................... 8

3.1 SOFTWARE/FIRMWARE ............................................................................................................................................... 8

3.2 EQUIPMENT (HARDWARE) .......................................................................................................................................... 8

3.3 TEST MATERIALS........................................................................................................................................................ 8

3.4 DELIVERABLE MATERIALS ......................................................................................................................................... 8

3.5 PROPRIETARY DATA ................................................................................................................................................... 9

4 TEST SPECIFICATIONS........................................................................................................................................... 10

4.1 HARDWARE CONFIGURATION AND DESIGN .............................................................................................................. 10

4.2 SOFTWARE SYSTEM FUNCTIONS ............................................................................................................................... 10

4.2.1 System Functional Testing.............................................................................................................................. 11

4.3 TEST CASE DESIGN................................................................................................................................................... 14

4.3.1 Hardware Qualitative Examination Design ................................................................................................... 14

4.3.2 Hardware Environmental Test Case Design .................................................................................................. 14

4.3.3 Software Module Test Case Design and Data................................................................................................ 14

4.3.4 Software Functional Test Case Design........................................................................................................... 14

4.3.5 System-level Test Case Design ....................................................................................................................... 14

4.3.6 Sampling Methodology................................................................................................................................... 15

4.4 EAC INTERPRETATIONS ........................................................................................................................................... 15

5 TEST DATA ................................................................................................................................................................. 16

5.1 DATA RECORDING .................................................................................................................................................... 16

5.2 TEST DATA CRITERIA ............................................................................................................................................... 16

5.3 TEST DATA REDUCTION ........................................................................................................................................... 16

6 TEST PROCEDURE AND CONDITIONS ............................................................................................................... 17

6.1 FACILITY REQUIREMENTS......................................................................................................................................... 17

6.2 TEST SETUP .............................................................................................................................................................. 17

6.3 TEST SEQUENCE ....................................................................................................................................................... 17

6.4 TEST OPERATIONS PROCEDURES .............................................................................................................................. 18

7 APPROVAL SIGNATURES....................................................................................................................................... 19

LIST OF TABLES 

Table 1 - Matrix of Terms & Abbreviations ........................................................................................................... 5

Table 2 - Matrix of Required Software/Firmware .................................................................................................. 8

Table 3 - Matrix of Required Hardware.................................................................................................................. 8

Table 4 - Matrix of Test Materials .......................................................................................................................... 8

Table 5 - Matrix of Hardware Configuration ........................................................................................................ 10

Table 6 - Matrix of System Functional Testing .................................................................................................... 11

Table 7 - Matrix of Other System Testing ............................................................................................................ 13
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 4 of 19 

1 INTRODUCTION 

This Master Certification Test Plan outlines the approach SysTest Labs will implement to perform 

Federal Election Commission (FEC) Voting System Standards Certification testing of the Unity 3.2.0.0 

Voting System by Election Systems and Software (ES&S). The purpose of this document is to provide 

a clear and precise plan for test elements required to ensure effective Certification testing as outlined in 

the VSS Vol. 1: Section 9.6.2.1, Vol. 1: 7.7, and Vol. 2: 2.2 through 2.8. The ES&S Unity 3.2.0.0 

Voting System consists of an Election Management System: Audit Manager (AM), Election Data 

Manager (EDM), ES&S Ballot Image Manager (ESSIM), Hardware Programming Manager (HPM), 

and Election Reporting Manager (ERM). The AM tracks users and provides security for the Unity 

Election System. The EDM is used to create the initial election data. The ESSIM provides the tools 

for designing and printing ballots. The main function of the HPM is to organize the election data. 

However, the HPM may also be used for election definition. (Election definition in the HPM is not 

supported in this certification.) The ERM reports the results of the election. 

The ES&S Unity 3.2.0.0 Voting System also consists of a DS200 precinct count scanner, which is an 

optical scan paper ballot tabulator that accepts ballots inserted in any direction and reads both sides of 

the ballot simultaneously. 

The objective of this test plan is to outline the certification test tasks. This test plan: 

• Identifies items that need to be tested; 

• Defines the test approach; 

• Identifies required hardware, support software, and tools to be used for testing; and 

• Identifies the types of tests to be performed. 

Certification Test Plan Attachments 

The following attachments apply to this Certification Test Plan: 

Attachment A: Technical Data Package Documents 

Attachment B: Supported Functionality Declaration 

Attachment C: List of Source Code Reviewed - PROPRIETARY

Attachment D: Hardware Test Plan 

SysTest Labs will provide certification testing for ES&S on the Unity 3.2.0.0 Voting System. The 

resulting certification will be to the FEC VSS 2002. 

This effort includes the Physical Configuration Audit (including the Technical Data Package 

documentation review and source code review) as defined by the EAC VVSG 2005 Vol. 2, section 2. 

The Functional Configuration Audit will consist of an assessment of ES&S’ testing to the ES&S 

System Requirements Specification, as outlined in the FEC VSS Vol. 1 section 2, as well as the 

performance of functional and system level integration tests. This includes developing a thorough test 

plan, managing system configurations, generating test cases as needed based on the set of test 

requirements (in addition to the test cases and procedures furnished by ES&S), test execution, and 

analysis of test results. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 5 of 19 

1.1 References 

1. Federal Election Commission Voting System Standards (FEC VSS), April 2002. Volumes I and 

II. 

2. Election Assistance Commission Voluntary Voting System Guidelines (EAC VVSG), 2005 

Version 1.0. Volumes I and II. 

3. NIST NVLAP Handbook 150: 2006. 

4. NIST NVLAP Handbook and 150-22: 2005. 

5. EAC Testing and Certification Program Manual, United States Election Assistance 

Commission, 2006 

6. SysTest Labs Quality System Manual, Revision 1.0, prepared by SysTest Labs, dated 

November 3, 2006. 

1.2 Terms and Abbreviations 

These terms and abbreviations will be used throughout this document: 

Table 1 - Matrix of Terms & Abbreviations 

Term Abbreviation Description 

Audit Manager AM Audit Manager is ES&S’ tracking program for the Unity 

software suite. AM tracks user activity in AM, EDM and 

ESSIM. 

Election Data Manager EDM Election Data Manager is a database system that stores all of a 

jurisdiction’s precinct, office, and candidate information. It is 

used in conjunction with other Unity software to format and 

print ballots, program ballot scanning equipment, and produce 

Election Day reports. 

ES&S Ballot Image Manager ESSIM ES&S Ballot Image Manager is a publishing tool used to 

design and print ballots with the election information stored in 

EDM. 

Election Reporting Manager ERM The Election Reporting Manager is ES&S’ election results 

reporting program, used to generate paper and electronic 

reports for poll workers, candidates, and the media. ERM can 

display updated election totals on a monitor as ballot data is 

tabulated and can send result reports directly to media outlets 

over the Internet. ERM is designed to support a wide range of 

ES&S ballot scanning equipment and can produce reports for 

both central count systems and precinct count systems. 

Hardware Programming Manager HPM Hardware Programming Manager enables the user to import, 

format, and convert the election definition files for ballot 

scanning equipment. 

intElect DS200 DS200 The intElect DS200 is the operational environment hardware 

for the Unity 3.2.0.0 Voting System. It is an optical scan 

paper ballot tabulator that accepts ballots inserted in any 

direction and reads both sides of the ballot simultaneously. 

Unity Release N/A The system configuration(s) of ES&S hardware and software 

voting system(s). 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 6 of 19 

2 PRE-CERTIFICATION TESTS 

2.1 Pre-Certification Test Activity 

SysTest Labs has conducted a Physical Configuration Audit of the documents submitted for review in 

the ES&S Technical Data Package, including Functional Requirements, Specifications, Procedures, 

System Overview, Configuration Management Plan, and Quality Assurance Program. The review was 

against the EAC VVSG 2005, Volumes I and II, for each of the submitted individual components of 

the Unity 3.2.0.0 Voting System. For any FEC VSS 2002 requirements that are more demanding than 

the EAC VVSG 2005, SysTest Labs reviewed the ES&S documents to those 2002 requirements. 

SysTest Labs has conducted a Functional Configuration Audit review of the vendor test cases 

delivered as part of the Technical Data Package. The review was conducted against the FEC VSS 

2002 Volume I, for each of the submitted components. Any requirements that were identified as not 

being tested, or insufficiently tested, have been included in the Test Cases that SysTest will execute. 

The results of these audit reviews, as well as the discrepancies generated, will be included in the 

Certification Report. 

Unity 3.2.0.0 is a full certification and thus all code is subject to a full review against the FEC VSS 

2002. SysTest Labs has conducted a source code review of the submitted code in the following 

languages: C, C++, and Cobol. Tools utilized by SysTest include Practiline Line Counter: a 

commercial application used to determine the counts of executable and comment lines; Module Finder: 

a SysTest proprietary application used to parse module names from C/C++ and VB code and populate 

the identified module names into the review documents; ExamDiff Pro: a commercial application used 

to compare revised code to previously reviewed code; and KEdit: a commercial text editor application 

running a SysTest proprietary macro used to parse module names from Cobol code and populate the 

identified module names into the review document. 

SysTest Labs utilizes a team approach in reviewing and managing the tasks of receiving the code to be 

reviewed, determining the volume of code to be reviewed, reviewing the vendor's internal coding 

standards and determining if there are any variances from the prescribed Standards, creating the review 

work documents, distributing the code to be reviewed along with the created work documents to the 

project code reviewers, reviewing the code, performing peer reviews, creating discrepancy reports, and 

receiving modified code and other vendor responses. 

 

Prior to testing, SysTest Labs will conduct a trusted build according to the approved standard lab 

procedure that details this process. The process includes interviews of key vendor staff to evaluate 

vendor processes and process conformance in the areas of configuration management and quality 

assurance. Preparation for the trusted build includes obtaining and reviewing the vendor-defined 

procedure for constructing the build platform, verifying the target build platform, and acquiring the 

necessary materials. Execution of the trusted build complies with SysTest Labs’ detailed build 

procedure and results in a compact disk containing the installation files. Finally, the conclusion of the 

trusted build consists of record-keeping and archiving procedures that occur at SysTest Labs, and the 

resulting media is submitted to the EAC-approved software repository. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 7 of 19 

2.2 Pre-Certification Assessment Results 

SysTest Labs has conducted a pre-certification test assessment. This assessment includes the review of 

the Technical Data Package (Documentation and Vendor Testing) and source code changes for each 

submitted component of the Unity 3.2.0.0 Voting System. 

Based upon the findings of the pre-certification assessment for ES&S, SysTest Labs has determined 

that ES&S’ Test Plan, Procedures, and Scripts are consistent with the FEC Voting System Standards 

for TDP documentation. Issues were noted in a discrepancy report that was provided to ES&S for 

resolution prior to completion of testing. 

Full assessment, review, and correction of the ES&S TDP will be completed as part of the execution of 

this Certification Test Plan. Also to be executed as part of this Certification Test Plan are the required 

Readiness Test, a sampling of ES&S’ test cases based on the supported functionality for this 

certification, the SysTest Labs Gen01 test case, and a system accuracy test. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 8 of 19 

3 MATERIALS REQUIRED FOR TESTING 

3.1 Software/Firmware 

Items identified in the table reflect all software and firmware required to perform hardware, software, 

telecommunications, security and integrated system tests. 

Table 2 - Matrix of Required Software/Firmware 

Manufacturer Application(s) Version Test Type 

ES&S Audit Manager 7.4.1.0 Pre-Voting 

ES&S ES&S Ballot Image Manager 7.6.0.0 Pre-Voting 

ES&S Election Data Manager 7.7.0.0 Pre-Voting 

ES&S Hardware Programming 

Manager 

5.5.1.0 Pre-Voting 

ES&S Election Reporting Manager 7.3.1.0 Post-Voting 

Manufacturer COTS Application(s) Version Unity Application 

Microsoft Corporation Microsoft Windows Windows ’98 or higher ERM 

RM/COBOL RM COBOL RUNTIME 

System 

11.01 ERM & HPM 

Linux Linux from scratch 6.0 DS200 

3.2 Equipment (Hardware) 

Equipment identified in the table reflects all hardware required to perform hardware, software, security 

and integrated system tests. 

Table 3 - Matrix of Required Hardware 

Item Manufacturer O/S Version Type 

intElect DS200 (Precinct Count Scanner) SN 

0002w/Ballot Box SN 3016 and SN 0010 

w/Ballot Box 2007, 

ES&S Hardware 1.0 

Firmware 1.0.0.0 

Voting 

Dell Optiplex gx260, s/n # 7B0WL21 

and SN#1GIJR1480FL

Dell Windows XP, SP2 Personal Computer

3.3 Test Materials 

Items identified in the table reflect all test materials required to perform hardware, software, security 

and integrated system tests. 

Table 4 - Matrix of Test Materials

Item 

Printer paper rolls 

PC Card 

USB Memory Stick 

3.4 Deliverable Materials 

In addition to the hardware and software identified in sections 3.1, 3.2, and 3.3, ES&S delivered the 

Technical Data Package for the Unity 3.2.0.0 Voting System. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 9 of 19 

Please see Attachment A for a complete list of TDP documents.

3.5 Proprietary Data 

SysTest Labs will indicate which portions of reports are considered proprietary information. We 

understand material that is not classified as proprietary, including test plans and test reports, will 

become publicly available. Proprietary information will be submitted in a separate attachment to the 

EAC, and marked “Proprietary” in the file name. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 10 of 19 

4 TEST SPECIFICATIONS 

4.1 Hardware Configuration and Design 

Certification testing will occur for conformance to Vol. 1 Sect. 4 Hardware Standards and Vol. 2 Sect. 

4 Hardware Testing of the EAC VVSG 2005. 

The Hardware Configuration Audit will confirm that the configurations match. In order to conduct 

system level integration tests, SysTest Labs may need to include minimal repeats of the operational 

tests to confirm there are no changes to systemic responses. 

Table 5 - Matrix of Hardware Configuration 

Item Manufacturer Serial Number O/S Version Hardware 

Test and test 

lab 

Description of 

Use 

In Precinct 

intElect DS200* ES&S 0001, 0002, 0003, 

0004, 0010, 0011 

FW v. 1.0.0.0 

HW v. 1.0 

See Hardware 

Test Plan 

Scanner 

Steel Ballot Box ES&S 2007, 3016, 2804 N/A See Hardware 

Test Plan 

Ballot Box 

AC Power Adapter Wall Industries S/N 72573403; S/N 

72573415; S/N 

72573407, S/N 

72573413; S/N 

72632719, S/N 

72632720 

N/A See Hardware 

Test Plan 

Power Adapter 

Thermal Paper* Seiko P/N NAP-0080-

025S - (SS080-

025B) 

N/A See Hardware 

Test Plan 

Paper 

*The Seiko printer is embedded in the intElect DS200 

4.2 Software System Functions 

The scope of testing for the software certification (Vol. 2, Sect. 5) and system-level tests (Vol. 2, Sect. 

6) as defined in the FEC VSS April 2002 includes: 

• Pre-Certification Test Assessment (Vol. 2, Section A.2), reflecting the Technical Data Package 

(EAC VVSG 2005 Vol. 2, Sect. 2) document examination portions of the Physical 

Configuration Audit and the Functional Configuration Audit 

• Physical Configuration Audit: (Vol. 2, Sect. 6.6) 

o Establishment of the software/hardware configuration baseline used in testing 

o Full Source Code Review (Vol.2 Sect. 5.4) 

o Review of ES&S’s functional specification for adequacy or discrepancy 

o Execution of the Trusted Build and comparison to the code tested 

• Functional Configuration Audit: (Vol. 2, Sect. 6.7) 

o Creation and issuance of a Master Certification Test Plan (Vol. 2, Section A) 

o Review, evaluation, creation, and execution of Functional Tests (Vol. 2, Section A) 

o Performance of System Level Integration Tests (Vol. 2, Sect. 6) 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 11 of 19 

4.2.1 System Functional Testing 

Review of ES&S’s functional specification, test plans, test cases, and test results demonstrates the 

following functional areas are included in the Unity 3.2.0.0 System’s overall capabilities, pre-voting, 

voting, and post-voting functions. This functionality will be verified by the tests performed (Vol. 2, 

Section A). System functional tests shall be performed on the ES&S AM, EDM, ESSIM, HPM, ERM, 

and DS200 for the purpose of assessing the response of the software to a range of conditions (Vol. 2, 

Sect. 4.3.5). 

 

Table 6 - Matrix of System Functional Testing 

Function Test Methodology 

Ballot Preparation Functions 

a. Ballot preparation subsystem Verify the election is defined for election day, and one more 

precinct/polling place can be defined. 

Before, During & After Processing of Ballots 

b.1. Logic Test – Interpretation of Ballot Styles & 

recognition of precincts 

Verify in Functional Tests: 

Verify voting variation functionality identified by ES&S for 

the ES&S Unity 3.2.0.0 Voting System (Vol. 1. Section 

2.2.8.2). 

b.2. Accuracy Tests- Ballot reading accuracy Verify recording of 1,549,703 consecutive ballot positions on 

the DS200 scanner. 

-Accuracy Ballot is loaded onto a USB 

-Report of the initialization process 

-Display the function selections 

-Open polls 

-Zero Report 

-Scan ballots, Close polls, Run totals report and Audit Log 13 

times 

b.3. Status Tests- Equipment statement &memory 

contents 

Verify in Functional Tests: 

Equipment statement & memory contents at the 

corresponding intervals outlined in user documentation for 

the functions a. b.4, c 1-7 and d. 1-8 

b.4. Report Generation – Produce test output data Verify in Functional Tests: 

Clearing Election Totals 

Manual data entry 

Generating a Zero Report 

Testing an Election 

Creating Test Reports 

Clearing Totals for Election Day 

Selecting Reporting Groups 

Loading Scanner Totals 

Producing Election Reports 

Displaying Election Information 

ERM Election Results 

b 5. Report Generation- Produce audit data Verify in Functional Tests: 

System audit reports voting 

Polling Place Functions 

c.1. Opening the polls, accepting & counting ballots Verify in Functional Tests: 

Zero Reports 

DS200: Scan paper ballots 

Alerts for over votes and under votes 

c.2. Monitoring equipment status Verify in Functional Tests: 

Equipment status as identified in user documentation 

c.3. Equipment response to commands Verify in Functional Tests: 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 12 of 19 

Function Test Methodology 

Equipment (DS200) response to all voter and poll worker 

commands as identified in user documentation 

Messages generated by the equipment that require an action 

by the voter or poll worker before operation continues--as in 

blank ballots, overvotes, and undervotes as defined in 

election setup. 

c.4. Generating real-time audit messages Verify Verified in Functional Tests: 

Print DS200 audit log 

Each audit message contains a timestamp. 

Election name, software, and firmware are listed at the 

beginning of each audit log. 

Count of ballots processed is included in log of uploaded 

results. 

Error messages. 

Precinct ID is identified for all results pertaining to insertions, 

additions, and deletions.

c.5: Closing polls and disabling ballot acceptance Verify in Functional Tests: 

Inability to cast additional ballots 

DS200: Close of polls 

Inability to scan additional ballots 

c.6. Generating election data reports. Verify in Functional Tests: 

Generation of precinct reports 

c.7. Transfer ballot count to central counting location Verify in Functional Tests: 

Reading the USB from the ERM 

Telecommunication is not supported in this certification 

Central Count Functions 

d.1.Process ballot deck for > 2 precincts with 3 split 

precincts per precinct for a total of 6 ballot styles 

Verify in Functional Tests: 

Process of ballot decks on the DS200 

d.2. Monitoring equipment status Verify in Functional Tests: 

Equipment status as identified in user documentation 

d.3. Equipment response to commands Verify in Functional Tests: 

Equipment (DS200) responds to all voter and poll worker 

commands as identified in user documentation 

(Messages generated by the equipment that require an action 

by the voter or poll worker before operation continues--as in 

blank ballots, overvotes, undervotes as defined in election 

setup) 

d.4. Integration with peripherals equipment or other data 

processing systems

See b.3 

d.5. Generating real-time audit messages. See b.4 

d.6. Generating precinct-level election data reports See b.3 

d.7. Generating summary election data reports See b.3 

d.8. Transfer of detachable memory module to the 

processing equipment

See b.3 

GEN 01 is SysTest Labs’ System level test case for a general non-straight party election. The object of 

the GEN 01 test case is to test the core functionality of the voting system. This is accomplished by 

using the vendor's manual(s) to create a general election, vote the general election, and tally the 

results. GEN 01 will test such items as: a minimum of two precincts; split precincts, with three splits 

per precinct; multi-member boards; group voting; write-in voting; overvotes; and undervotes. GEN 01 

will cover all aspects of the voting cycle, from pre-voting functions, such as ballot creation (election 

definition) to post-voting functions (tally reporting and audit logs). 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 13 of 19 

Test Case 

GEN 01 The object of this test case is to verify core functionality by using the vendor's manual to create general 

election ballots, vote, and tally, with the following required functionality: 

- Define election contests, candidates, issues etc. (V1:2.2.6) 

- Maintain accurate and complete audit records (V1:2.2.5.2.1) 

- Maintain accurate and complete error and status messages (v1:2.2.5.2.2, 2.2.5.2.3) 

- Accurately record scanned ballots 

- Ensure undervotes are counted as cast votes 

- Reporting of separate accumulation of Undervotes and Overvotes 

- Ensure Overvotes are allowed when scanned 

- Maintain integrity of Vote and Audit data 

- Party affiliation is identified on the ballot 

- Accurate Definition, Count, Reporting for Election Day, Absentee - paper, with the results tallied 

(V1:2.2.2.1) 

- Write-in voting: Voting position identified for write-ins 

- Correctly tabulate (V1:2.2.8.1) 

- Have a Ballot Counter (V1:2.2.9) 

- Blank ballots

Table 7 - Matrix of Other System Testing 

System Tests Test Methodology 

Volume Test 

Accuracy Test Case System’s response to processing more than the expected number of 

ballots/voters per precinct, to processing more than the expected number of 

precincts, or to any other similar conditions that tend to overload the 

system’s capacity to process, store, and report data. 

Stress Tests 

Hardware test Plan Hardware test labs test the limits outside the range of ‘normal’ but within 

specifications for the units as defined in the VSS standards Vol.1: Section 3. 

System’s responses to transient overload conditions. Subject polling place 

devices to ballot processing at the high volume rates, evaluate software 

response to hardware-generated interrupts and wait states.

Usability Tests 

GEN01 Test Case Responses to input, text syntax, error message content, and audit message 

input. 

Accessibility Test 

Accessibility is not being supported in this 

certification. 

N/A 

Security Test 

Security Test Case Incorporates systems security provisions, unauthorized access, deletion or 

modification of data, audit trail data, and modification or elimination of 

security mechanisms. The vendor documentation will be reviewed to 

ensure sufficient detail is present to operate the voting system in a secured 

implementation. Where the vendor statements assert the voting system is 

secured via mechanisms and seals, procedures will test the presence and 

effectiveness of such controls. 

Telecom Test 

Telecom Test Case ES&S does not support data transmission via telecommunications. 

SysTest will not test transmission on the DS200. The DS200 does have an 

internal modem; however, it needs DAM to transmit. The DAM is not 

supported in this event. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 14 of 19 

System Tests Test Methodology 

Performance Tests 

GEN 01 Test Case and 

Accuracy Test (Test Hardware Test Plan) 

Test accuracy, processing rate, ballot format, handling capability and other 

performance attributes claimed by the vendor. Perform End-to-End System 

Level Testing validating a predicted result. 

Perform a Data Accuracy test validating reading of 1,549,703 consecutive 

ballot positions on the system Accuracy Test. Percept tests to requirements 

as defined in VSS Vol.1: Section 3. 

Recovery Tests 

Hardware Test Plan Exercise system’s ability to recover from hardware and data errors. 

Validate battery backup and recovery from error conditions, incorporated 

into Hardware test lab tests to requirements as defined in VSS Vol. 1: 

Section 3. (See also Hardware Tests)

4.3 Test Case Design 

4.3.1 Hardware Qualitative Examination Design 

SysTest Labs reviewed the overall system capabilities, pre-voting, voting, and post-voting 

functions. ES&S Unity 3.2.0.0 hardware is incorporated into the standard set of system level 

test cases with the augmentation of functionality-specific validation steps. 

4.3.2 Hardware Environmental Test Case Design 

Hardware environmental certification testing will be performed for conformance to Vol 1. Sect. 

3 of the FEC VSS April 2002. SysTest Labs’ environmental hardware test subcontractor, 

Percept Technology Laboratories, performs full environmental certification testing for 

conformance to Vol. 1 Sect. 3 Hardware Standards and Vol. 2 Sect. 4 Hardware Testing of the 

April 2002 FEC VSS as outlined in the ES&S Unity 3.2.0.0 Log, prepared by Percept 

Technology Laboratories (Vol .2, Section A.4.3.2). 

4.3.3 Software Module Test Case Design and Data 

SysTest Labs reviewed the test case design documents and data as provided by ES&S. In 

evaluating each module, with respect to flow control parameters and data on both entry and 

exit, SysTest Labs assesses the logical correctness, the adequacy of the code’s modularity and 

construction, the implementation of the algorithms in assembly language (if used), the absence 

of hidden code, and the extent to which “industry standard” characteristics are incorporated 

(Vol. 2, Section A.4.3.3). 

4.3.4 Software Functional Test Case Design 

SysTest Labs has reviewed the functional test case design documents and data as provided by 

ES&S against a detailed matrix of system functions and the test cases that exercise them. 

SysTest Labs has prepared a test procedure describing all test ballots, operator procedures, and 

the data content of output reports. SysTest Labs will design and conduct all appropriate module 

and integrated functional tests found necessary (Vol. 2, Section A.4.3.4). 

4.3.5 System-level Test Case Design 

SysTest Labs reviewed the system-level test case design documents and data as provided by 

ES&S. SysTest Labs will conduct all appropriate module and integrated functional tests found 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 15 of 19 

necessary, in addition to the standard set of system level tests run against all voting systems 

(Vol.2, Section A.4.3.5). 

4.3.6 Sampling Methodology 

SysTest Labs reviewed the system-level and functional test case documents and data provided 

by ES&S. SysTest Labs will repeat a sample of the vendor’s test cases according to the 

following guideline: 

Review all vendor test cases and select 1 test case from a high-risk area for sampling, such as: 

• Security 

• Data Accuracy 

• Audit log 

• Tabulating 

Test Cases that will be sampled: 

SysTest Labs has selected test case 5.4 - Polls Closed: Audit Log Report as the sample test 

case because audit log reporting is an integral part of testing voting systems. The audit log 

records every action taken during the course of the election. The audit log not only keeps a 

record of tally information, but also keeps a record of status and error messages. The audit log 

can also be useful in litigation if an election is ever in question.

4.4 EAC Interpretations 

This test engagement utilizes only standard VSTL test methods that conform to the EAC Testing and 

Certification Program Manual and the appropriate voting system standard. No additional EAC 

interpretations affect the test plan and test methodology. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 16 of 19 

5 TEST DATA 

 

5.1 Data Recording 

The FEC Voting System Standards, Volume 2 Test Standards, will measure certification-testing 

progress against the standards defined for paper-based systems. SysTest Labs will create forms for the 

source code, TDP, and testing reviews. They will be stored in electronic format at SysTest Labs. 

SysTest Labs will record all activity via status report emails to the voting system vendor, ES&S. 

The testing process involves the assessment of: 

• Operational accuracy in the recording and processing of voting data, as measured by the error 

rate articulated in Volume I, Section 3. 

• Operational failure or the number of unrecoverable failures under conditions simulating the 

intended storage, operation, transportation, and maintenance environments for voting systems, 

using an actual time-based period of processing test ballots. 

• System performance and function under normal and abnormal conditions; completeness and 

accuracy of the system documentation and configuration management records to enable 

purchasing jurisdictions to effectively install, test, and operate the system. 

5.2 Test Data Criteria 

SysTest Labs will evaluate test results against the documents and software provided by ES&S. These 

documents shall be used to customize a standard set of system level tests. Testing will be conducted as 

an independent verification and validation across the entire voting system. A greater depth of testing 

will be given to places where there are code changes and changes to documentation. In the standard 

system level tests, elections are customized to the functionality supported by the Unity 3.2.0.0 Voting 

System. System performance shall be measured against a predicted result. 

5.3 Test Data Reduction 

SysTest Labs will process the test data by manually recording data in the Test Case records and 

SysTest Labs templates. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 17 of 19 

6 TEST PROCEDURE AND CONDITIONS 

6.1 Facility Requirements 

Testing will be performed on site at ES&S, Omaha, NE, in a secured room. All TDP and test 

documentation is stored on site at SysTest Labs, Denver, CO, in the project directory on the Voting 

server. 

VSTL testing at a client site must meet the conditions under which testing is performed at SysTest 

Labs’ laboratory. 

SysTest Labs performs VSTL functional and system level testing at SysTest Labs’ facility unless 

testing at the client site is necessitated by logistics related to the characteristics of one or more 

components of the voting system under test. Environmental hardware testing for voting systems may 

only be executed at the environmental hardware testing subcontractor’s facility or their alliance lab 

facilities. 

Prior to any VSTL test activities that occur at a client’s site, both equipment and facility will be 

examined and analyzed to ensure that competent and up-to-date temporary facility support exists for 

testing of voting system technologies, in compliance with SysTest Labs’ VSTL test standards and 

accepted practices of test engineering. If the equipment or facility is found not to be in compliance, 

SysTest Labs will identify, to the client, any necessary improvements in the equipment and facility. 

SysTest Labs assures that these improvements are made before client-site testing can begin. 

When testing is performed at a Vendor facility, SysTest Labs requires that our test staff follow all 

protocols associated with recording, reporting, maintaining and controlling all test results. In addition, 

to ensure the integrity of all tests and recorded results, SysTest Labs requires that all test results be 

stored only on computer equipment provided and controlled by SysTest Labs, e.g., test staff laptop 

computers. SysTest Labs will control access to the test equipment, including hardware, software and 

firmware and the test room. 

SysTest Labs always ensures voting rooms doors are kept locked at all times, unless the current 

activity requires that the door be opened. Vendors are never left unattended in a voting room at any 

time. 

6.2 Test Setup 

The ES&S Voting System test platform will be set up, as part of the Physical Configuration Audit, in 

the standard configuration identified in the vendor TDP documents listed in Attachment A TDP 

Documents. The software will be installed, versions verified, and made operational. The hardware will 

also be set up and versions verified according to the vendor TDP documents. Once the hardware and 

software have been set up, SysTest Labs will proceed with testing the system. 

6.3 Test Sequence 

While there is no required sequence for performing voting system certification testing and audits, 

predecessor tasks are required for some testing. Tasks and any applicable predecessor tasks are 

identified in Table 6 - Matrix of System Functional Testing. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 18 of 19 

6.4 Test Operations Procedures 

The SysTest Labs VSTL Test Team will provide step-by-step procedures for each test case to be 

conducted. Each step shall be assigned a test step number and this number, along with critical test data 

and test procedures information, shall be tabulated onto a test report form for test control and the 

recording of test results. 

An inventory will be performed to verify the voting equipment received contains hardware and 

software elements as defined in the TDP. Prior to commencement of Functional System testing, the 

PCA will include verification that the system can be configured using the system operations manuals. 

Throughout the testing effort, test procedures will be marked with the test result of Accept or Reject. 

If a failure of a test procedure precludes attempting subsequent test procedures, the test procedures that 

cannot be executed will be marked as NT, Not Testable. NS (not supported) indicates requirements 

that apply to features that are not supported in the configuration being tested. For expected 

functionality that is not implemented, the test procedure will be marked as NT, Not Testable. If a test 

procedure is not applicable to the current certification test effort it will be marked as NA, Not 

Applicable. NA would also be entered for any subsequent step that is not applicable. Test results 

Reject, NT, and NA will include comments by the tester explaining the reason for the result. 

Issues encountered during review and testing will be documented on the Unity 3.2.0.0 Voting 

Discrepancy Report. Any non-conformities to the requirements of the FEC VSS April 2002 will be 

marked as Documentation Discrepancies or Functional Discrepancies (a discrepancy occurs when 

the voting system component does not meet defined requirements or specifications). ES&S must 

address all discrepancies prior to issuance of the Certification Report. Issues that are encountered 

during testing, but are not addressed by the FEC VSS April 2002 will be added to the Discrepancy 

report and noted as Informational. ES&S has the option to address Informational issues. All 

responses provided by ES&S are noted in the Discrepancy Report appendix to the Certification Report.

The test cases and procedures are contained in a separate document. 
 

Certification Test Plan 

Report No. 07-V-ESS-035-CTP-01 Rev 03 Saved date 9/27/2007 2:47 PM 

Template Rev 04 10/20/06 Confidential Page 19 of 19 

7 Approval Signatures 

SysTest Labs: 

James M Nilius 

Vice President, Compliance Services 

September 27, 2007 

Client: 

Sue McKay 

Director of Certification 

September 27, 2007 

End of Certification Test Plan