2675 South Abilene Street, #300,, Aurora, Colorado, 80014

Form- G VSTL Test Plan 

ES&S Unity 3.2.0.0 Rev. 1 Voting System 

Certification Test Plan for DS200 Modifications 

to the EAC Certified ESSUNITY3200

Prepared for 

Election System and Software

11208 John Galt Blvd.

Omaha, NE 68137

Version 2.0

EAC application # ESS1002

Trace to Standards

NIST Handbook 150-22

4.2.3, 5.3.5, 5.3.6, 5.4.2, 5.4.6, 5.5.1, 5.7 thru 5.7.3

HAVA

301

VVSG

Vol. # Section(s) # 

1 2, 3, 4, 5, 6, & 7

2 1.8.2.1

2 2, 3, 4, 5, & 6

2 Appendix A

iBeta Quality Assurance is accredited for Voting System Testing under:

EAC Lab Code: 0702 - Effective thru 7/16/11
ii

Version History

Ver # Description of Change Author Approved by Date

v.1.0 Initial submission to the EAC C. Coggins, J. Garcia, 

&S. Brown

C Coggins iBeta & 

Sue McKay ES&S

6/24/10

v.2.0 Correction of errors from EAC review: 

Title page & section 1: added EAC 

Application #ESS1002 

Section 4.4.1 Clarify assessment 

and correct ECO List

Section 7.1 typo corrected "not 

supported"

C. Coggins, J. Garcia, 

&S. Brown

C Coggins iBeta 6/25/10

This Test Plan follows the format identified NOC 09-001 & 09-005
iii

TABLE OF CONTENTS

1 INTRODUCTION ............................................................................................................................................ 5

1.1 MODIFICATIONS TO EAC CERTIFIED SYSTEMS........................................................................................... 5

1.1.1 Description of the ESSUNITY3200 Baseline ...................................................................................... 5

1.1.2 Location of the Modification Test Plan Required Content ................................................................. 5

1.1.3 RFI & NOC- Validity for the Test Campaign...................................................................................... 6

Table 1 EAC Requests for Interpretation and Notices of Clarification ........................................................................ 6

1.2 REFERENCES........................................................................................................................................... 7

1.2.1 Internal Documentation ..................................................................................................................... 8

Table 2 Internal Documents........................................................................................................................................ 8

1.2.2 External Documentation .................................................................................................................... 8

Table 3 External Documents ...................................................................................................................................... 8

1.2.3 Changes to the ESSUNITY3200 Technical Data Package Documents ............................................. 9

Table 4 Unity 3.2.0.0 Rev 1 voting system Technical Data Package Documents....................................................... 9

1.3 TESTING RESPONSIBILITIES ...................................................................................................................... 9

1.3.1 Project Schedule................................................................................................................................ 9

Table 5 –Sequence of Certification Test Tasks Schedule ........................................................................................ 10

1.4 TARGET OF EVALUATION DESCRIPTION OF CHANGES FROM ESSUNITY3200 ............................................ 11

1.4.1 System Overview............................................................................................................................. 11

1.4.2 Block Diagram ................................................................................................................................. 11

1.4.3 System Limits .................................................................................................................................. 11

1.4.4 Supported Languages ..................................................................................................................... 11

1.4.5 System Functionality ....................................................................................................................... 12

Table 6 –DS200 Functional Changes to ESSUNITY3200.......................................................................................... 12

1.4.6 Requirements Included and Excluded............................................................................................. 13

2 PRE-CERTIFICATION TESTING AND ANALYSIS .................................................................................... 14

2.1 EVALUATION OF PRIOR VSTL TESTING.................................................................................................... 14

2.2 REPORTS OF FIELD ISSUES..................................................................................................................... 14

3 MATERIALS REQUIRED FOR TESTING ................................................................................................... 16

3.1 VOTING SYSTEM SOFTWARE................................................................................................................... 16

Table 7 Voting System Software .............................................................................................................................. 16

3.2 VOTING SYSTEM HARDWARE AND EQUIPMENT......................................................................................... 17

Table 8 Voting System Hardware and other Equipment........................................................................................... 17

3.3 TESTING SOFTWARE, HARDWARE AND MATERIALS................................................................................... 17

Table 9 Testing Software, Hardware and Materials.................................................................................................. 17

3.4 DELIVERABLE MATERIALS ....................................................................................................................... 18

Table 10 Changes to Voting System User Documents from Certified ESSUNITY3200 ............................................. 18

Table 11 Changes to Voting System Materials from Certified ESSUNITY3200 ......................................................... 18

3.5 PROPRIETARY DATA............................................................................................................................... 18

4 TEST SPECIFICATIONS ............................................................................................................................. 19

4.1 REQUIREMENTS (STRATEGY FOR EVALUATING SECTIONS OF THE VVSG 2005) ......................................... 19

4.2 HARDWARE CONFIGURATION AND DESIGN............................................................................................... 19

4.3 SOFTWARE SYSTEM FUNCTIONS............................................................................................................. 19

4.4 TEST CASE DESIGN................................................................................................................................ 19

4.4.1 Hardware Environmental Test Case Design ................................................................................... 19

Table 12 DS200 Engineering Change Order Hardware Test Assessment ............................................................... 20

4.4.2 Software Module Test Case Design and Data ................................................................................ 20

Table 13 DS200 Functional Testing for Changes to ESSUNITY3200 ........................................................................ 21

5 TEST DATA.................................................................................................................................................. 23

5.1 TEST DATA RECORDING ......................................................................................................................... 23

5.2 TEST DATA CRITERIA ............................................................................................................................. 23

5.3 TEST DATA REDUCTION.......................................................................................................................... 23

6 TEST PROCEDURES AND CONDITIONS ................................................................................................. 24
iv

6.1 FACILITY REQUIREMENTS ....................................................................................................................... 24

6.2 TEST SET-UP ......................................................................................................................................... 24

6.3 TEST OPERATIONS PROCEDURES ........................................................................................................... 24

7 APPENDIX- TEST METHODS..................................................................................................................... 25

7.1 DS200 FUNCTIONAL TEST METHOD........................................................................................................ 25

7.1 DS200 ENVIRONMENTAL HARDWARE TEST METHOD ............................................................................... 31
Page 5 of 32

1 Introduction

This Test Plan identifies iBeta Quality Assurance’s (iBeta) approach to VSTL Certification Testing of the 

changes to the certified DS200 Precinct Count Scanner in the ESSUNITY3200 voting system supporting 

EAC application number ESS1002. The changes submitted in Unity 3.2.0.0 Rev 1 are tested to the EAC 

Voluntary Voting System Guidelines (VVSG 2005). The changes to the DS200 submitted in Unity 

3.2.0.0 Rev 1 have also been submitted to the Unity 3.2.1.0 certification test effort. In the instances 

where tests or reviews for these certification efforts are identical iBeta used these common results.

1.1 Modifications to EAC Certified Systems

EAC Notice of Clarification NOC 09-005: Development and Submission of Test Plans for Modifications 

to EAC Certified Systems identifies that "Test Plans submitted for modifications to previously EAC 

certified voting systems should be brief and structured to minimize test plan development and review". 

The NOC 09-005 identifies specific content of the test plan. The sections listed below address this 

content or lists it's location in the test plan.

1.1.1 Description of the ESSUNITY3200 Baseline

The ESSUNITY3200 certified voting system is a paper-based voting system that includes the:

Election management system election (EMS) preparation software: Election Data Manager, 

ES&S Ballot Image Manager, Hardware Programming Manager, AutoMARK Information 

Management System

EMS audit software: Audit Manager and LogMonitor

Pre-vote hardware: Ballot-on-Demand COTs printer

Polling place optical scanner hardware and firmware: Model DS200

Polling place ballot marker hardware and firmware: AutoMARK Voter Assist Terminal A100, 

AutoMARK Voter Assist Terminal Model A200

Central count hardware and firmware: Model 650

Central count EMS software: Election Reporting Manager

The EAC Certificate of Conformance ES&S Unity 3.2.0.0, found on the EAC website, provides the 

official description of the ESSUNITY3200 baseline.

1.1.2 Location of the Modification Test Plan Required Content

The DS200 change identification, description of the firmware changes, the category, functional area, 

reason for the change and applicable documentation changes are listed in Section 1.4.5 System 

Functionality. Changes to the DS200 hardware are listed in Section 4.4.1 Hardware Environmental Test 

Case Design

Detail regarding the review and initial assessment process for documentation, hardware and firmware is 

found in Section 2 Pre-certification Testing and Analysis. This section also includes a description of 

what will be tested (regression) to establish assurance that the change(s) create no inconsistencies with 

the TDP and further are correctly documented and reflected in the TDP. . Issues identified in the field 

are identified in Changes Related to Field Issues.

The description of what will be tested (regression) to establish assurance that the change(s) have no 

adverse impact on the compliance, integrity or the performance of the equipment is identified in Section 

4 Software Functional and System Level Test Case Design and the DS200 Functional Test Method, a.

summary of the test methods that will be used to validate compliance.

Titles of test lab personnel responsible for each aspect of the test campaign and the project schedule 

are found in Section 1.3 Testing Responsibilities.
Page 6 of 32

1.1.3 RFI & NOC- Validity for the Test Campaign

Handling of the EAC Requests for Interpretation and Notices of Clarification in Unity 3.2.0.0 Rev 1 are 

listed below.

Table 1 EAC Requests for Interpretation and Notices of Clarification

EAC RFI and NOC Title Addressed in Unity 3.2.0.0 Rev 1

EAC Decision on Request for Interpretation 2007-01, Rev. 2 2005 

VVSG Vol. 1 Section 3.2.2.1 (e)(AutoMARK Keypad)

Complied in ESSUNITY3200, no changes to the 

AutoMARK keypad

EAC Decision on Request for Interpretation 2007-02, 2002 Voting 

Systems Standards, Vol. 1, Section 4.2.5* (Single character names)

Names with a single character difference are 

identified and assessed in the review of source 

code changes

EAC Decision on Request for Interpretation 2007-03, 2005 VVSG 

Vol. 1 Section 3.1.1*(usability test review)

Complied in ESSUNITY3200, no changes to the 

VAT

EAC Decision on Request for Interpretation 2007-04, 2005 VVSG 

Vol. 1 Section 3.1.3*(Alternate language)

Complied in ESSUNITY3200, no changes to the 

VAT which is the voting equipment that addresses 

v.1: 3.1.3 Alternative Language 

EAC Decision on Request for Interpretation 2007-05, 2005 VVSG Vol. 

1 Section 4.2.1 (Testing Focus & Applicability)*

Complied in ESSUNITY3200, no COTS hardware 

changes to ESSUNITY3200

EAC Decision on Request for Interpretation 2007-06, 2005 VVSG Vol. 

1 Section 4.1.1, 2.1.2c &f, 2.3.3.3o & 2.4.3c&d. (Recording and 

reporting undervotes) *

Complied in ESSUNITY3200, no changes to 

counting of undervotes

EAC Decision on Request for Interpretation 2008-01, 2002 VSS Vol. 

II, 2005 VVSG Vol. II, Section 4.7.1 & Appendix C* (Temperature & 

power variation testing) 

Complied in ESSUNITY3200, no changes to 

hardware requiring temperature and power 

variation testing

EAC Decision on Request for Interpretation 2008-02, Battery Backup 

for Optical Scan Voting machines*

Complied in ESSUNITY3200, no changes to 

battery backup

EAC Decision on Request for Interpretation 2008-03 (Operating 

System Configuration) 2002 VSS Vol. 1: 2.2.5.3, 4.1.1, 6.2.1.1, Vol. 2: 

3.5; 2005 VVSG Vol. 1: 2.1.5.2, 5.1.1, 7.2.1, Vol. 2: 3.5*

OS configuration documentation is tested for 

completeness, clarity and consistency as part of 

the test environment configuration.

EAC Decision on Request for Interpretation 2008-04, 2002 VSS Vol. I, 

Section 2.3.1.3.1a 2005 VVSG Vol. II, Section 2.2.1.3a Ballot 

Production*

Complied in ESSUNITY3200, no changes to ballot 

production

EAC Decision on Request for Interpretation 2008-05 2002 VSS Vol. I, 

Section 3.4.2 2005 VVSG Vol. I, Section 4.3.2, Durability*

Complied in ESSUNITY3200, no changes to 

hardware that impact durability

EAC Decision on Request for Interpretation 2008-06, 2002 VSS Vol. I, 

Sections 3.2.2.4c, 3.2.2.5 2005 VVSG Vol. I, V. 1.0, Sections 4.1.2.4c 

(Electrical Supply), 4.1.2.5 (Electrical Power Disturbance) *

Complied in ESSUNITY3200, no changes to 

battery backup

EAC Decision on Request for Interpretation 2008-07; 2002 VSS Vol. I, 

Sections, 2.3.4, 2.3.5, 2.3.6, 2.4.1, 4.4.3, 9.4; 2002 VSS Vol. II, 

Sections, 3.3.1, 3.3.2; 2005 VVSG Vol. I, Sections, 2.2.4, 2.2.5, 2.2.6, 

2.3.1, 5.4.3; 2005 VVSG Vol. II, Sections, 1.3, 3.3.1, 3.3.2*

Complied in ESSUNITY3200, no changes to zero 

count r

EAC Decision on Request for Interpretation 2008-08 2002 VSS Vol. I, 

2005 VVSG Vol. I, Glossary (Bar code readers)

No bar code reader

EAC Decision on Request for Interpretation 2008-09 (Safety Testing) 

2002 VSS Vol. I, Section, 3.4.8 2005 VVSG Vol. I, Section 4.3.8*

Complied in ESSUNITY3200, no changes to 

hardware that impact safety

EAC Decision on Request for Interpretation 2008-10 (EFT) 2005

VVSG Vol. I, Sect. 4.1.2.6 2005 VVSG Vol. II, Sect,4.8*

Tested to IEC-61000-4-4 (2004-02)

EAC Decision on Request for Interpretation 2008-12 (Ballot marking 

Device/ Scope of Testing) 2005 VVSG Vol. 1: 2.1.5. System Audit 

2005 VVSG Vol. 1: 2.1.5.2 Shared Computing Platform*

Complied in ESSUNITY3200, no changes to the 

VAT ballot marking device

EAC Decision on Request for Interpretation 2009-001 (VVPAT 

Accessibility) 2005 VVSG Volume1: 7.8.2, 7.9.7*

No VVPAT

EAC Decision on Request for Interpretation 2009-02 (Alternate 

Languages) 2002 VSS Vol. I: 2.2.1.3a Ballot Production; 2005 VVSG 

Vol. I: 3.1.3 Alternate Languages*

Complied in ESSUNITY3200, no changes to the 

VAT which is the voting equipment that addresses 

v.1: 3.1.3 Alternative Language

EAC Decision on Request for Interpretation 2009-03 (Battery Back Up 

for Central Count): RFI 2008-06 (Battery Back Up for Central Count); 

2002 VVSS Volume I, Sections 3.2.2.4c, 3.2.2.5; 2005 VVSG Volume 

Complied in ESSUNITY3200, no changes to M650 

central count scanner
Page 7 of 32

EAC RFI and NOC Title Addressed in Unity 3.2.0.0 Rev 1

I, Version 1.0, Sections 4.1.2.4c (Electrical Supply), 4.1.2.5 (Electrical 

Power Disturbance) *

EAC Decision on Request for Interpretation 2009-04 (Audit Log 

Events); 2002 VSS VoI: 2.2.4.1, Common Standards, 2.2.5.1 System 

Audit; 2005 VVSG VoI: 2.1.4 Integrity, 2.1.5 System Audit, 2.1.5.1 

Operational Requirements, 5.4.3 In-Process Audit Records*

All test cases include verification that events are 

recorded in the audit log.

EAC Decision on Request for Interpretation 2009-05; 2002 VSS Vol. I, 

Sections, 2.2.7.2 c & d; 2005 VVSG Vol. I, Sections, 3.2.2.2 c ii & iii*

Complied in ESSUNITY3200, no changes to the 

VAT which is the voting equipment that supports 

audio capabilities 

EAC Decision on Request for Interpretation 2009-06 (Temperature & 

Power Variation Tests); 2002 VSS Vol. I, Sect. 3.4.3 2002 VSS Vol.

II, Sect. 4.7.1, 4.7.2, Appx Sec. C.4 2005 VVSG Vol. I, Sect. 4.3.3 

2005 VVSG Vol. II, Sect. 4.7.1, 4.7.3, Appx Sec. C.4 EAC Decision on 

RFI 2008-1*

Complied in ESSUNITY3200, no changes to 

hardware requiring temperature and power 

variation testing

EAC Decision on Request for Interpretation 2010-01; 2002 VSS Vol. I, 

Sections, 3.2.2.8; 2005 VVSG Vol. I, Sections, 4.1.2.8* (ESD Voltage 

Levels)

Tested at 2, 4, 6 & 8 kV

EAC Decision on Request for Interpretation 2010-02

2005 VVSG Vol. I, Section, 5.2.3 d, 5.2.5, 5.2.6, 5.2.7 b, c, d & e 2005 

VVSG Vol. II, Section, 5.4.2 (Coding conventions)

ES&S is using the VVSG coding conventions

Notice of Clarification NOC 07-001: Timely Submission of 

Certification Application*

ES&S is coordinating with the EAC project 

manager, VSTL selection has been declared

Notice of Clarification NOC 07-002: VSTL Work with Manufacturers 

Outside of Voting System Certification Engagements*

Unity 3.2.0.0 Rev 1 project is not outside of a 

certification engagement

Notice of Clarification: NOC 07-003: State Testing Done in 

Conjunction with Federal Testing within the EAC Program*

There is no state testing in conjunction with this 

project

Notice of Clarification: NOC 07-004: Voting System Manufacturing 

Facilities*

NOC addresses manufacture issues outside the 

VSTL's scope

Notice of Clarification 07-05: Voting System Test Laboratory (VSTL) 

responsibilities in the management and oversight of third party testing*

Changes do not require third party testing

Notice of Clarification NOC 08-001: Validity of Prior Non-Core 

Hardware Environmental and EMC Testing*

There is no reused of non-EAC certification 

hardware testing. 

Notice of Clarification: NOC 08-002: Clarification of EAC Mark of 

Certification Requirement*

NOC addresses manufacture issues outside the 

VSTL's scope

Notice of Clarification NOC 08-003: Clarification of EAC Conformance 

Testing Requirements for VSTLs*

Tests and reviews are developed by iBeta to 

confirm the changes conform to the VVSG

Notice of Clarification: NOC 09-001 Clarification of the Requirements 

for Voting System Test Laboratories (VSTLs) Development and 

Submission of Test Plans*

This test plan for modification to a certified system 

follows the sequence outlined in NOC 09-001 

edited with the NOC 09-005 content and traces 

Notice of Clarification: NOC 09-002: Clarification of EAC Laboratory 

Independence Requirement*

Tests and reviews shall not be conducted in the 

presence of ES&S. 

Notice of Clarification NOC 09-003: Clarification of De Minimis 

Change Determination Requirements*

Changes are to documentation are subjected to 

testing and/or review.

Notice of Clarification NOC 09-004: Development & Submission of 

Test Reports*

NOC required content will be incorporated into the 

test report

Notice of Clarification NOC 09-005: Development and Submission of

Test Plans for Modifications to EAC Certified Systems*

NOC required content is identified in Section 1.1

* Public document found on the EAC website

1.2 References

The documents listed below are used in the Unity 3.2.0.0 Rev 1 voting system certification test effort.
Page 8 of 32

1.2.1 Internal Documentation

The documents identified below are iBeta internal documents used in certification testing of the changes 

to the ESSUNITY3200.

Table 2 Internal Documents

Version # Title Abbreviation Date Author (Org.)

Voting Certification Master Services 

Agreement-Prepared for Election 

Software & Services (ES&S)

MSA contract 11/15/08 iBeta Quality Assurance

Statement of Work 11 SOW 11 iBeta Quality Assurance

DS200 Functional TC Unity 3.2.0.0 

Rev 1

6/21/10 iBeta Quality Assurance

FCA Environmental Test Case Unity 

3.2.1.0

3/25/10 iBeta Quality Assurance

Unity 3.2.0.0 Rev 1 DS200 Trusted 

Build Read Me

6/14/10 iBeta Quality Assurance

1.0 Trusted Build of the DS200 firmware 

1.4.3.0

2/12/10 iBeta Quality Assurance

4.0 Election Systems & Software Unity 

3.2.0.0 Voting System VSTL 

Certification Test Report 

(V)2009-30Jun-001(D) *

7/22/09 iBeta Quality Assurance

* Public document found on the EAC website

1.2.2 External Documentation

The documents identified below are external resources used to in certification testing of the changes to 

the ESSUNITY3200.

Table 3 External Documents

Version # Title Abbreviation Date Author (Org.)

Help America Vote Act* HAVA October 29, 

2002

107th Congress

NIST 

Handbk 150 

2006 Ed

NVLAP Voting System Testing NIST 150 February 2006 National Voluntary Lab 

Accreditation Program

NIST 

Handbk 

150-22

NVLAP Voting System Testing NIST 150-22 December 2005 National Voluntary Lab 

Accreditation Program

Voluntary Voting System 

Guidelines*

VVSG December 2005 Election Assistance 

Commission

EAC Certification of Conformance 

ES&S Unity 3.2.0.0 Election 

Systems and Software*

ESSUNITY3200

voting system

July 29, 2009 Election Assistance 

Commission

v.1.0 Testing and Certification Program 

Manual*

January 1, 2007 Election Assistance 

Commission

v.1.0 Voting System Test Laboratory 

Program Manual*

July 2008 Election Assistance 

Commission

Unity 3.2 Rev 1 Testing (Field Issue 

2 EAC email)

6/21/10 Election Assistance 

Commission

DS200 (Field Issue 2 ES&S email) 6/21/10 ES&S

See section 1.1.3 for a list of RFIs & 

NOCs *

Various dates Election Assistance 

Commission

Environmental Test Reports

EMC Qualification Test Report 

Election Systems and Software 

IntelElect Precinct Ballot Counter 

12/3/09 Criterion Technology
Page 9 of 32

Version # Title Abbreviation Date Author (Org.)

DS200 HW Rev. 1.2.1 Report 

Number: 090924-1464

EMC Qualification Test Report 

Election Systems and Software 

IntelElect Precinct Ballot Counter 

DS200 HW Rev. 1.2.1 Report 

Number: 091130-1503R

3/31/2010 Criterion Technology

* Public document found on the EAC website

1.2.3 Changes to the ESSUNITY3200 Technical Data Package Documents

The changes to the ESSUNITY3200 submitted for this certification test effort are listed below. 

Unchanged documents are found in the ESSUNITY3200 Test Report. 

Table 4 Unity 3.2.0.0 Rev 1 voting system Technical Data Package Documents

Title Version Date Author

Unity 3.2.0.0 Rev 1

Election Systems & Software System Overview Unity v. 3.2.0.0 Revision 1 2.0 06/21/10 ES&S

Unity 3.2.0.0 Revision 1 System Change Notes 3.0 no date ES&S

Requirements of the 2005 VVSG Trace to Vendor Testing and Technical 

Data Package

None 06/07/10 ES&S

Election Systems & Software ES&S Software Design Specifications 

DS200 Unity v. 3.2.0.0 Revision 1

3.0 12/22/09 ES&S

DS200 System Maintenance Manual Hardware Version 1.2.1.0 Firmware 

Version 1.4.3.0

None 05/07/10 ES&S

ES&S DS200 System Operations Procedures Hardware Version 1.2.1 

Firmware Version 1.4.3.0

None 06/21/10 ES&S

DS200 Precinct Ballot Scanner Election Day Training Manual Version 

Number 1.4.x

None 06/07/10 ES&S

1.3 Testing Responsibilities

1.3.1 Project Schedule

The VVSG 2005 identifies a general sequence of pre-test, test and post test activities. Any anomalies 

or errors are communicated to the manufacturer during all these activities. The schedule defined by 

iBeta details the tasks, dependencies, personnel and test phase. Predecessor tasks must be 

completed prior to task initiation. Tasks that do not have predecessors or dependencies may be 

performed in parallel.

1.3.1.1Owner Assignments

Staff assigned to the Unity 3.2.0.0 Rev 1 voting system certification test effort includes: 

Project Manager: Carolyn Coggins

Lead Tester/Reviewer: Jennifer Garcia

Tester/Reviewers: Gail Audette, Kirby Austin, Steven Brown, Sridevi Jakileti, Alastair Meyer, Kevin 

Wilson

Owner Assignments are identified in Table 5.

1.3.1.2Test Case Development

Test methods identified in section 7 Test Methods provide an outline of the content of the test cases. 

Detail regarding test method and test case development are incorporated into section 4 Test Case 

Design.

1.3.1.3Test Procedure Development and Validation

Test procedures and validations are documented in the test cases.
Page 10 of 32

1.3.1.4Third Party Tests

Changes have no impact on the ESSUNITY3200 third party tests.

1.3.1.5EAC and Manufacturer Dependencies 

EAC and Manufacturer dependencies are incorporated into Table 5.

Table 5 –Sequence of Certification Test Tasks Schedule

Certification Test Task Predecessor Tasks or Dependencies Owner Phase

Pre-Test Activities

Identify scope of project for 

contract negotiation

1. Preliminary discussion and overview examination 

of the system and documents for identification of 

the scope of testing, test standards, components 

and documents needed to conduct testing.

2. Delivery of the EAC certification application, 

implementation statement, System Overview, 

System Limits and Functional Design 

3. Identification of the EAC application and status of 

approval 

Coggins, Garcia June

Set up Project Repositories 1. Contract Authority Coggins June

Reporting of Discrepancies 1. Commencement of the project All staff On going

Identify any TDP changes 

relevant to the PCA TDP 

Document Review 

1. Project repository set up

2. TDP document delivery, check in & CM version 

check

Austin June

Restore & rebuild 

ESSUNITY3200 System 

Configuration

1. Delivery of the Unity 3.2.0.0 Rev 1 Overview, 

Functional Description, changed SW & HW TDP 

documentation

2. Restore Unity EMS on ESSUNITY3200 test 

hardware to matching images

Garcia, Austin, 

Brown

June

PCA Source Code Review 

of changed code

1. Project source code repository set up (shared

code will reside in Unity 3.2.1.0 project. Read 

me document will identify location until copied 

into Unity 3.2.0.0 Rev 1 final project archive)

2. Confirm no changes to ESS coding practices 

from ESSUNITY3200

3. Source code received, checked in, & CM version 

check

4. Identify changed code and prepare review sheets 

Jakileti, Meyer June

FCA Testing Review and 

Test Scope/ Requirements 

Identified; identification of 

Test Methods to test 

changes

1. Identification of changes to SW, HW and 

processes from ESSUNITY3200

2. Import the HW environmental test results from 

the Unity 3.2.1.0 project for the DS200 ECOs

Coggins, Garcia, 

Austin

June

Certification Test Plan 1. PCA pre-test examination changes to TDP, SW, 

HW, system configuration and source code.

2. Identification of scope in the PCA TDP Document 

Review, FCA Testing Review, ECO Assessments 

for changes to hardware

3. Preparation of Test Method for changes from 

ESSUNITY3200

Coggins, Garcia, 

Austin, Brown

June

Certification Test Plan 

Approval

1. Delivery of the Test Plan to the EAC

2. EAC Program Director review and approval

EAC staff June

FCA Test Case Preparation:

Functional and System Level

Regression

1. Preparation and peer review of the Test Method

2. Gather and review document changes to 

develop the Test Cases

3. Test Plan completion and approval may occur 

prior to Test Case development. Only the Test 

Method is a required pre-requisite 

Garcia, Austin June
Page 11 of 32

Test Activities

Trusted Build 1. PCA Source Code Review completed with no 

open source code or functional discrepancies

2. Validate COTS

3. Review and validation build procedure including 

user selections and configuration changes

4. Obtain and document build hardware

Jakileti, Meyer June

FCA Functional Test 

Execution

1. FCA Test Method & Test Case development is 

completed

2. DS200 build and installed; ESSUNITY3200

escrow retrieved and installed with the PCA 

System Configuration verified

Garcia, Austin, 

Brown

June

FCA System Level 

Regression Test Execution

1. Functional testing completed Garcia, Austin, 

Brown

June

FCA Environmental Test 

Execution

1. Use DS200 environmental test results and report 

from the Unity 3.2.1.0 certification 

Coggins, Garcia, 

Austin

June

Post Test Activities

VSTL Certification Test 

Report 

1. Completion of FCA and PCA tasks

2. EAC Program Director approves the Test Plan

3. Delivery of "Warrant of Accepting Change 

Control Responsibilities"

4. Update of as-run Test Plan

5. Implementation Statement

Coggins, Garcia, 

Austin, Brown

June

Deliver the Certification 

Report for EAC Review

1. Completion of VSTL Certification Test Report 

and matrix

Coggins June

Delivery of the Manufacturer 

letter signed by iBeta; 

trusted build, hash & images

1. EAC Technical Review of the Test Report

2. VSTL and/or manufacturer delivery to the EAC of 

any materials stipulated by the EAC review

3. EAC Technical Reviewer report to the EAC 

Program Director

4. Initial Decision issued by the EAC Executive 

Director 

ES&S July

Re-issue the Certification 

Report with the EAC 

Certification Number

1. Certification number issued by the EAC Coggins July

1.4 Target of Evaluation Description of Changes from ESSUNITY3200

The Unity 3.2.0.0 Rev 1 voting system changes to ESSUNITY3200 are identified below.

1.4.1 System Overview

The System Overview reflects the DS200 firmware version change to 1.4.3.0. All other aspects of the 

System Overview are unchanged from ESSUNITY3200.

1.4.2 Block Diagram

Changes have no impact on the ESSUNITY3200 block diagram.

1.4.3 System Limits

ES&S has submitted no change to the ESSUNITY3200 declared system limits.

1.4.4 Supported Languages

Enhancements 16120, 16291, and 16336 update the Spanish translations for errors. All other 

supported language functionality is unchanged from ESSUNITY3200.
Page 12 of 32

1.4.5 System Functionality

The DS200 enhancements and bug fixes submitted In Unity 3.2.0.0 Rev 1 are listed below. All other 

functionality is unchanged from ESSUNITY3200.

Table 6 –DS200 Functional Changes to ESSUNITY3200

ES&S 

Change ID

ES& S 

Category

DS200 

Menu or 

Function 

Change Description Reason for 

Change

Documentation 

Changed

General

BUG15827 Overvote 

handling

System Resolved an issue where the 

scanner failed to divert overvoted 

write-in ballots when the Divert 

Write-ins option was selected

Fixed Issue N/A – Bug Fix

BUG17375 Code Source 

Code

Fix source code discrepancies per 

VSTL review.

Fixed Issue N/A – Bug Fix

BUG17664 Code Source 

Code

Fix source code discrepancies per 

VSTL review.

Fixed Issue N/A – Bug Fix

BUG17666 Reports Status 

Report

Added the protected count to the 

status report that prints 

automatically when a DS200 is reopened for voting.

Fixed Issue N/A – Bug Fix

ENH14725 Interface Display Remove Image Drive icon from 

DS200 if images are not being 

saved

System 

Enhancement

N/A

ENH14726 Interface Display Extend the time that "Thank you for 

voting" displays

System 

Enhancement

N/A

ENH14728 Interface Display Provide clear indication that modem 

transfer was successful

System 

Enhancement

SOP – DS200

ENH14729 Poll Opening Reports Allow multiple zeros tapes to be 

printed before first ballot cast

System 

Enhancement

SOP – DS200 

ENH14730 Interface Audio Change alert beeping to just two

beeps

System 

Enhancement

N/A

ENH14731 Interface Audio Issue audible alarm when ballot is 

accepted

System 

Enhancement

N/A

ENH14732 Reports Results 

Reports

Repeat machine ID and poll number 

at end of results tape

System 

Enhancement

SOP – DS200

ENH14745 Exception 

Handling

Accept/Reje

ct Ballots

Provide override for Overvote or 

blank ballot rejection

System 

Enhancement

SOP – DS200

ENH15287 Reports Precinct 

Report

Add Early Voting Ballot Styles per 

Precinct Report

System 

Enhancement

SOP – DS200

System Functionality 

Description – DS200

ENH15288 Interface Display Increase Font Size of Thank you for 

Voting message

System 

Enhancement

SOP – DS200

ENH15418 Tabulation Scan 

Accuracy`

Small white dots "hickeys" causing 

read problems

System 

Enhancement

N/A

ENH15890 System Firmware Implement new scanner board 

firmware

System 

Enhancement

System Overview –

Unity 3.2.0.0 Rev1 

ENH15891 Security Counterfeit 

Detection

Implement new administration 

function to calibrate counterfeit 

sensor

System 

Enhancement

SOP – DS200

ENH15892 System Firmware Update scanner client to work with 

new scanner board firmware

System 

Enhancement

N/A

ENH16085 Interface Display Install New Icons on Welcome System SOP – DS200
Page 13 of 32

ES&S 

Change ID

ES& S 

Category

DS200 

Menu or 

Function 

Change Description Reason for 

Change

Documentation 

Changed

Screen Enhancement

ENH16120 Interface Display Updated the Over Vote warning 

screen 

System 

Enhancement

SOP – DS200

ENH16211 Reports Reports Print Machine ID & Poll Number in 

Audit Log and after report 

cancellations

System 

Enhancement

SOP – DS200

ENH16231 Audit Admin 

Menu

Enhanced audit logging to log all 

user actions in the Administration 

menu and attempts to access the 

Administration menu.

System 

Enhancement

SOP – DS200

ENH16291 Interface Display DS200: Additional language 

translations for the Over Vote screen

System 

Enhancement

SOP – DS200

ENH16336 Interface Display DS200: Update language 

translations for the Over Vote screen

System 

Enhancement

SOP – DS200

ENH16382 Tabulation System Expanded support from a maximum 

of 10 Election Day precincts to 18 

precincts.

System 

Enhancement

SOP – DS200

Unity 3.2.0.0 rev1 

System Limitations

ENH17266 Versioning System Update scanner board version System

Enhancement

System Overview – Unity 

3.2.0.0 Revision 1 

(U3200R1_OVR00)

ENH17268 Versioning System Promoted DS200 version 

implemented in Florida for use in 

Unity 3200r1. 

System 

Enhancement

N/A - Versioning

ENH17538 Protected 

Count

System Added a protected count to the 

DS200 firmware. The protected 

count resides on the compact flash 

card in the ES&S firmware partition.

It will increment with every sheet 

accepted and dropped into the ballot 

box. The counter must appear in 

printed reports

System 

Enhancement

SOP – DS200

Field 1

BUG16775 Tabulation System Resolved an issue that caused L&A 

test decks to yield incorrect vote 

totalsFixed Issue N/A – Bug Fix

BUG16782 Scanning System Resolved an issue that caused 

scanned Logic and Accuracy ballot 

test decks to yield incorrect results.

Fixed Issue N/A – Bug Fix

Withdrawn

ENH15009 Security Counterfeit 

Detection

ENH15009 Counterfeit ballot 

detection functionality has been 

withdrawn. ENH15890 to 15892 

remain as settings for the 

Counterfeit Ballot Sensor are 

required. 

System 

Enhancement

SOP – DS200

1.4.6 Requirements Included and Excluded

The Unity 3.2.0.0 Rev 1 voting system changes to ESSUNITY3200 are tested to the VVSG 2005. All 

other system inclusions and exclusions are unchanged from ESSUNITY3200.
Page 14 of 32

2 Pre-certification Testing and Analysis

A review of the testing performed by ES&S on the ESSUNITY3200 changes was conducted to assess 

the required scope of testing in conformance with the VVSG 2005.

PCA Source Code Review

The changes to the ESSUNITY3200 DS200 escrowed source code are reviewed to the VVSG 2005 Col. 

2 Section 5.4 source code review criteria. Source code review issues are identified in a discrepancy 

report delivered to ES&S. Issues identified included missing header and variable declaration comments 

and undefined constants. These issues were reported to ES&S on source code review discrepancy 

reports. All source code resolution resubmissions from ES&S (BUG 17375 and 17664 submitted to both 

Unity 3.2.0.0 Rev 1 and Unity 3.2.1.0) are reviewed and validated. iBeta confirms issues are addressed 

prior to closing the discrepancy.

PCA TDP Document Review and FCA Document Review

A comparison of the DS200 documents submitted in Unity 3.2.0.0 Rev 1 and the certified DS200 

ESSUNITY3200 documents identified all changes. The changes were reviewed to determine if there 

was any impact on the ESSUNITY3200 PCA Document Review or if the changes in the document 

require verification of a process or functional description of the voting system.

No Impact: Cosmetic, version update, format, or changes to software or a process which is not 

required by the VVSG.

Impact: Content required in Vol. 2 Section 2.2. of the VVSG is deleted; new documents are 

added; or addition, deletion or modification that describes changes to software, hardware and 

processes required by the VVSG must be verified

Items determined to impact the ESSUNITY3200 review or testing of the DS200 firmware or process were 

further examined to identify the type of review or testing required by the change

PCA Document Review: Changes removed any required VVSG Vol.2: section 2.2 content. New 

documents were reviewed for to the VVSG Vol.2: section 2.2.

FCA Functional Testing: Change to DS200 firmware that impacts voting system functional or 

security election operation

The document review found that no changes to the DS200 documents impacted the results of the 

ESSUNITY3200 PCA Document Review. Document changes which impacted DS200 functionality are 

incorporated in the DS200 Functional Test.

Hardware Engineering Change Order Review

The DS200 engineering changes submitted to Unity 3.2.0.0 Rev 1 were submitted and tested in Unity 

3.2.1.0. As the submitted (DS200 hardware is identical, the Unity 3.2.1.0 assessments of the 

engineering changes, required testing, and test results will be used in both certifications. 

2.1 Evaluation of prior VSTL Testing 

iBeta initiated an assessment to identify the changes to DS200 firmware and hardware in Unity 3.2.0.0 

Rev 01 (and Unity 3.2.1.0). Changes notes submitted for the DS200 were reviewed to identify the 

functional changes. Hardware engineering change orders submitted for Unity 3.2.1.0 and Unity 3.2.0.0 

Rev 1 were reviewed and assessed to identify changes that required repeating of any hardware test. 

2.2 Reports of Field Issues

Two reports of field issues have been identified for the DS200 certified in ESSUNITY3200.

Field Issue 1 is included in Unity 3.2.0.0 Rev 1 

1. The DS200 was sporadically reporting a mark present in row 44 and row 45 of column D on the 

back of the ballot when no actual mark was present. It was found that the specific ballot had 
Page 15 of 32

been printed with a slight skew. Additional the condition could only be reproduced when the 

ballot was inserted with a skew slight enough to not generate a rejection of the ballot. The 

abnormal skew revealed that the scanner was tolerance was reading text slightly outside the 

channel as a mark.

Field Issue 2 is currently under examination by ES&S. It will be addressed in a subsequent certification.

2. DS200 system lockup condition was observed during pre-election logic and accuracy testing for 

the Cuyahoga County primary election held in May 2010. Systems locked up and had to be 

restarted. Systems restarted immediately. Lock ups were random and could not be repeated. 

Cuyahoga County observations identified that the problem occurred more frequently when 

workers were conducting the Administrator functions. These functions tend to occur before or 

after the polls are closed at times of greater touch screen interaction. Touch screen interaction 

during voting is very limited. iBeta requested confirmation that there was no documented report 

of any issue resulting in the loss or corruption of voting data. Its presence shall be noted in the 

test report per VVSG Vol.2 section B.5.
Page 16 of 32

3 Materials Required for Testing 

The System Identification stipulates the following materials required for testing of ES&S Unity 3.2.0.0 

Rev 1.

3.1 Voting System Software

The software listed in below is the documented configuration of the ES&S Unity 3.2.0.0 Rev 1.

Table 7 Voting System Software

Application Manufacturer OS or 

Version

Description (identify COTS)

Election Management Software

Election Data Manager (EDM) ES&S 7.8.1.0 EMS software for election definition 

and ballot preparation for M650, 

DS200, and M100

ES&S Ballot Image Manager 

(ESSIM) 

ES&S 7.7.1.0 Unity election management system 

desktop publishing tool to layout and 

format paper ballots

Audit Manager (AM) ES&S 7.5.2.0 A Unity election management 

system audit logging software 

application including security and 

user tracking for the Election Data 

Manager and Ballot Image Manager

Hardware Programming Manager 

(HPM)

ES&S 5.7.1.0 A Unity election management 

system software application to 

import, format, and convert an 

election file and create election 

definitions for ballot scanning 

equipment

Election Reporting Manager (ERM) ES&S 7.5.4.0 A Unity central count software 

application to compile and report 

election results.

Log Monitor ES&S 1.0.0.0 A software application that checks 

the status of the Windows Event Log 

feature and closes all ES&S 

applications if the Event Log feature 

is disabled or not configured 

properly.

Microsoft Windows XP SP3 Microsoft SP3 COTS EMS Operating System

SQLXML Microsoft 3.0 service 

pack 3

COTS: XML support for Unity

Windows Internet Explorer Microsoft 7 COTS: AIMS & Unity PC 

ES&S does not want Internet 

Explorer to be run on the election 

System PCs. However, Internet 

Explorer must be resident on the PC 

to contain the latest security 

updates.

Adobe Acrobat v9 Adobe 9.0 COTS - Used with ESSIM

Adobe Type Manager 4.1 Adobe 4.1 COTS - Used with ESSIM

RM/COBOL v11.01 Runtime System Liant 11.01 COTS interpreter software used in 

HPM & ERM

Norton AntiVirus 2005 Symantec Corporation COTS Anti Virus protection for PCs 

and Servers

Precinct Scanner

DS200 Firmware

DS200 Power Management 

Firmware

DS200 Scanner Firmware

ES&S 1.4.3.0

1.2.0.1

2.20.0.0

Precinct count digital scanner paper 

ballot tabulator including a 12-inch 

touch screen display providing voter 

feedback and poll worker 

messaging. DS200 scanner reads 

marks on both one- and two-sided 

ballots.
Page 17 of 32

Application Manufacturer OS or 

Version

Description (identify COTS)

Administrators can request custom 

ballot acceptance criteria, which 

ES&S programs onto the scanner’s 

election definition.

3.2 Voting System Hardware and Equipment

The equipment listed in below is the documented configuration of the ES&S Unity 3.2.0.0 Rev 1.

Table 8 Voting System Hardware and other Equipment

Hardware or Equipment Manufacturer Version Description (identify COTS)

Election Management Hardware

(1) Dell GX260 computer 

desktop with monitor, keyboard 

& mouse

SN: Tower: 7D0WL21,

Dell Windows XP 

Professional 

Version 2002 

SP3

Pre-Vote & Post-Vote: COTS Unity PC for the 

Unity election management system

Condition: Good

Precinct Scanner

DS200 

(S/N: 2093900001)

ES&S Vote: A Unity Voting System precinct count 

optical scanner paper ballot tabulator 

including a 12-inch touch screen display 

providing clear voter feedback and poll 

worker messaging, Condition: Good

Steel Ballot Box

(P/N: 76246, S/N: C4243)

ES&S Vote: Precinct steel ballot box, with diverter to 

segregate ballots into multiple chambers as 

programmed in the EMS

Condition: Good

(3) SanDisk 2.0 GB USBs SanDisk COTS: Media for installing election definition, 

recording and reporting votes and audit logs

Thermal paper rolls NCR N/A COTS: DS200 reports

Paper Ballots Paper Ballots - 14" Supplied by ES&S: Ballots for DS200 with 

preprinted election content

Ballot Marker Pens Marking Device COTS: VL Ballot Pen to mark paper ballots

3.3 Testing Software, Hardware and Materials

The software, hardware and materials listed below are needed to support testing and in test simulations 

of elections of the ES&S Unity 3.2.0.0 Rev 1.

Table 9 Testing Software, Hardware and Materials 

Software, Hardware or Material Description Description of use in testing

Multiple desktop and laptop PCs A variety of PCs running Microsoft 

operating systems

Supplied by iBeta: Preparation, 

management and recording of test plans, 

test cases, reviews and results

Repository servers Separate servers for storage of test 

documents and source code, 

running industry standards 

operating systems, security and 

back up utilities

Supplied by iBeta: Documents are 

maintained on a secure network server. 

Source code is maintained on a separate 

data disk on a restricted server 

Microsoft Office 2003 & 2007 Excel and Word software and 

document templates

Supplied by iBeta: The software used to 

create and record test plans, test cases, 

reviews and results

SharePoint 2003 TDP and test documentation 

repository

Supplied by iBeta: TDP and test 

documentation repository and 

configuration management tool 

Other standard business application 

software

Internet browsers, PDF viewers 

email

Supplied by iBeta: Industry standard tools 

to support testing, business and project 

implementation

Center 325 Mini Sound Level Meter IEC 651 Type 2 handheld sound 

level meter

Supplied by iBeta: Measure decibel level

Visual Studio 2003 v.7.1.3808 Build and source code review Supplied by iBeta: View source code 
Page 18 of 32

Software, Hardware or Material Description Description of use in testing

(Microsoft) Integrated Development 

Environment

review 

RSM v.6.92 

(M Squared Technologies)

C, C++, Java & C# static analysis 

tool

Supplied by iBeta: identify line counts and 

cyclomatic complexity

Beyond Compare 2 v.2.4.3 (Scooter 

Software)

Comparison utility Supplied by iBeta: used to compare 

file/folder differences

WinDiff 5.1 (Microsoft) Comparison utility Supplied by iBeta: used to compare 

file/folder differences

Hash.exe v.7.08.10.07.12 (Maresware) Hash creation utility Supplied by iBeta: used to generate hash 

signatures for Trusted Builds

SLAX LIVE W/ SHA1DEEP v.5.1.8 Hash creation tool Supplied by iBeta: used to generate hash 

signatures for the DS200

Knoppix 5.1.0 Hash creation utility Supplied by iBeta: used to generate hash 

signatures

Symantec Ghost v.11 Image capture tool Supplied by iBeta: used to capture and 

test environments

Bart PE 3.1.10a Image capture utility Supplied by iBeta: used with Ghost 

process

Visual Studio 2008 v. 9.0.21022.8 

(Microsoft)

Build and source code review 

Integrated Development 

Environment

Supplied by iBeta: View source code 

review 

Killdisk v.4.1 PC clear utility Supplied by iBeta: used to wipe clean PCs 

and servers prior to testing

3.4 Deliverable Materials

Documents listed below are delivered, to the purchaser as part of the Unity 3.2.0.0 Rev 1 voting system 

changes.

Table 10 Changes to Voting System User Documents from Certified ESSUNITY3200

Version # Title Abbreviation Date Author 

(Organization.)

2.0 Election Systems & Software System 

Overview Unity 3.2.0.0 Revision 1

U3200r1_OVR00.pdf 06/21/2010 ES&S

ES&S DS200 System Operations 

Procedures Hardware Version 1.2.1 

Firmware Version 1.4.3.0

U3200r1_SOP00_ 

DS200.pdf

06/21/2010 ES&S

1.3.x AutoMARK Pre-Election Day Checklist U3200r1_TRN00_ 

AutoMARK_ 

PreElection.pdf

07/31/2009 ES&S

1.4.x DS200 Precinct Ballot Scanner Election 

Day Training Manual

U3200r1_TRN00_ 

DS200_Election.pdf

6/7/2010 ES&S

The materials listed below delivered as part of the ES&S Unity 3.2.0.0 Rev 1 voting system.

Table 11 Changes to Voting System Materials from Certified ESSUNITY3200

Material Material Description Use in the Voting System

intellect DS200 (DS200)

FW: 1.4.3.0

A Unity precinct count digital 

scanner

Digital Scan tabulator

The EAC Certificate of Conformance ES&S Unity 3.2.0.0, found on the EAC website, provides the 

official description of the ESSUNITY3200 baseline.

3.5 Proprietary Data

Proprietary data is handled in the same manner identified in ESSUNITY3200.
Page 19 of 32

4 Test Specifications

The analysis and assessments performed for the modification source code review, PCA Document 

review, and FCA Document Review is included in section 2. Testing for conformance to the VVSG 2005

shall be conducted as identified in the below. The test methods for testing the changes to the 

ESSUNITY3200 are contained in the appendix. Documentation of all test iterations shall be maintained 

with a separate record of the configuration and results of all test executions.

4.1 Requirements (Strategy for evaluating sections of the VVSG 2005)

The strategy for evaluation of any voting system submitted for certification is to confirm that the changes 

to ESSUNITY3200 conform to the Volume 1 requirements of the VVSG 2005 with validation in a 

functional system level regression test and review of ES&S internal test documentation.

4.2 Hardware Configuration and Design

The hardware configuration of the ES&S Unity 3.2.0.0 Rev 1 voting system submitted for testing is 

identified in Section 3.2 Voting System Hardware and Equipment. All test cases include verification and 

documentation of the test environment.

4.3 Software System Functions

Testing of the software system functions defined in the VVSG 2005 include:

Identification of the functional test scope based upon an analysis of the TDP changes to the 

ESSUNITY3200 from the PCA Document Review (Vol. 2, Sect. 2) and FCA review of prior 

testing (Vol.2 Appendix A.2)

PCA TDP Source Code Review of all new or changed code (Vol.2 Sect. 5.4) 

Witness the build of the reviewed code for the updated version of the system intended to be 

sold by the vendor and delivered to the jurisdiction. (Vol.2. Sect. 6.2)

Development of a Certification Test Plan and Test Cases (Vol. 2, Appendix A.)

Execution of Functional/System Integration Tests listed below (Vol. 2, Sect. 6)

Testing of the performance and sequence of the changed software functions identified in 

System Operations, Maintenance and Diagnostic Testing Manuals (Vol. 2. Sec. 6.8)

Completion of a trusted build by the VSTL with file signatures provided to the EAC stipulated 

escrow agency.

4.4 Test Case Design

4.4.1 Hardware Environmental Test Case Design

ES&S submitted the hardware and administrative engineering changes (ECOs) for the ESSUNITY3200

DS200 precinct scanner and the plastic ballot box/case for certification in Unity 3.2.1.0 and Unity 3.2.0.0 

Rev 1.

The administrative ECOs covered documentation relevant to the manufacturing administrative 

process. They incorporated details such as label changes, production status changes, out of 

scope equipment, documentation updates and drawings. These ECO's were assessed and 

determined to have no impact on the hardware which would require further environmental 

hardware assessment for impact to electrical, transportation or storage testing. This included 

the ECOs 000315, 000337, 000340, 000342, 000366, 000375, 000423, 000466, 000523, 

000545, 000554, 000562, 000566, 000570, 000582, 000618, 000628, 000665, 000669 000674, 

836, 837, 838, 839, 845, 846, and 851 The SysTest’s subcontractors Criterion Technology Inc. 

of Rollinsville, CO performed hardware testing for the Unity 3.2.1.0 certification test effort. The 

test record and report is applicable to both certifications.
Page 20 of 32

The hardware changes generally addressed end-of-life parts and second source suppliers

required further environmental hardware assessment for impact to electrical, transportation or 

storage. Descriptions and the findings of the assessments for testing required of these ECOs 

are listed in the table below.

Table 12 DS200 Engineering Change Order Hardware Test Assessment

ECO # Description of DS200 ECOs Operating- EMC Non-operating 

Transportation & Storage

000332 DS200 Plastic Ballot Box new lock. 4.8.3 Electrostatic Disruption 

EN-61000-4-2

No testing required

000339 DS200 Ballot Box carry case add washer & 

rivet to hold foam during manufacturing 

4.8.3 Electrostatic Disruption 

EN-61000-4-2

No testing required

000359 DS200 Plastic Ballot Box Adding metal 

bottom edge

4.8.3 Electrostatic Disruption 

EN-61000-4-2

No testing required

000529 DS200 Carry Case remove micro switch 

bracket and switch cable 

No testing required No testing required

000534 DS200 add clamps to chassis No testing required No testing required

000535 DS200 Tape and holes for attaching clamps, 

no change to wire routing

No testing required No testing required

000576 DS200 End of life SMT Power Inductor No testing required No testing required

841 DS200 add Rod Lens Array, capacitor, and 

protected power switch

All EMC Tests No testing required

843 Steel Ballot Box added a new diverter cable All except 4.8.8 Magnetic Fields 

Immunity EN-61000-4-8

No testing required

844 DS200 end-of-life capacitors, resistors and 

diode 

4.8.3 VSS Electro-static 

Disruption EN-61000-4-2

No testing required

847 DS200 alternate LCD Backlight Inverter All except 4.8.8 Magnetic Fields 

Immunity EN-61000-4-8

No testing required

4.4.2 Software Module Test Case Design and Data

The same changes to the DS200 firmware from the certified ESSUNITY3200 are being submitted in two 

separate EAC certification projects, Unity 3.2.0.0 Rev 1 and Unity 3.2.1.0. The identical firmware 

version is used in both efforts.

General Enhancements and Issues- Unity 3.2.1.0 DS200 Functional Test Case Scenarios 1 

through 7 were previously created and tested to address enhancements and issues. Unity

3.2.1.0 includes a later version of HPM and ERM. In order to verify that the DS200 works with 

the original ESSUNITY3200 certified versions of HPM and ERM the election definition created in 

Unity 3.2.1.0, using the original EDM v.7.8.1.0 (unchanged from ESSUNITY3200), will be used to 

execute the Unity 3.2.0.0 Rev 1 DS200 Functional Test Case. In scenario 1 the election 

definition will be pulled into the ESSIM v.7.7.1.0 (unchanged from ESSUNITY3200), and election 

media will be written in the original HPM v.5.7.1.0. Ballots will be cast on the DS200. The 

election results will be transferred to the original ERM v.7.5.4.0. It will be confirmed that the 

audit and election results are correctly reported. Scenarios 2 and 3 will be validated by writing 

the media in HPM, confirming expected results occur on the DS200 and that all applicable 

election results can be properly read into ERM. The changes on Scenarios 4, 5, 6 and 7 were 

limited to functions on the DS200; test results from Unity 3.2.1.0 will require no further 

verifications in Unity 3.2.0.0 Rev 1.

Field Issue 1: BUG16775 and BUG16782 were also submitted to both certification efforts but 

were handled in a different manner. Due to the highly unique conditions required to reproduce 

the skewed ballot issue identified in the field, iBeta assessed that it would be unlikely that they 

could unquestionably reproduce the test conditions. As permitted by Vol. 2: Section 6.7 of the 

VVSG, a document review of ES&S' testing is being conducted to confirm the issue was 

acceptably tested.

Greater detail is found in Section 7.1 DS200 Functional Test Method.
Page 21 of 32

Table 13 DS200 Functional Testing for Changes to ESSUNITY3200

Function Tested Enhancement/Issue ID

General Enhancements and Issues

DS200 Functional TC Scenario 1

Test the Diverter is set for "Overvoted Write-ins & Blank Ballots". Verifying these 

ballots will be separated from all other ballots.

BUG15827

The DS200 scanner options are set to "Do not save any ballot images" in HPM. 

Verifying an X appears on over the disk icon to indicate no images are being saved.

ENH14725

From the "Polls Opened Menu" verify the zero reports can be printed 3 times prior to 

opening of polls.

ENH14729

Verify the DS200 sounds out 2 beeps and a "successful" message displays once the 

scanner has been calibrated (Calibrate scanner option)

ENH14730

ENH15890

ENH15891

ENH15892

Verify that there is an audible beep as each ballot is accepted ENH14731

Test the 5 Ballot Control settings to Query in HPM will force the prompt to the voter to 

either "Accept" or "Reject" ballots. Override the "Overvoted ballots and Blank ballots 

only" querying setting on the DS200; verify this will automatically accept ballots due to 

the override.

ENH14745

Verify battery status, power status, image status, election definition and open polls 

icons are displayed on the "Welcome screen".

ENH16085

Verify a "Failure" error and a "Successful" status display on the DS200 and in the audit 

log when entering an incorrect and a correct administrator password.

ENH16231

Verify at the end of an audit log and results tape the "Machine ID" and "Poll Number" 

are display. Verify this occurs with and without canceling the reports.

ENH16211

ENH14732

Verify the "Machine ID" and "Poll Number" display at the end of the Results tape. ENH14732

Verify the DS200 limit of 18 precincts ENH16382

DS200 Functional TC Scenario 2

Verity an Early Voting election displays a Ballot Styles report option allowing the Poll 

workers to view each of the ballots styles assigned to that DS200.

ENH15287

View the icon bar for each of the icons on the "Welcome screen". Verify no "X" appears 

over the Disk Icon (indicator that images are being saved.)

ENH14725

Test the 5 Ballot Control settings to Query in HPM will force the prompt to the voter to 

either "Accept" or "Reject" ballots. Override the "Overvoted ballots and Blank ballots 

only" querying setting on the DS200; verify this will automatically accept ballots due to 

the override.

ENH14745

DS200 Functional TC Scenario 3

Verify that testing in scenario 1 confirms the DS200 limit of 18 precincts (exceeding the 

System Limit of 10 precincts) is correctly handled through ERM. 

ENH16382

DS200 Functional TC Scenario 4

Measure and verify the "Thank you for voting" text is larger. (Compare: old v.1.3.10.0 to 

the new version of the DS200 firmware)

ENH15288

Time and verify the "Thank you for voting" screen displays 3 to 5 seconds longer. 

Compare: old v.1.3.10.0 to the new version of the DS200 firmware

ENH14726

Test an Election database and ballots with stray marks in the time track (provided by 

ES&S). Scan each ballot of the older DS200 version (v. 1.3.10.0) and on the new 

version (v.1.4.3.0). The old version will display an error "BALLOT DRAGGED/Turn 

Ballot Over and Try Again". The new version will accept the ballot.

ENH15418

DS200 Functional TC Scenario 5

Test the Added Protective Counter (for Maryland). Test by recording the protected 

count, scan ballots, close polls and record the protected count. Using an "updated" FW 

version provided by ES&S, upgrade the FW on the DS200. Again note the protected 

count. Scan ballots and record the protected count. Close the polls and record the 

protected count. Restore the original FW and again note the protected count. Verify the 

protected count is accurate and correct on the Audit logs, Initial state and status reports

ENH17538

ENH17666

DS200 Functional TC Scenario 6

Support for the Counterfeit Ballot Detection has been withdrawn. (Calibrating the

Counterfeit Sensor is required in Unity 3.2.0.0 Rev 1. It is tested in Scenario 1)

ENH15009

DS200 Functional TC Scenario 7

Test Additional language translations for the Over Vote screen by going to Regression 

REV01 TC Scenario 1 for testing of Overvote translations.

Verify updated Overvote error translation text is displayed 

ENH16291

ENH16336

ENH16120
Page 22 of 32

Function Tested Enhancement/Issue ID

Using an On a Spanish ballot verify the following message appears when too 

many votes have been selected on 1 or more contests: "Correct your ballot/Corrija 

Su Boleta" or "Cast your ballot with errors/Emitir su boleta con errores".

Field Issue 1

Document Review of ES&S Test Results

The DS200 was sporadically reporting a mark present in row 44 and row 45 of column 

D on the back of the ballot when no actual mark was present. The ballot sizes in 

question were the 17 inch and/or 19 inch. ES&S provided iBeta with their assessment 

of the issue on the DS200 FW (v.1.3.10.0) as well as their test case and fixes. 

Conditions to generate the problem involved a very narrow set of variables including: 

location of the contest from the vertical timing tracks, proximity to the top or bottom of 

the ballot, left side of the column oval placement, extension of the text next to the oval 

the edge of the allowable print area specified in the print layout manuals and feeding of 

the ballot skewed fashion and no selection on the last contest in column D iBeta 

reviewed and verified the highly unique conditions required to generate the error were 

identified by ES&S so that they were able to consistently repeat the error in v.1.3.10.0. 

The resolution of the error was consistent with the VSS, i.e. tolerance adjustments so 

that ballots inserted in this skewed manner would be rejected and require reinsertion. 

A review of ES&S test case confirmed their testing was sufficient and accepted. No 

additional testing is required.

BUG16775

BUG16782
Page 23 of 32

5 Test Data

5.1 Test Data Recording

The results of testing and review to the Unity 3.2.0.0 Rev 1 voting system to the VVSG 2005 are 

recorded in the test case and review forms prepared by iBeta. Electronic copies of all testing and 

reviews will be maintained. 

5.2 Test Data Criteria

The results of the voting system tests and reviews results shall be evaluated against the documentation 

of the Unity 3.2.0.0 Rev 1 voting system TDP, and the requirements of the VVSG 2005. The changes 

submitted in the Unity 3.2.0.0 Rev 1 voting system shall be evaluated for performance against the 

standard with the expected results identified in each test case.

Test cases shall identity the election, ballot and vote inputs. Outputs shall be verified against the 

expected results including the ballot displays, functions initiated during voting and the precinct and 

summary reports.

5.3 Test Data Reduction

Test data will be processed manually.
Page 24 of 32

6 Test Procedures and Conditions

6.1 Facility Requirements

All software testing and review will be performed at iBeta laboratory in Aurora. All electrical tests on the 

DS200 hardware was performed at the facilities of sub-contractor Criterion Technologies, Inc. 

Rollinsville, Colorado.

All ES&S documentation, test documentation and results will be maintained in the ES&S Unity 3.2.0.0 

Rev 1 project folder on the SharePoint server. Only project assigned test personnel will have access to 

the repository. ES&S source code will be maintained on a separate server. Only project assigned test 

personnel will have access to the source code repository. Repositories are backed up daily using 

industry standard utilities.

6.2 Test Set-up

As part of the PCA, the ES&S Unity 3.2.0.0 Rev 1 test platform will be set-up in the manner identified in 

the system configuration. The test platform will be documented. Installation of the trusted build for the 

changes from ESSUNITY3200 will be observed and documented. Unchanged software will be installed 

from the ESSUNITY3200 escrow. An inventory of any accessories or preloaded applications will be 

documented. Conditions necessary to reproduce the test set up are contained in the Test Case and 

PCA Configuration documents.

6.3 Test Operations Procedures

Test cases and review criteria are contained in separate documents. They are provided to the iBeta 

test staff with step-by-step procedures for each test case or review conducted. Test and review 

instructions identify the methods for test or review controls. Results are recorded for each test or review 

step. Possible results include:

Accept: the expected result of the test case is observed; an element of the voting system 

meets the VVSG 2005.

Reject: the expected result of the test case is not observed; an element of the voting system did 

not meet the VVSG 2005.

Not Applicable (NA): test or review steps that are not applicable to the scope of the current 

Certification are marked NA.

Not Testable (NT): rejection of a previous test step prevents execution of this and subsequent 

test steps.

Reject, Not Applicable and Not Testable results are marked with an explanatory note. The note for 

rejected results contains the discrepancy number.

Issues identified in testing or reviews are logged of Unity 3.2.1.0 Rev 1 on the Discrepancy Report. 

Issue types include:

Document Defects: a documentation element of the voting system did not meet the VVSG 2005. 

Resolution of the defect is required for certification. 

Functional Defects: a hardware or software element of the voting system did not meet the 

VVSG 2005. Resolution of the defect is required for certification.

Informational: an element of the voting system which meets the VVSG 2005 but may be 

significant to either the vendor or the jurisdiction. Resolution of Informational issues is optional. 

Unresolved issues are disclosed in the certification report.

Test steps are numbered and a tabulation of the test results is reported in the test case. Test operation 

personnel and their assignments are identified in the Project Schedule. Steps necessary to reproduce 

the test results are contained in the Test Case documents.
Page 25 of 32

7 Appendix- Test Methods

7.1 DS200 Functional Test Method

The test method description incorporates all election conditions and details as it was set up in the Unity 3.2.1.0 

DS200 Functional Test Case and the sequence of execution identified in the Test Case Design

iBeta Definition DS200 Functional Test Method

Test Case Name DS200 Functional TC

Scope - identifies the 

type of test

The changes to the DS200 submitted in Unity 3.2.0.0 Rev 1 have also been submitted in the Unity 3.2.1.0 

certification test effort. Seven test scenarios are listed within this test method. Scenarios 1, 2 & 3 require 

execution in Unity 3.2.0.0 Rev 1 to confirm that the changes to the DS200 integrate with the 

ESSUNITY3200 Election Management System. . Scenarios 4 through 7 require no additional testing 

because they are isolated to the DS200. The ES&S Unity 3.2.1.0 test results will be use. In Unity 

3.2.1.0 the election creation was performed on a Peer-to-Peer configuration (multiple PCs). In Scenarios 

1, 2 & 3 election data will be copied to the ESSUNITY3200 Election Management System Stand Alone 

configuration. New media will be burned, ballots will be scanned on the DS200 and election results will 

be read into ERM. Testing shall confirm the functional changes in the DS200 firmware v. 1.4.3.0: 

BUG15827, ENH14725, ENH14726, ENH14729, ENH14730, ENH14731, ENH14732, ENH14745, 

ENH15287, ENH15288, ENH15418, ENH15890, ENH15891, ENH15892, ENH16085, ENH16211, 

ENH16231, ENH16382, ENH15418, ENH17538, ENH17666, ENH16120, ENH16291, ENH16336, A 

Document Review shall be conducted of ES&S testing of BUG16775 & BUG16782 (Field Issue 1) to 

confirm sufficiency. 

(Note: Support of counterfeit ballot detection ENH15009 is withdrawn. Modem functionality is not 

supported in this certification effort (ENH14728)

Test Objective The objective is to validate the ES&S functional changes to the DS200 from the ESSUNITY3200 certified 

baseline. These changes were submitted in Unity 3.2.1.0. Scenario 1 through 3 shall confirm that the 

same results from Unity 3.2.1.0 are observed. Unity 3.2.1.0 results from document reviews and testing 

conducted in Scenarios 4 through 7 shall be used.

Test Variables: 

Volume

Stress

Performance

Error Recovery

Scenario 1:

Reuse the Illinois Open Primary without Party Preference election defined in the Unity 3.2.1.0 test effort 

with the following settings:

ENH16382•18 Precincts election day and 1 Polling Place

9 District Types (this does not including Countywide)

18 District Name (two contests to ac district name)

18 ballot style (each precinct has its own ballot style)

14" 36 "standard 14" ballot

2 Partisan contest per style

4 candidates per contest (2 candidates per party for the mayor contest, 4 candidates for the Senator 

contest (total of 216 candidates) 

Write-ins (1 for the Mayor and 2 for the Senator race)

DEM and REP Parties

ENH14745 & BUG15827•The Scanner Options

•Diverter is set for "Overvoted Write-ins & Blank Ballots".

•Ballot Control is set to Query for "Overvoted ballots, Cross voted ballots, Unreadable marks, Undervoted 

ballots and blank ballots"

ENH14725•The DS200 scanner options are set to "Do not save any ballot images" 

Run Election set up reports in EDM

Scenario 2: Same as Scenario 1 except:

ENH15287 •Changes "election day" to "early voting" in HPM by selecting the "Include all Precincts" 

option. The early voting option allows the Poll worker to view ballot styles for that specific ballot box.

ENH14725• The DS200 scanner options are set to "save all ballot images"

Scenario 3: Use Scenario 1 test results:

ENH16382•DS200 18 precincts Election Day can exceed the declared System Limit of 10 precincts. 

Scenario 4: Reuse Unity 3.2.1.0 test results

ENG15418 •ES&S will provide ballots with speckling. Use the DS200 Functional S1 election. Each ballot 

will contain stray marks in the time track (left hand side of the ballot). Marks will emulate various levels of 

white speckling. Ballot will be read on DS200 with v. 1.3.10.0 and re-read on version 1.4.3.0

ENH14726 & ENH15288 •After scanning a valid ballot, time how long the "Thank you for voting" screen 

displays. Note the time. Measure the font (text) on the "Thank you for voting" screen display. Note the 

size. Change firmware version and repeat test. Compare times and sizes.

Scenario 5: Reuse Unity 3.2.1.0 test results ENH17538 & 17666: •Only testing the Protective Counter 

(for Maryland). Do not need the ballot box. ES&S will provide an "updated" FW version that will have a 

different version number. To test, begin scanning ballots. Upgrade the firmware with the FW provided by 

ES&S. Continue with voting. When the voting is complete, Verify the counter did not reset or lose count 
Page 26 of 32

iBeta Definition DS200 Functional Test Method

and the counter appears in the printed reports. Restore the DS200 to the original firmware version. Verify 

the counter did not lose count.

•Note the counter number from the DS200 and the Initial State report

• Load the election and scan a few (2 to 5) ballots, close the polls and note the counter in the results 

report and audit log. 

•Upgrade the firmware, note the counter on the Initial State report. 

•Re-burn the media for the S1 election, load the election, and scan more ballots. 

•Close the polls. Examine the protected counter on the DS200, Audit log and the Results report. 

•Restore to the original firmware version. • Verify the protected counter did not lose count.

Scenario 6: ENH15009 (ES&S has withdrawn counterfeit ballot detection functionality)

Scenario 7: Use the Unity 3.2.1.0 test results

In Regression REV01 TC Scenario 1 the Overvote translations match ENH16291/16336. 

A description of the 

voting system type &

operational environment

The Unity 3.2.0.0 Rev 1 EMS is a single PC

• DS200 Precinct Count scanner 

• Steel ballot box with diverter

VVSG 2005 vol. 1 2.1.5.1.b I thru vii, c, 2.1.6, 2.1.7.2, 2.2.1, 2.2.1.3, 2.3.1 thru 2.3.1.2, 2.1.4.j, 2.2.4, 2.3.3.1.b,c,d, 2.3.3.2, 

2.4.1.b, 2.4.3, 2.4.2 thru 2.4.3, 4.1.5.1.d, 4.1.6.1, 4.1.8 thru 4.1.8

VVSG 2005 vol. 2 6.2, 6.2.1, 6.2.2, 6.3, 6.6, 6.7

A4.3.5 Volume (maximum and exceeding more than the maximum number of precincts in a Polling Place)

A4.3.5 Volume/Stress (Processing, storing and reporting data when overloading the number of precincts 

in a Polling Place)

A4.3.5 Performance/Recovery (Ballot format handling capability-graceful shut down and recovery without 

loss of data)

A4.3.5 Performance/Recovery (Processing rates-graceful shut down and recovery without loss of data)

Hardware, Software 

voting system 

configuration and test 

place 

The Unity 3.2.0.0 Rev 1 Voting System consists of the following: 

Audit Manger (AM), Election Data Manger (EDM), ES&S Ballot Image Manager (ESSIM), Hardware

Program Manger (HPM), Model Election Reporting Manager (ERM), LogMonitor Service.

1 @ DS200 1-DS200 starting with version 1.3.10.0 and upgrade to version 1.4.3.0 (upgrade required for f 

Scenario 5)

1 Steel ballot box with diverter

Pre-requisites and 

preparation for 

execution of the test 

case. 

Complete the prerequisites: 

• Retrieve the election data databases archived from the DS200 Functional Test Case in Unity 3.2.1.0, 

Scenario 2 Rev 02

• Copy the Unity contents from the archive for scenario 1 into the Standalone PC. 

• Copy the Unity and Elecdata contents from the archive for scenario 2 & 3 into the Standalone PC.

Getting Started Checks Getting Started: Complete the prerequisites;

Check the voting system to: 

• Verify the test environment and system configuration is documented in the PCA Configuration and 

matches the system used in the 48 hr. temp & power variation test and vendor described configuration.

Validate installation of the witnessed build. 

• Testers understand that no change shall occur to the test environment without documentation in the 

test record and the authorization of the project manager

• During testing an operational readiness test was performed. 

• The environment is set up as a standalone configuration. 

Documentation of Test 

Data & Test Results

Test Data: 

- Record all programmed & observed election, ballot & vote data fields and field contents on the 

corresponding tabs to provide a method to repeat the test. 

- Preserve all tabs for each instance the test is run.

Test Results: 

- Enter Accept/Reject on the Test Steps. 

- In Comments enter any deviations, discrepancies, or notable observations. 

- Log discrepancies on the Discrepancy Report and insert the discrepancy number in the Comments field 

of Test Step.

Volume: Paper-based 

voting systems

Processing

Ballot Prep:

Scenario 1) Create DS200 media with the previously defined Test Variables for Scenario 1

Scenario 2) Same as scenario 1 except early voting and ballot images are saved

Scenario 3) Use scenario 1 test results to confirm exceeding the declared system limit of 10 precincts do 

not cause system errors or iIf there are any system errors that cause the EMS ballot preparation 

applications to crash then verify the applications recover without any loss of data. If no error is given prior 

to leaving HPM continue the test through ERM; verify there are no system errors. .

Scenario 4, 5 & 7) Reuse of Unity 3.2.1.0 testing results
Page 27 of 32

iBeta Definition DS200 Functional Test Method

Scenario 6) ENH15009 (ES&S has withdrawn counterfeit ballot detection functionality).

Volume: Scenario 3: ENH16382•Use scenario 1 test results to confirm when exceeding the declared system limit 

of 10 precincts Overloading systems capacity to process, store, and report data. ): 

• Overloading the HPM with more than the declared number of precincts in a single polling place.

Stress Scenario 3: ENH16382•Use scenario 1 test results to confirm when exceeding the declared system limit 

of 10 precincts the system responses to overloading conditions

Performance Scenario 3: ENH16382•Use scenario 1 test results to confirm when exceeding the declared system limit 

of 10 precincts there is no system degradation (Ballot format handling capability and Processing rates): 

• The system does not slow down throughout the testing

Error Recovery Voting system gracefully shuts down (no crash) and recovers from errors caused by overloading the 

number of precincts and ballots styles. 

• Processing capabilities-graceful shut down and recover without loss of data

• Critical Status Messages

Readiness Testing and 

Poll Verification

Scenario 1: Voting system is ready for the election: 

Verify the voting system is ready for the election: 

- The election is correctly installed (Election ID, polling place name, precincts) . 

- Test data (run a precinct to validate the system is ready) is segregated from voting data, with no residual 

effect'. 

Test confirmation that there are: 

- No hardware/software failures. 

- The device is ready to be activated to accept votes (No Identification of any failures & corrective 

action)•Run 1 precinct to validate the system is ready; confirm the test data is segregated from voting 

data, with no residual effect. Verify totals and audit logs. 

ENH14729•"Polls Opened Menu": Print the zero reports (first of 3 zero reports prior to opening of polls) 

ENH16231•a "Failure" error and a "Successful" message are displayed on the DS200; the audit log 

displays the Failure and Successful PW login attempts.

ENH14730•2 beeps and a "successful" message displays once the scanner has been calibrated 

(Calibrate scanner option).

ENH14730 (15891, 15890, 15892)•Calibrate the counterfeit sensor; 2 beeps and a "successful" message 

displays (testing that only the counterfeit sensor can be set up as required, a message is displayed as 

well as 2 beeps can be heard once the counterfeit sensor has been successfully calibrated).

ENH14729•The zero report can be printed. (2nd of 3 zero reports prior to opening the polls) The report 

continually displays zeros for the contest, candidates, precincts (all 18), under/overvotes and Write-Ins.

ENH15287•The "Polls Opened Menu" doesn't display a "Ballot Style Report" button. (this displays only for 

Early Voting)

• Scan pre-election test ballots, 

ENH14745 • Select the options to override Overvoted ballots and Blank ballots only, do not select any 

other options. Overvoted and Blank ballots will not prompt an "Accept" or "Reject" message to the voter. 

The ballots will automatically be accepted due to the override. The Cross voted, Undervoted and 

unreadable ballots will provide a prompt to "Accept" or "Reject" and the voter will be required to make a 

selection 

•Tally pre-election test ballots; ballots match the predicted results outlined in the test case

ENH14732 & ENH16211•run the results tape; at the end of the Results tape and audit log they display 

the "Machine ID" and "Poll Number"

Scenario 2:

Reuse the same election as Scenario 1 except with the changes noted in the Scenario 2 Voting 

Variations

ENH15287• Verify the Ballot styles report button on the "Open Polls Menu" screen displays in the upper 

right hand of the icon bar and the report is accurate.

ENH14745 Reset "Override" options: Selection the options to override (undervotes and crossover ballots) 

Verify ballots will automatically be accepted due to the override and "Overvoted W/I ballots", "Unreadable 

marks", and "Blank Ballots" will provide a prompt to "Accept" or "Reject" and the voter will be required to 

make a selection 

ENH14730 • Unplug the DS200 from the ballot box verify 2 beeps are heard. Print a zero report and 

open printer door while printing is occurring. Verify printing will stop and DS200 will beep two times

Scenario 3:

•Verify scenario 1 is correctly handled by the system with 18 precincts (exceeding the declared system 

limit of 10 precincts).

Scenario4, 5 & 7 Reuse Unity 3.2.1.0 test results

Scenario 6: ENH15009 ES&S has withdrawn the detection of counterfeit ballots.

Pre- vote:

Opening the Polls 

Verification

Verify the polling place voting system for scenario 1, 2 & 3:

- Zero count report has no results. All test results have been zeroed out during readiness testing.

- Election identification including, Election Name/ID, Precinct ID/Name, Firmware Version
Page 28 of 32

iBeta Definition DS200 Functional Test Method

Voting: 

Ballot Activation and 

Casting Verifications

Scenario 1: Using the predetermined vote pattern, reuse previously marked ballots from Unity 3.2.1.0 

and scan ballots.

ENH16231 • attempt to reopen Polls using an incorrect and a correct password. Verify message appears 

on DS200 and in audit log.

ENH14729 •Clear pre-election readiness test totals and run another zero report

ENH14745 •Reset the "Override" options. Selecting the options to override Overvoted ballots and Blank 

ballots only, do not select any other options. Overvoted and Blank ballots will not prompt an "Accept" or 

"Reject" message to the voter. The ballots will automatically be accepted due to the override. The Cross 

voted, Undervoted and unreadable ballots will provide a prompt to "Accept" or "Reject" and the voter will 

be required to make a selection 

ENH14725 & ENH16085 •View the icon bar for each of the icons on the "Welcome screen". Verify an X 

appears on over the disk icon to indicate no images are being saved. Verify ballot status, power status, 

image status, election definition and open polls icons are displayed.

ENH16382 •vote ballots for all 18 precincts and verify the ballots for all 18 precincts are accepted without 

a precinct error

ENH14731 •Verify that there is a beep as each ballot is accepted

ENH15288 • Measure and verify the "Thank you for voting" text is larger. (Compare using old v.1.3.10.0 

and new v.1.4.3.0 DS200 firmware)

ENH14726 •Time and verify the "Thank you for voting" screen displays 3 to 5 seconds longer. (Compare 

using old v.1.3.10.0 and new v.1.4.3.0 DS200 firmware)

ENH14725 •The Welcome screen displays the Disk icon (on the icon bar) has a small red X (not saving 

ballot images)

ENH16085 • Verify the Welcome screen displays the following on the icon bar, "Battery Status Indicator 

Icon, AC Power Status Indicator Icon, Image Saving Status Indicator Icon, Election Definition Status Icon"

Scenario 2: 

ENH14725 •View the icon bar for each of the icons on the "Welcome screen". Verify no X appears on/

over the disk icon to indicate images are being saved. 

All early voting ballots can be scanned without error into the correct ballot bin

Scenario 3:

ENH16382•Use scenario 1 test results to confirm exceeding the declared system limit of 10 precincts

Scenario 4: Reuse Unity 3.2.1.0 test results

ENG15418 •ES&S will provide ballots with speckling. Each ballot will contain stray marks in the time track 

(left hand side of the ballot). Marks will emulate various levels of white speckling. Ballot will be read on 

DS200 with v. 1.3.10.0 and re-read on version 1.4.3.0. 

•Scan ballots

ENH14726: After scanning a valid ballot, time how long the "Thank you for voting" screen displays. Note 

the time.

ENH15288: After scanning a valid ballot, measure the font (text) on the "Thank you for voting" screen 

displays. Note the size.

Change the firmware version on the DS200 and scan ballots

ENH14726: After scanning a ballot, time how long the "Thank you for voting" screen displays. Note the 

time.

ENH15288: After scanning a ballot, measure the font (text) on the "Thank you for voting" screen displays. 

Note the size.

Voting: 

Voting System Integrity, 

System Audit, Errors & 

Status Indicators

The system audit provides a time stamped, always available, report of normal/abnormal events found 

within the percentage of sampled test 

Error messages are. 

- Generated, stored and reported as they occur. 

- Errors requiring intervention by the voter or poll worker are clearly display issues and action instructions 

in easily understood non-technical text language or with indicators. 

- The text for any numeric codes is contained in the error or affixed to the inside of the voting system. 

- Incorrect responses will not lead to irreversible errors. 

- Nested conditions are corrected in the sequence to restore the system to the state before the error 

occurred

Status Messages: 

- Display and report critical status messages using unambiguous indicators or English language text. 

- Non-critical status messages are displayed but do not have to be at the time of occurrence and may be 

numerical codes for subsequent interpretation and reported in unambiguous text. . 

- Status messages are part of the real-time audit record. 

Scenario 4: Reuse Unity 3.2.1.0 test results

ENH15418 •Using a ballot with an estimated 10% of the timing track scratched out. Scan each ballot of 

the older DS200 version (v. 1.3.10.0) and on the new version. The old version will display an error the 

new version will allow the scanning of the ballot. 
Page 29 of 32

iBeta Definition DS200 Functional Test Method

ENH15418 •Using a ballot with an estimated 50% of the timing track scratched out. Scan each ballot of 

the older DS200 version (v. 1.3.10.0) and on the new version. The old version will display an error the 

new version will allow the scanning of the ballot. 

ENH15418 •The white marks (speckles) in the ballot time track will display error "BALLOT 

DRAGGED/Turn Ballot Over and Try Again" on version 1.3.10.0. (Cause: The ballot did not enter the 

feed mechanism smoothly, which caused misalignment during scanning.) . 

End of Scenario 4 testing

Scenario 5: Reuse Unity 3.2.1.0 test results. 

ENH17538 & 17666:

• Scan a few (2 to 5) ballot, close the polls and note the counter in the results report and audit log. 

•Upgrade the firmware and note the counter on the Initial State report. 

•Re-burn the media for the S1 election, load the election, and scan more ballots. •Close the polls. 

•Examine the protected counter on the DS200, Audit log and the Results report.

•Restore to the original firmware version and verify the protected counter did not lose count. 

End of Scenario 5 testing

Scenario 6: ENH15009 ES&S has withdrawn the detection of counterfeit ballots.

Scenario 7 - Reuse results from Unity 3.2.1.0 to confirm ENH16291/16336 Overvote translations have 

been updated. End of Scenario 7 testing.

Document Review of ES&S Testing. 

BUG16775 & BUG16782 (Field Issue 1)

Review the ES&S assessment, resolution and testing regarding the DS200's sporadic reporting of a mark 

in row 44 and row 45 of column D (back of the ballot) when no actual mark was present. Review the 

assessment of the issue on the DS200 FW (v.1.3.10.0) to confirm that it identified: 

• A very narrow set of specific variables required to generate the error. 

• ES&S demonstrated they were able to consistently and reliably repeat the error; and. 

• ES&S' resolution was consistent with the VSS.

Review the resolution testing to confirm that the testing included all conditions and was sufficient to 

accept without additional testing by iBeta.

Variables Identified: Location of the contest from the vertical timing tracks, proximity to the top or bottom 

of the ballot, left side of the column oval placement, extension of the text next to the oval the edge of the 

allowable print area specified in the print layout manuals, and insertion in a skewed fashion of ballots with 

no selection on the last contest in column D.

Resolution: Tolerance adjustments such that ballots with these unique variables inserted in a skewed 

manner will be rejected and require reinsertion.

Post-vote:

Closing the Polls

Scenario 1: Once the polls are closed the voting system

• Printed reports of ballots counted by tabulator

• Reported votes match predicted votes from tabulator with votes and undervotes.

• In the polling place print the summary report with all of the 18 precincts in a single polling place. 

ENH16211 •Cancel printing of audit log only and view the log displays for the "Machine ID" and "Poll 

Number". 

Print audit log

• totals match the predicated reports (using the vote tab)

ENH16211 •Audit Log stops printing after cancellation and displays the "Machine ID" and "Poll Number" 

at the end of the Audit Log

•the Audit Log can be re-printed. The audit log will display the history of this election. The pretest and 

the election day audit log matches the pre-election activities outlined above.

ENH15827 •Overvoted Write-in ballots and blank ballots were separated from the other ballots

Scenario 2 

• Same as Scenario 1 excluding the listed enhancements 

Scenario 3: 

ENH16382•Use scenario 1 test results to confirm exceeding the declared system limit of 10 precincts

Scenario 6: ENH15009 (withdrawn counterfeit ballot detection), 

Post-vote:

Central Count

Scenario 1: ERM Zero report is printed and no totals display on the report prior to reading in the results.

ERM: Vote Consolidation:

ENH14725 • attempt to upload DS200 ballot images and a message displays stating no images saved. 

Ballot images from the DS200 CANNOT be extracted/ viewed. The image was not saved in HPM. (Ballot 

images not saved was set in Test Variables)

•Votes match predicted votes (compare to vote tab. Vote tab is what was used to create paper ballots) 

• reports will display election identification

• EL30A - Precinct Report–Group Detail, individual precincts & contest results. 

• Precinct Report contains votes, undervotes & overvotes

• EL45- Election Summary, total number of votes for each candidate/question 

• Verify DS200 SN is displayed in ERM, once the USB flash drive is read into ERM

• Ensure audit logs are accurate & complete. 
Page 30 of 32

iBeta Definition DS200 Functional Test Method

Scenario 2: Same as Scenario 1 except ENH14725 • Upload DS200 ballot images. Ballot images from 

the DS200 can be extracted. The image was saved in HPM. (Ballot images saved was set in Test 

Variables)

Scenario 3:

ENH16382•Use scenario 1 test results to confirm exceeding the declared system limit of 10 precincts

Expected Results are 

observed

Review the test result against the expected result:

• Accept: the expected result is observed

• Reject: the expected result of the test case is not observed

• Not Testable (NT): rejection of a previous test step prevents execution of this step, or tested in another 

TC. 

• Not Applicable (NA): not applicable to test scope

Record observations 

and all input/outputs for 

each election;

All inputs, outputs, observations, deviations and any other information impacting the integrity of the test 

results will be recorded in the test case. 

• Any failure against the requirements of the EAC guidelines will mean the failure of the system. And 

shall be reported as such

• - Failures will be reported to the vendor as Defect Issues in the Discrepancy Report. 

• The vendor shall have the opportunity to cure all discrepancies prior to issuance of the Certification 

Report. 

• If cures are submitted the applicable test will be rerun. Complete information about the rerun test will be 

preserved in the test case. The cure and results of the retest will be noted in the - Discrepancy Report

and submitted as an appendix of the Certification Report. 

• Operations which do not fail the requirements but could be deemed defects or inconsistent with 

standard software practices or election practices will be logged as Informational Issues on the 

Discrepancy Report. It is the vendor's option to address these issues. Open items will be identified in the 

report.
Page 31 of 32

7.1 DS200 Environmental Hardware Test Method

The test method description incorporates the assessment, test conditions and details as it was set up in the Unity 

3.2.1.0 Environmental Hardware Test Case and the engineering changes are listed in the Test Case Design

Method Detail Environmental Test Method

Test Case Name Environmental Test Case 

Scope - identifies the type 

of test

Assessment and testing of the hardware of the DS200 from ESSUNITY3200:

Identify and assess hardware changes from the certified baseline and engineering change orders to 

determine the extent of testing required, including execution and the provision of test results as required.

This set of hardware environmental test cases is outside the scope of iBeta's VSTL accreditation. Electrical 

testing was performed by Criterion Laboratories NVLAP #100396-0 (Electrical) with supervision of testing 

by iBeta. iBeta reviews and documents test records, results and reports to confirm testing was performed 

under an appropriate mode as a voting system and to determine acceptance or rejection of some or all 

testing

Test Objective Validation of the Unity 3.2.0.0 Rev 1 (identical to Unity 3.2.1.0) DS200 hardware to meet the NonOperating/Operating Environmental test standards of the EAC VSS 2002/VVSG 2005, including: 

DS200 Assessment of the ECOs from the ESSUNITY3200 baseline to verify reuse of the Non-Operating 

Transportation and Storage test results from ESSUNITY3200 and test execution of the Operating Electrical 

tests.

Test Variables: Use the test results from testing in Unity 3.2.1.0: 

DS200 - Electrical ECOs from the ESSUNITY3200 baseline (identical to Unity 3.2.1.0) impact on operation

Power disturbance disruption - IEC 61000-4-11 (1994-06). 

Electromagnetic radiation- FCC Part 15 Class B requirements - ANSI C63.4. 

Electrostatic disruption - IEC 61000-4-2 (1995-01). 

Electromagnetic susceptibility - IEC 61000-4-3 (1996). 

Electrical fast transient protection - IEC 61000-4-4 (1995-01).(2004-02)

Lightning surge protection - IEC 61000-4-5 (1995-02). 

RF immunity - IEC 61000-4-6 (1996-04). 

AC magnetic fields RF immunity - IEC 61000-4-8 (1993-06).

DS200, reuse the certified baseline in the Unity 3.2.0.0 Test Report- (ECOs have no impact)

MIL-STD810-D:

High temperature method 501.2 Procedures I-Storage maximum 140 F degrees 

Low temperature method 502.2, Procedure I-Storage minimum -4 F degrees 

Temperature & power variations method 501.2 & 502.2 

Humidity method 507.2 

Vibration method 514.3-1 Category 1 Basic Transportation Common Carrier 

Bench handling method 516.3 procedure VI 

Safety OSHA CFR Title 29, part 1910

A description of the voting 

system type and the 

operational environment

Precinct Count scanner/tabulator: Model 200 (DS200)

VSS 2002 vol. 1 3.2.2 thru 3.2.2.14, 3.4.8

VSS 2002 vol. 2 4.6.1.5 thru 4.7.1 & 4.8 RFI 2008-01, 2008-05, 2008-06, 2008-09, 2008-10

Hardware, Software voting 

system configuration and 

test location

DS200 Electrical Testing Test Location: Criterion Labs, Rollinsville CO

iBeta provided Criterion with the environmental hardware test case outlining methods for 

preparation of their test plan; iBeta documented the configuration, test environment, lab 

accreditations, tester qualifications, and operational status check performance

iBeta personnel execute the operational status checks and operate the equipment as a voting system 

during the EMI/EMC test execution. 

Pre-requisites and 

preparation for execution 

of the test case. 

DS200 Electrical Testing: Complete the prerequisites;

Validation and documentation of the subcontractor test labs' NVLAP accreditation in the specific 

test method identified in the Test Variables

Record the testers & date

System has been set up as identified in the user manual

Gather any necessary materials or manuals. 
Page 32 of 32

Method Detail Environmental Test Method

Ensure customization of the test case template is complete

The iBeta approved Operational Status Check script is provided that includes:

o Checking the operation of all buttons, switches and lights

o Opening the polls & running a zero totals report

o Checking appropriate error conditions for correct prompts or responses. (Error conditions 

will depend upon the type of equipment being tested.)

o Accessibility features are operational.

o Power off and on with no loss of function.

Close the polls and print all reports. (Totals & Audit Logs)

Getting Started Checks DS200 Electrical Testing: Check the voting system to: 

Verify the test environment and system configuration is documented in the PCA Configuration and 

matches the vendor described configuration. 

Validate installation of the Trusted Build

Testers understand that no change shall occur to the test environment without documentation in 

the test record and the authorization of the project manager

Confirm the tester understands the recording requirements of the iBeta test case

Operational status check procedures are available and successfully run

An automated script to loop system operation for use during the EMC operational tests exercises all 

necessary functionality.

Documentation of Test 

Data & Test Results

DS200 Electrical Testing: Test Results: 

• Enter Accept/Reject on the Test Steps

• In Comments enter any deviations, discrepancies, or notable observations

Log discrepancies on the Discrepancy Report and insert the number in the Comments

Standard Environmental 

Tests

DS200 Electrical Testing: Follow the test methods in all of the international electrical standards listed 

above to executed the EMC tests 

Expected Results are 

observed

DS200 Electrical Testing: Review the test result against the expected result: 

• Accept: the expected result is observed

• Reject: the expected result of the test case is not observed

• Not Testable (NT): not testable; provide a reason in the comments

• Not Applicable (NA): not applicable to test scope

Record observations and 

all input/outputs for each 

election;

DS200 Electrical Testing: All test results will be recorded in the test case

Any failure against the requirements will mean the failure of the system and shall be reported as such. 

Failures will be reported to the vendor as Defect Issues in the Discrepancy Repot. The vendor shall have 

the opportunity to cure all discrepancies prior to issuance of the Certification Report. If cures are submitted 

the applicable test will be rerun. Complete information about the rerun test will be preserved in the test 

case. The cure and results of the retest will be noted in the Discrepancy Report and submitted as an 

appendix of the Certification Report

Operations which do not fail the requirements but could be deemed defects or inconsistent with 

standard software practices or election practices will be logged as Informational Issues on the 

Discrepancy Report. It is the vendor's option to address these issues. Open items will be identified in 

the report