TP-CBG-2018-01.02 Rev. A

700 Boulevard South

Suite 102

Huntsville, AL 35802

Phone (256)713-1111

Fax (256)713-1112

Test Plan for EAC 2005 VVSG Certification Testing

Clear Ballot Group ClearVote 1.4.5 Voting System

EAC Project Number: CBG1450

Version: Rev. A

Date: 08/16/2018

TP-CBG-2018-01.02 Rev. A

REVISIONS

Revision Description Date

NR Initial Release 07/24/2018

Rev. A Updates per EAC comments, added Section 1.1.5 08/16/2018
TP-CBG-2018-01.02 Rev. A

TABLE OF CONTENTS

1.0 INTRODUCTION.......................................................................................................................... 1

1.1 Description and Overview of EAC System Being Modified .......................................................... 1

1.1.1 Baseline Certified System................................................................................................................ 3

1.1.2 Description of Modification ............................................................................................................ 8

1.1.3 Initial Assessment ............................................................................................................................ 9

1.1.4 Regression Test.............................................................................................................................. 10

1.1.5 System Modification Evaluation.................................................................................................... 10

1.2 References...................................................................................................................................... 10

1.3 Terms and Abbreviations............................................................................................................... 11

1.4 Project Schedule ............................................................................................................................ 11

1.5 Scope of Testing ............................................................................................................................ 11

1.5.1 Block Diagram............................................................................................................................... 12

1.5.2 System Limits................................................................................................................................ 14

1.5.3 Supported Languages..................................................................................................................... 15

1.5.4 Supported Functionality................................................................................................................. 15

1.5.5 VVSG............................................................................................................................................. 16

1.5.6 RFIs................................................................................................................................................ 16

1.5.7 NOCs ............................................................................................................................................. 16

2.0 PRE-CERTIFICATION TESTING AND ISSUES .................................................................. 16

2.1 Evaluation of Prior VSTL Testing................................................................................................. 16

2.2 Evaluation of Prior Non-VSTL Testing......................................................................................... 16

2.3 Known Field Issues........................................................................................................................ 16

3.0 MATERIALS REQUIRED FOR TESTING............................................................................. 17

3.1 Software ........................................................................................................................................ 17

3.2 Equipment ..................................................................................................................................... 20

3.3 Test Materials ................................................................................................................................ 22

3.4 Deliverable Materials .................................................................................................................... 22

3.5 Proprietary Data ............................................................................................................................ 22
TP-CBG-2018-01.02 Rev. A

4.0 TEST SPECIFICATIONS .......................................................................................................... 23

4.1 Requirements (Strategy of Evaluation).......................................................................................... 23

4.1.1 Rational for ‘Not Applicable’ requirements .................................................................................. 24

4.2 Hardware Configuration and Design ............................................................................................. 24

4.3 Software System Functions............................................................................................................ 24

4.4 Test Case Design ........................................................................................................................... 25

4.4.1 Hardware Qualitative Examination Design ................................................................................... 26

4.4.2 Hardware Environmental Test Case Design .................................................................................. 26

4.4.3 Software Module Test Case Design and Data ............................................................................... 26

4.4.4 Software Functional Test Case Design and Data........................................................................... 26

4.4.5 System-Level Test Case Design .................................................................................................... 27

4.5 Security Functions ......................................................................................................................... 27

4.6 TDP Evaluation.............................................................................................................................. 27

4.7 Source Code Review...................................................................................................................... 30

4.8 QA & CM System Review ............................................................................................................ 30

4.9 PCA................................................................................................................................................ 30

4.10 FCA................................................................................................................................................ 31

4.11 Accuracy ........................................................................................................................................ 31

4.12 System Integration ......................................................................................................................... 31

5.0 TEST DATA................................................................................................................................. 31

5.1 Test Data Recording ...................................................................................................................... 31

5.2 Test Data Criteria........................................................................................................................... 32

5.3 Test Data Reduction....................................................................................................................... 32

6.0 TEST PROCEDURES AND CONDITIONS ............................................................................ 32

6.1 Facility Requirements.................................................................................................................... 32

6.2 Test Set-Up .................................................................................................................................... 32

6.3 Test Sequence ................................................................................................................................ 33

6.4 Test Operations Procedure............................................................................................................. 33

PROJECT SCHEDULE............................................................................................................A-1
TP-CBG-2018-01.02 Rev. A 1

1.0 INTRODUCTION

The purpose of this Test Plan is to document the procedures that Pro V&V, Inc. will follow to 

perform certification testing during a system modification campaign for the Clear Ballot Group 

(CBG) ClearVote 1.4.5 Voting System to the requirements set forth for voting systems in the U.S. 

Election Assistance Commission (EAC) 2005 Voluntary Voting System Guidelines (VVSG), 

Version 1.0. Prior to submitting the voting system for testing, CBG submitted an application 

package to the EAC for certification of the ClearVote 1.4.5 Voting System. The application was 

accepted by the EAC and the project was assigned the unique Project Number of CBG1450. 

At test conclusion, the results of all testing performed as part of this test campaign will be 

submitted to the EAC in the form of a national certification test report.

1.1 Description and Overview of EAC System Being Modified

The EAC Certified System that is the baseline for the submitted modification is described in the 

following subsections. All information presented was derived from the previous Certification Test 

Report, the EAC Certificate of Conformance and/or the System Overview.

The following subsections describe the baselined ClearVote 1.4 System. The following sections 

contain a product description and an overview of the design methodology of the ClearVote 1.4

Voting System, as taken from the Clear Ballot Group technical documentation.

The ClearVote 1.4 Voting System is a paper-based optical scan voting system consisting of the 

following major components: ClearDesign (ballot design and EMS), ClearCount (central count, 

tabulation, and reporting), ClearCast (precinct count and tabulation), and ClearAccess (accessible 

voting and ballot marking device). 

ClearDesign

ClearDesign is an Election Management System consisting of an interactive set of applications 

which are responsible for all pre-voting activities necessary for defining and managing elections. 

This includes ballot design, ballot proofing, ballot layout, and ballot production. The ClearDesign 

system consists of the physical components listed below. All of the components and generation of 

voting machine election definition file packages are unmodified COTS that are connected via a 

wired, closed, and isolated network not connected to any other systems or the Internet.

 DesignServer: A laptop or desktop computer running Ubuntu with the ClearDesign 

software and hosting the election database.

 DesignStation(s): One or more laptop or desktop running Windows used to connect to 

the DesignServer. A browser is used to perform the necessary tasks. A user with 

administration privileges will be able to define users and manage the elections. 

 Router: Used to connect the DesignStations to the DesignServer using a wired, closed 

Ethernet-based network.
TP-CBG-2018-01.02 Rev. A 2

ClearCount

ClearCount is a central, high-speed, optical scan ballot tabulator coupled with ballot processing 

applications. The ClearCount software runs on unmodified COTS laptop or desktop computers 

running the Windows operating system and supports specific models of Fujitsu scanners. The 

ClearCount central-count system running an Ubuntu Linux operating system, with Ethernet 

connections to workstations running the Windows operating system consists of the physical 

components listed below. All of the components are unmodified COTS that are connected via a 

wired, closed, and isolated network not connected to any other systems or the Internet.

 ScanServer: A laptop or desktop computer running the ClearCount software and hosting 

its election database and the web server that serves its election reports.

 ScanStation(s): One or more laptop or desktop/scanner pairs used to scan and tabulate 

ballots.

 Router: Used to connect the ScanStations to the ScanServer using a wired, closed 

Ethernet.

 Election Administration Station and/or Adjudication Station: One or more Windows 

laptop or desktop computers installed with browser software, linked by a wired Ethernet 

connection to the ScanServer using the router. This station can serve multiple uses: user 

administration, election administration, adjudication, and reporting. This station is also 

used to consolidate the vote totals and ballot images from the ClearCast precinct 

tabulator. The vote totals and ballot images are consolidated by the ClearCount Software 

via the ClearCast USB drive. 

All files that make up the ClearCount software reside on a single ScanServer that is shared by all 

client ScanStations. The Tabulator software is executed by the ScanStations at run-time from files 

that reside on the ScanServer. The only software programs that have to be installed on 

ScanStations, apart from the Windows operating system, are the Fujitsu ScandAll Pro software 

and drivers required by the scanner hardware.

The ClearCount software consists of the following components:

 Tabulator: The Tabulator application handles ballot tabulation. The Tabulator software is 

stored on the ScanServer and an instance of Tabulator runs on each ScanStation. The 

Tabulator program analyzes the incoming image and transfers them to the local output 

folder named CBGBallotImages. The ScanServer retrieves the images from the folder 

and uploads them into the Election database.

 Election Database: A centralized election database that resides on the ScanServer and 

collects the output of each Tabulator.

 Election Reports: A browser-based suite of reports that provides election results and 

analysis and allows election officials to review individual ballot images. A web server on 

the ScanServer serves the reports.

 Card Resolutions tool: A web application that allows election officials to review and 

appropriately resolve unreadable voted ballots. 

 User and Election Database Management through web applications: On the User 

Administration dashboard, the administrator can add, rename, or delete users, assign 

permissions, and change user passwords. On the Election Administration dashboard, the 

administrator can create or delete an election, set an election as active, and backup or 

restore an election.
TP-CBG-2018-01.02 Rev. A 3

ClearCast

The ClearCast tabulator is a precinct count ballot scanning solution suitable for early and election 

in-person voting, including processing ballots printed by the ClearAccess accessible ballot 

marking device. The ClearCast application runs on the precinct count-based tabulator, and is used 

to scan, count and tally marked ballots. Its functionality is divided into three essential modes, 

Election Mode (Early Voting and/or Election Day), which is used to process voter cast ballots, 

Pre-Election Mode, this occurs prior to Election Mode, and is used to test all system functionality 

subsequent to the start of the election, and Post-Election Mode, which is used to perform 

administrative functions following the close of the election.

ClearAccess

ClearAccess is an accessible touchscreen ballot marking device (BMD) used for the creation of 

paper ballots that can be scanned and tabulated by ClearCast or ClearCount. The ClearAccess 

components of the ClearVote 1.4 voting system uses both unmodified and modified off-the-self 

hardware. Laptop and desktop computers are combined with personal assistive devices, printers, 

and uninterruptible power supplies to form a ballot-marking device.

1.1.1 Baseline Certified System

The baseline system for this modification is the ClearVote 1.4 Voting System. The tables below 

describe the certified equipment and firmware versions. 

Detailed descriptions of the ClearVote 1.4 test campaign are contained in Pro V&V Report No. 

TR-01-01-CBG-001-01.01, Rev. C, which is available for viewing on the EAC’s website at 

www.eac.gov.

The individual components listed below are compiled to create the ClearVote 1.4 voting system

(ClearCast 1.4, ClearCount 1.4, ClearDesign 1.4, and ClearAccess 1.4). 

Table 1-1. Voting System Software

Firmware/Software Version

ClearDesign Components, Version 1.4.3

Windows 10 Pro 1607

Google Chrome 55.0.2883.87

Ubuntu 14.04.4 LTS

MySQL 5.5.55

Apache 2.4.7

libapache2-mod-fcgid 2.3.9

PhantomJS 1.9.0

Usbmount 0.0.22

Unzip 6.0.9
TP-CBG-2018-01.02 Rev. A 4

Table 1-1. Voting System Software (continued)

Firmware/Software Version

Samba 4.3.11

Python PIP 1.5.4

Zip 3.0.8

Pyinstaller 3.0

Python JSMIN 2.2.1

Python 2.7.6

Python webpy 0.38

Python MySQL DB 1.2.3

SQLAlchemy 1.0.15

Python Pillow 2.3.0

Python Flup 1.0.2

Python DBUtils 1.1

Python XLRD 0.9.4

Python FontTools library 3.0

Python RTF 0.2.1

OpenSSL FIPS Object Module 2.0.10

OpenSSL (standard) 1.0.2g

DataTable 1.10.5

DataTable-TableTools 2.2.3

DataTable-ColVis 1.1.1

DataTable-ColReorder 1.1.2

DataTablePlugins 1.10.10

bootstrap 3.0.0

jquery 1.10.2

jquery-impromptu 5.2.3

jquery-qrcode 1.0

jquery-splitter 0.14.0

jquery-ui 1.10.4

jscolor 1.4.2

tinymce 4.1.9

fastclick 1.0.4

libmp3lame 0.5.0

jszip 3.1.2

papaparse 4.1.2

jsmin 12/4/2003

ClearAccess Components, Version 1.4.1

Windows 10 Pro 1607
TP-CBG-2018-01.02 Rev. A 5

Table 1-1. Voting System Software (continued)

Firmware/Software Version

Google Chrome 61.0.3163.100

nsis 3.01

PyInstaller 3.2

Python 2.7.10

webpy 0.38

Python-future 0.15.2

pefile 2016.3.28

pywin 220

jquery 1.10.5

DataTables 1.10.5

ColVis 1.1.1

ColReorder 1.1.2

jsmin 2003-12-04

Brother printer driver 1.0.1.0

Okidata printer driver 1.0.0.0

ClearCast Components, Version 1.4.2

scanner_control 0.0.28

UPSBatteryMontior 1.0

Ubuntu 14.04.5 LTS

google_chrome 62.0.3202.75-1

zeromq 4.2.0

arduino tools 1.8.0

adafruit tools 1.4.9

pyinstaller 3.2.1

OpenSSL FIPS Object Module 2.0.10

OpenSSL (standard) 1.0.1f

libPDIScan.so 7.1.0

pdi_ps3_drv_scanner.ko 2.0.5

DataTables 1.10.5

JTSage DateBox 4.0.0

jQuery.NumPad 1.4

jQuery 1.10.2

jquery.ui 1.11.3

ClearCount Components, Version 1.4.2

Windows 10 Pro 1607

Google Chrome 55.0.2883.87

Ubuntu 16.04.1 LTS
TP-CBG-2018-01.02 Rev. A 6

Table 1-1. Voting System Software (continued)

Firmware/Software Version

Apache 2.4.18

libapache2-mod-fcgid 2.3.9

Python(part of Ubuntu) 2.7.12

Pillow (part of Ubuntu) 3.1.2

MySQLdb (part of Ubuntu) 1.3.7

PyInstaller 3.2.1

PollyReports 1.7.6

OpenSSL FIPS Object Module 2.0.10

OpenSSL (standard) 1.0.2g

JavaScript Bootstrap library 2.3.2

JavaScript Chosen library 1.0.0

JavaScript jQuery library 1.10.2

J JavaScript jQuery-migrate library 1.2.1

JavaScript DataTables library 1.9.4

ColVis 1.0.8

JavaScript TableTools library 2.1.5

ZeroClipboard 1.0.4-TableTools2

JavaScript FixedHeader library 2.0.6

JavaScript hotkeys library 1.0

JavaScript tooltip library 1.3

JavaScript pep library 1.0

JavaScript LESS library 1.3.3

Fujitsu fi-6400 PaperStream 1.30.0

Fujitsu fi-6800 10.10.710

Fujitsu fi-7180 PaperStream 1.4.0

Table 1-2. Voting System Equipment

Component Model Serial Number

ClearDesign Components

Dell Latitude Laptop 5580 7L6M3G2

Dell PowerEdge Server T630 2K5YFK2, JLPYHK2, & JLPXWK2

Dell 24 inch Monitor SE2416H FVWV5G2

Dell 22 inch Monitor E2216HV 36765D2 & 90665D2

Dell Mini Tower T3620 IHCLXK2 & IHCKXK2

TP-LINK VPN Router TL-R600VPN 2149342000209, 2166306000413, & 

2168351001114
TP-CBG-2018-01.02 Rev. A 7

Table 1-2. Voting System Equipment (continued)

Component Model Serial Number

Lenovo USB Portable DVD 

Burner LN-8A6NH11B 8SSDX0H33226L1CB7107099

Brother Printer HL-L2340DW U63879A7N416353

ClearAccess Components

Dell OptiPlex AIO 5250 6PW4GK2, BPYXCH2, HGCMGK2, 

& 6PWZFK2

Dell 15” Inspiron 7000 series 80S1YD2, 7TT1YD2, & 22S1YD2

Brother Laser Printer HL-L2340DW 

U63879M4N62861,

U63879M4N628617,

U63879A7N416353, & 

U63879M4N628535

Oki Data Laser Printer B432dn

AK5B007647A0, AK76030925A0, 

AK76030928A0, AK62030437A0, 

AK62030440A0, & AK76030928A0

Storm EZ Access Keypad EZ08-222013 15000005, 15000007, & 15020478

Origin Instruments Sip/Puff 

Breeze with Headset AC-0313-H2 CBG-SP-001, CBG-SP-002, & CBG-SP-003

Hamilton Buhl Over-Ear 

Stereo Headphones HA7 CBG-HP-001 & CBG-HP-002

ElectionSource Table Top 

Voting Booth (Privacy 

Screen)

VB-60B CBG-VB-001

APC Smart-UPS SMT2200 

AS1602232215, AS1721142050, 

AS1638230963, AS1721132721, & 

AS1625141816

Ergotron Stand for Dell 

OptiPlex 5250 AIO (portrait 

mode)

Neo Flex 1274839-0061 & 1358124-0005

Wurth Ferrite (for Oki 

printer) 742-416-33S 742-416-33S-CBG1

Wurth Ferrite 

(for Brother printer) 742-416-22S 742-416-22S-CBG1

3M EMI Copper Foil 

Shielding Tape, , 1⁄4 inch 1181 1181-CBG1

Lexan or acrylic plastic cover 

(8 mm) 2”x4” Cover-CBG1

ClearCount Components

Dell Latitude Laptop 

(multiple units) 5580 

2F3L3G2, C9S22G2, CF3L3G2, 

90356H2, BDH46H2, 8TM46H2, 

4PM46H2, 4QM46H2, 3CH46H2, & 

FPM46H2

Dell PowerEdge Server T330 5RRFGK2, 5712JK2, & FHV9RD2

Dell OptiPlex AIO 7440 JXDFHH2, JXDFDH2, & 64WPXG2

Dell Precision Workstation T3620 GW6XHH2 & H0PZFK2
TP-CBG-2018-01.02 Rev. A 8

Table 1-2. Voting System Equipment (continued)

Component Model Serial Number

Fujitsu Scanner fi-7180 A20DC10302, A20DC10378, 

A20D000798, & A20DC08933

Fujitsu Scanner fi-6800 A9HCC00543, A9HCC00535, & 

100295

Fujitsu Scanner fi-6400 AKHCC00609, AKHCC00337, & 

AKHCC00362

Lenovo USB Portable DVD 

Burner LN-8A6NH11B 8SSDX0H33226L1CB7107099

Dell 22 inch Monitor E2216HV GD965D2

Dell 22 inch Monitor P2217 7818672

Dell 22 inch Monitor S2240M CN-0CFGKT-64180-58B-0X3T

Dell 27 inch Monitor P2717H CDMS672 & HPWD072

Cisco Catalyst Switch 

(1 Gigabit Router or Switch) 2960-X Series FCW2039B6QF & FCW2110A1E0

TP-LINK Easy Smart Switch

(1 Gigabit Router or Switch) TL-SG108E 216C319009010 & 216C319009012

NetGear ProSafe VPN 

Firewall 

(1 Gigabit Router or Switch)

FVS318G 40F266BA00280

APC Smart-UPS SMT1500 3S1525X07491, 3S1525X07421, & 

4B1448P39979

Western Digital External 

Hard Drive WDBBGB0040HBK WCC7K5CHA3DK

EZ Scanning Shelf (fi-6400 

or fi-6800) Model: WorkEZ CBG-EZ-001, CBG-EZ-002, CBG-EZ003, & CBG-EZ-004

ClearCast Components

ClearCast Model: 1 Version A Cast0011, Cast0014, Cast0015,

Cast0017, Cast0018, and Cast0020

Ballot Box 1224UBB-CB CBG-BB-001, CBG-BB-002

1.1.2 Description of Modification

The ClearVote 1.4.5 is a modified voting system configuration that includes upgrades to the 

ClearCount and ClearDesign components of the ClearVote 1.4 system. The modifications include 

the specific changes between the ClearVote 1.4.5 system and the previously certified baseline 

ClearVote 1.4 system, as taken from the ClearVote Change Notes. Detailed information on how 

the modifications will be addressed is included in Section 1.1.5.

The submitted modifications include the following:

ClearCount

 Resolution of “service unavailable” error while running the Tabulator and DeleteBox

 Updates to the use of “Ballot” vs. “card” in the user interface
TP-CBG-2018-01.02 Rev. A 9

 Additional error messages implemented when importing the BDF

 Resolution of “FATAL: SweeperProcess” error that occurred when running DeleteBox 

on one ScanStation and the Tabulator on another ScanStation

 Fixed issue concerning incorrect comparison results files (CRF) counts

 Updated misleading unreadable message in a “By Style” election

 Corrected issue concerning incorrect redaction of Statement of Votes Cast PDF report

 Corrected issue concerning Districts filter on Statement of Votes Cast PDF report

 Corrected vote and ballot totals when filtering by multiple districts in “by Precinct” PDF 

reports

 Resolved issue in which the “Grand Total” section is no longer redacted in Statement of 

Votes Cast PDF report

 Implemented addition of ‘type’ attribute to the <Contest> element in the XML results file

 Implemented optional XML report conversion utility

 Use of encrypted USB drives and encrypted hard drives

ClearDesign

 Implemented support for Exclusive One-Touch Straight Party Type functionality

 Use of encrypted USB drives

1.1.3 Initial Assessment

An initial assessment on the submitted modifications was performed to determine the scope of 

testing. Testing from the previous test campaign was used to establish the baseline. Based on the 

assessment, it was determined the following tasks would be required to verify compliance of the 

modifications:

 Source Code Review, Compliance Build, Trusted Build, and Build Document Review

 System Level Testing 

― System Integration

― Accuracy (ClearCount component only)

 Technical Documentation Package (TDP) Review

 Functional Configuration Audit (FCA)

1.1.4 Regression Test

Regression testing for this test campaign will consist of the execution of the System Integration 

and Accuracy Tests.
TP-CBG-2018-01.02 Rev. A 10

1.1.5 System Modification Evaluation

The submitted modifications to the ClearVote 1.4.5 System consist primarily of minor software 

changes to accommodate defect resolutions and enhancements to the ClearCount and 

ClearDesign components. No hardware changes are submitted for this evaluation.

To verify the modifications are successfully addressed throughout the test campaign, each 

modification will be tracked and verified to be addressed during the execution of the relevant test 

area. For example, source code changes will be verified during the source code review. 

Modifications requiring functional test verification will be evaluated by executing the standard 

Accuracy Test, the System Integration Test, or during performance of the FCA. Modifications 

that are not adequately evaluated during the performance of these tests will be subjected to 

specifically designed test cases.

 

1.2 References

 Election Assistance Commission 2005 Voluntary Voting System Guidelines (VVSG) Version

1.0, Volume I, “Voting System Performance Guidelines”, and Volume II, “National 

Certification Testing Guidelines”

 Election Assistance Commission Testing and Certification Program Manual, Version 2.0

 Election Assistance Commission Voting System Test Laboratory Program Manual, Version 

2.0

 National Voluntary Laboratory Accreditation Program NIST Handbook 150, 2016 Edition, 

“NVLAP Procedures and General Requirements (NIST Handbook 150-2016)”, dated July 

2016

 National Voluntary Laboratory Accreditation Program NIST Handbook 150-22, 2008 

Edition, “Voting System Testing (NIST Handbook 150-22)”, dated May 2008

 United States 107th Congress Help America Vote Act (HAVA) of 2002 (Public Law 107-

252), dated October 2002

 Pro V&V, Inc. Quality Assurance Manual, Revision 7.0

 Election Assistance Commission “Approval of Voting System Testing Application Package” 

letter dated July 12, 2018

 EAC Requests for Interpretation (RFI) (listed on www.eac.gov)

 EAC Notices of Clarification (NOC) (listed on www.eac.gov)

 Pro V&V Test Report No. TR-01-01-CBG-001-01.01 Rev. C, “Test Report for EAC 2005 

VVSG Certification Testing Clear Ballot Group ClearVote 1.4 Voting System”, dated 

01/29/18

 Clear Ballot Group’s Technical Data Package (A listing of the ClearVote 1.4.5 documents 

submitted for this test campaign is listed in Section 4.6 of this Test Plan)
TP-CBG-2018-01.02 Rev. A 11

1.3 Terms and Abbreviations

This subsection lists terms and abbreviations relevant to the hardware, the software, or this Test 

Plan.

“ADA” – Americans with Disabilities Act 1990

“BMD” – Ballot Marking Device

“CBG” – Clear Ballot Group

“CM” – Configuration Management

“COTS” – Commercial Off-The-Shelf

“DRE” – Direct Record Electronic

“EAC” – United States Election Assistance Commission

“EMS” – Election Management System

“FCA” – Functional Configuration Audit

“HAVA” – Help America Vote Act

“ISO” – International Organization for Standardization

“NOC” – Notice of Clarification

“PC” – Personal Computer

“PCA” – Physical Configuration Audit

“QA” – Quality Assurance

“RAM” – Random Access Memory

“RFI” – Request for Interpretation

“TDP” – Technical Data Package

“UPS” – Uninterruptible Power Supply

“VSTL” – Voting System Test Laboratory

“VVSG” – Voluntary Voting System Guidelines

1.4 Project Schedule

The Project Schedule for the test campaign is located in Appendix A. The dates on the schedule 

are not firm dates but planned estimates based on the anticipated project work flow.

1.5 Scope of Testing

The scope of testing focused on evaluating the modifications detailed in Section 1.1.2 of this Test 

Plan. Primarily, these modifications focused on upgrades to the components of the previously 

certified ClearVote 1.4. To determine the ClearVote 1.4.5 test requirements, the submitted 

modifications were evaluated against each section of the EAC VVSG 1.0 to determine the 

applicable tests to be performed. Based on this assessment, it was determined that multiple areas 

within the EAC VVSG 1.0 would be evaluated to encompass the required tests. 

A breakdown of the areas and associated tests is listed below:
TP-CBG-2018-01.02 Rev. A 12

 EAC VVSG 1.0 Volume 1, Section 2: Functional Requirements

― System Integration Testing 

― Functional Configuration Audit (FCA)

― Physical Configuration Audit (PCA), including System Loads & Hardening

― Technical Documentation Package (TDP) Review

― Accuracy Testing (ClearCount component only)

 EAC VVSG 1.0 Volume 1, Section 5: Software Requirements

― Source Code Review, Compliance Build, Trusted Build, and Build Document 

Review

― Technical Documentation Package (TDP) Review

― Functional Configuration Audit (FCA)
TP-CBG-2018-01.02 Rev. A 13

1.5.1 Block Diagram

The system overview of the submitted voting system is depicted in Figure 1-1.

Figure 1-1. ClearVote 1.4.5 System Overview 
TP-CBG-2018-01.02 Rev. A 14

1.5.2 System Limits

The system limits that CBG has stated to be supported by the ClearVote 1.4.5 Voting System are 

listed in the tables below.

Table 1-3. System Limits for ClearDesign

Characteristic Limit

Precincts in an election 3200

Contests in an election 3200

Candidates/Counters in an election 3200

Ballot Styles in an election 3200

Contests in a ballot style 60

Candidates in a contest 300

Ballot styles in a precinct 50

Number of political parties 50

“vote for” in a contest 50

Supported languages in an election 15

Number of write-ins 50

The maximum ballot positions for the ClearVote 1.4.5 Voting System are as follows: 

Table 1-4. Maximum Oval Positions for ClearDesign

Ballot Size Oval positions per side

5 inch 60

11 inch 180

14 inch 240

17 inch 300

19 inch 360

22 inch 420

 

Table 1-5. System Limits for ClearCount 

Scanner 

Model

Sustained (not burst speed) ballots per hour

8.5x5 8.5x11 8.5x14 8.5x17 8.5x19 8.5x22

Typical county 

size

(central count)

fi-6400 5592 3624 2928 2448 2350 2236 Large 

(>100k voters)

fi-6800 7822 5508 4155 3352 3000 2800 Large 

(>100k voters)

fi-7180 3396 2040 1692 1400 1300 1200 Small

(<25k voters)

ClearCount can have a maximum of 10 ScanStation/Scanner pairs 
TP-CBG-2018-01.02 Rev. A 15

1.5.3 Supported Languages

The submitted voting system supports: 

 English

 Spanish

 Chinese

 Korean

 Vietnamese

 Danish

 Dutch

 Flemish

 French

 German

 Italian

 Japanese

 Norwegian

 Portuguese

 Swedish

Due to the limited scope of testing, only English and Spanish language ballots will be cast during 

the performance of functional testing. Additionally, one character based language (Chinese) will 

be tested during System Integration Testing.

1.5.4 Supported Functionality

The ClearVote 1.4.5 is designed to support the following voting variations:

 General Election

 Closed Primary

 Open Primary

 Early Voting

 Partisan/Non-Partisan Offices

 Write-In Voting

 Primary Presidential Delegation Nominations

 Straight Party Voting

 Split Precincts

 Vote for N of M

 Ballot Rotation

 Provisional or Challenged Ballots
TP-CBG-2018-01.02 Rev. A 16

1.5.5 VVSG

The ClearVote 1.4.5 shall be evaluated against the relevant requirements contained in the EAC 

VVSG 1.0.

1.5.6 RFIs

There are no RFIs released by the EAC as of the date of this Test Plan that pertain to this test 

campaign that were not in effect at the time of the baseline system certification.

1.5.7 NOCs

There are no NOCs released by the EAC as of the date of this Test Plan that pertain to this test 

campaign that were not in effect at the time of the baseline system certification.

2.0 PRE-CERTIFICATION TESTING AND ISSUES

This section describes previous testing performed prior to submitting the voting system to the 

EAC.

The following pre-certification testing has been performed on an “At-Risk” basis:

Technical Date Package (TDP) Review

An initial TDP review was performed on the documents submitted for this test campaign. The 

initial review is an abbreviated review performed in order to determine if the documents contain 

sufficient information.

Physical Configuration Audit (PCA)

An initial PCA has been performed to baseline the system prior to test campaign commencement. 

The PCA was performed by documenting each hardware and software component of the voting 

system by name, model, serial number, major component, and any other relevant information 

needed for identification. Photographs of each hardware component were also taken.

2.1 Evaluation of Prior VSTL Testing

Pro V&V evaluated to the published Final Test Report for the ClearVote 1.4.5 in order to baseline 

the current system under test.

2.2 Evaluation of Prior Non-VSTL Testing

No prior non-VSTL testing of the ClearVote 1.4.5 modifications were considered for this test 

campaign

2.3 Known Field Issues

The ClearVote 1.4.5 is a modification to a previously certified system and has not been fielded 

for use. 
TP-CBG-2018-01.02 Rev. A 17

3.0 MATERIALS REQUIRED FOR TESTING

The following sections list all materials needed to enable the test engagement to occur.

The materials required for testing of the ClearVote 1.4.5 Voting System include all materials to 

enable the test campaign to occur. This includes the applicable hardware and software as well as 

the TDP, test support materials, and deliverable materials, as described in the following 

subsections.

3.1 Software

This subsection lists the proprietary and COTS software to be provided by the manufacturer as 

part of the test campaign. The individual components are compiled to create the ClearVote 1.4.5

Voting System (ClearCast 1.4.2, ClearCount 1.4.5, ClearDesign 1.4.4, and ClearAccess 1.4.1). 

Table 3-1. ClearVote 1.4.5 Voting System Software

Firmware/Software Version

ClearDesign Components, Version 1.4.4

Windows 10 Pro 1607

Google Chrome 55.0.2883.87

Ubuntu 14.04.4 LTS

MySQL 5.5.55

Apache 2.4.7

libapache2-mod-fcgid 2.3.9

PhantomJS 1.9.0

Usbmount 0.0.22

Unzip 6.0.9

Samba 4.3.11

Python PIP 1.5.4

Zip 3.0.8

Pyinstaller 3.0

Python JSMIN 2.2.1

Python 2.7.6

Python webpy 0.38

Python MySQL DB 1.2.3

SQLAlchemy 1.0.15

Python Pillow 2.3.0

Python Flup 1.0.2

Python DBUtils 1.1

Python XLRD 0.9.4

Python FontTools library 3.0

Python RTF 0.2.1
TP-CBG-2018-01.02 Rev. A 18

Table 3-1. ClearVote 1.4.5 Voting System Software (continued)

Firmware/Software Version

OpenSSL (FIPS) 2.0.1.0

OpenSSL 1.0.2g

DataTable 1.10.5

DataTable-TableTools 2.2.3

DataTable-ColVis 1.1.1

DataTable-ColReorder 1.1.2

DataTablePlugins 1.10.10

bootstrap 3.0.0

jquery 1.10.2

jquery-impromptu 5.2.3

jquery-qrcode 1.0

jquery-splitter 0.14.0

jquery-ui 1.10.4

jscolor 1.4.2

tinymce 4.1.9

fastclick 1.0.4

libmp3lame 0.5.0

jszip 3.1.2

papaparse 4.1.2

jsmin 12/4/2003

ClearAccess Components, Version 1.4.1

Windows 10 Pro 1607

Google Chrome 61.0.3163.100

nsis 3.01

PyInstaller 3.2

Python 2.7.10

webpy 0.38

Python-future 0.15.2

pefile 2016.3.28

pywin 220

jquery 1.10.5

DataTables 1.10.5

ColVis 1.1.1

ColReorder 1.1.2

jsmin 2003-12-04

Brother printer driver 1.0.1.0

Okidata printer driver 1.0.0.0
TP-CBG-2018-01.02 Rev. A 19

Table 3-1. ClearVote 1.4.5 Voting System Software (continued)

Firmware/Software Version

ClearCast Components, Version 1.4.2

scanner_control 0.0.28

UPSBatteryMontior 1.0

Ubuntu 14.04.5 LTS

google_chrome 62.0.3202.75-1

zeromq 4.2.0

arduino tools 1.8.0

adafruit tools 1.4.9

pyinstaller 3.2.1

openssl-fips 2.0.10

openssl 1.0.1f

libPDIScan.so 7.1.0

pdi_ps3_drv_scanner.ko 2.0.5

DataTables 1.10.5

JTSage DateBox 4.0.0

jQuery.NumPad 1.4

jQuery 1.10.2

jquery.ui 1.11.3

ClearCount Components, Version 1.4.5

Windows 10 Pro 1607

Google Chrome 55.0.2883.87

Ubuntu 16.04.1 LTS

Apache 2.4.18

libapache2-mod-fcgid 2.3.9

Python(part of Ubuntu) 2.7.12

Pillow (part of Ubuntu) 3.1.2

MySQLdb (part of Ubuntu) 1.3.7

PyInstaller 3.2.1

PollyReports 1.7.6

OpenSSL 1.0.2g

OpenSSL FIPS Object Module 2.0.10

JavaScript Bootstrap library 2.3.2

JavaScript Chosen library 1.0.0

JavaScript jQuery library 1.10.2

J JavaScript jQuery-migrate library 1.2.1

JavaScript DataTables library 1.9.4

ColVis 1.0.8
TP-CBG-2018-01.02 Rev. A 20

Table 3-1. ClearVote 1.4.5 Voting System Software (continued)

Firmware/Software Version

JavaScript TableTools library 2.1.5

ZeroClipboard 1.0.4-TableTools2

JavaScript FixedHeader library 2.0.6

JavaScript hotkeys library 1.0

JavaScript tooltip library 1.3

JavaScript pep library 1.0

JavaScript LESS library 1.3.3

Fujitsu fi-6400 PaperStream 1.30.0

Fujitsu fi-6800 10.10.710

Fujitsu fi-7180 PaperStream 1.4.0

3.2 Equipment

This subsection lists the proprietary and COTS equipment provided by the manufacturer as part 

of the test campaign.

For COTS equipment, every effort will be made to verify that the COTS equipment has not been 

modified for use. This will be accomplished by performing research using the COTS equipment 

manufacturer’s websites based on the serial numbers and service tag numbers for each piece of 

equipment. Assigned test personnel will evaluate COTS hardware, system software and 

communications components for proven performance in commercial applications other than 

voting. For PCs, laptops, and servers, the service tag information will be compared to the system 

information found on each machine. Physical external and internal examination will also be 

performed when the equipment is easily accessible without the possibility of damage. Hard 

drives, RAM memory, and other components will be examined to verify that the components 

match the information found on the COTS equipment manufacturer’s websites.

Table 3-2. ClearVote 1.4.5 Voting System Equipment

Component Model Serial Number

ClearDesign Components

Dell Latitude Laptop 5590 9W5D1N2

Dell PowerEdge Server T330 G7LGHL2

Dell PowerEdge Server T130 G9BGRP2

Dell 24 inch Monitor SE2416H FVWV5G2

Dell 22 inch Monitors E2216HV 36765D2 & 90665D2

Dell Precision Tower T3620 GSKSMN2

TP-LINK VPN Router TL-R600VPN 2149342000209

TRENDnet 8-Port Gigabit Switch TEG-S80g CA11238032857
TP-CBG-2018-01.02 Rev. A 21

Table 3-2. ClearVote 1.4.5 Voting System Equipment (continued)

Component Model Serial Number

Lenovo USB DVD Burner LN-8A6NH11B 8SSDX0H33226L1CB7107099

CORSAIR Flash Padlock 3 CMFPLA3B-32GB CBG-Pad-001

ClearAccess Components

Dell OptiPlex AIO 5250 GSXKHL2

Dell 15” Inspiron 7000 series 1QWKQJ2

Brother Laser Printer HL-L2350DW U64964A8N263525

Oki Data Laser Printer B432dn AK7A044093A0

Storm EZ Access Keypad EZ08-222013 15000005

Origin Instruments Sip/Puff 

Breeze with Headset AC-0313-H2 CBG-SP-001

Hamilton Buhl Over-Ear Stereo 

Headphones HA7 CBG-HP-001

ElectionSource Table Top 

Voting Booth (Privacy Screen) VB-60B CBG-VB-001

APC Smart-UPS SMT2200C AS1721132721

Ergotron Stand for Dell OptiPlex 

5250 AIO Neo Flex 1274839-0061

ClearCount Components

Dell Latitude Laptops 5590 3T8D1N2, 4P4F1N2, B5TD1N2

Dell PowerEdge Server T330 G7LGHL2

Dell PowerEdge Server T130 G9BHRP2

Dell Precision Tower T3620 FP5KHL2

Fujitsu Scanner fi-7180 A20DC10302

Fujitsu Scanner fi-6800 100295

Fujitsu Scanner fi-6400 AKHCC00362

Lenovo USB DVD Burner LN-8A6NH11B 8SSDX0H33226L1CB7107099

NETGEAR 8-Port Gigabit 

VPN Firewall FVS318G 40F266BA00280

Western Digital 4 TB External HD My Book WDBFJK0040HBK-NESN

Netac Keypad Encryption 

Portable Hard Drive K390 CBG-disk-001

CORSAIR Flash Padlock 3 CMFPLA3B-32GB CBG-Pad-002

Dell 22 inch Monitor E2216HV GD965D2

Dell 22 inch Monitor P2217 7818672

Dell 22 inch Monitor S2240M CN-0CFGKT-64180-58B-0X3T
TP-CBG-2018-01.02 Rev. A 22

Table 3-2. ClearVote 1.4.5 Voting System Equipment (continued)

Component Model Serial Number

Cisco Catalyst Switch 

(1 Gigabit Router or Switch) 2960-X Series FCW2039B6QF

TP-LINK 5-Port Gigabit Switch TL-SG105E 216C319009010

TP-LINK VPN Router

(1 Gigabit Router or Switch) TL-R600VPN 2157090000334

APC Smart-UPS SMT1500 3S1525X07491

EZ Scanning Shelves Model: WorkEZ CBG-EZ-001, CBG-EZ-002

ClearCast Components

ClearCast Rev 1 CCASFB1732001028

Ballot Box 1224UBB-CB CBG-BB-001

CORSAIR Flash Padlock 3 CMFPLA3B-32GB CBG-Pad-003

3.3 Test Materials

This subsection lists the test materials required to execute the required tests throughout the test 

campaign.

The following materials are expected to be supplied by Clear Ballot to facilitate testing:

 USB Flash Drives, 8 and 32 GB capacity 

 Test Decks

 Power Cords

 Ballot Paper, 4 in. thermal paper, 24 lb. bond or similar paper for results reports

 Labels

 Other materials and equipment as required

3.4 Deliverable Materials

This subsection lists the materials identified by the manufacturer as materials deliverable to the 

end user for the system being tested.

Table 3-3. Voting System Deliverables

Material Version Description

ClearDesign 1.4.4 EMS Software

ClearAccess 1.4.1 BMD software

ClearCount 1.4.5 Central Count and Tabulation Software

ClearCast 1.4.2 Precinct Count Software

ClearVote 1.4.5 TDP* Technical Data Package

* Listed in Section 4.6

3.5 Proprietary Data

All data and documentation considered by the manufacturer to be proprietary will be identified 

and documented in an independent submission along with a Notice of Protected Information.
TP-CBG-2018-01.02 Rev. A 23

4.0 TEST SPECIFICATIONS

Certification testing of the Clear Ballot Group ClearVote 1.4.5 Voting System submitted for 

evaluation will be performed to ensure the applicable requirements of the EAC 2005 VVSG and 

the EAC Testing and Certification Program Manual, Version 2.0, are met. Additionally, all EAC 

Requests for Interpretation (RFI) and Notices of Clarification (NOC) relevant to the system under 

test will be incorporated in the test campaign. A complete listing of the EAC RFIs and NOCs is 

available on the EAC website.

4.1 Requirements (Strategy of Evaluation)

To evaluate the ClearVote 1.4.5 test requirements, the submitted modifications were evaluated 

against each section of the EAC VVSG 1.0 to determine the applicable tests to be performed. 

Based on this assessment, it was determined the following evaluations would be required to 

verify compliance of the modifications:

Limited Technical Documentation Package (TDP) Review

A limited TDP Review will be performed to ensure that all submitted modifications are 

accurately documented and that the documents meet the requirements of the EAC 2005 VVSG. 

Source Code Review, Compliance Build, Trusted Build, and Build Document Review

The source code review will be based on the source code changes made since the previous system 

was certified. 

Physical Configuration Audit (PCA)

A PCA will be performed to compare the voting system submitted for certification testing to the 

manufacturer’s technical documentation. The purpose of the PCA will be to verify that the 

submitted hardware is unmodified from the previously certified voting system

Limited Functional Configuration Audit (FCA)

The FCA for this test campaign will include an assessment of the submitted modifications and 

will include inputs of both normal and abnormal data during test performance. This evaluation 

will utilize baseline test cases as well as specifically designed test cases and will include 

predefined election definitions for the input data. As part of the FCA, one primary and one 

general election will be executed to verify that each of the submitted modifications have been 

successfully implemented.

System Integration Test

The system integration tests will be performed to ensure the ClearVote 1.4.5 functions as a 

complete system. The system integration testing addresses the integration of the hardware and 

software. This testing focuses on the compatibility of the voting system software components and 

subsystems with one another and with other components of the voting system. During test 

performance, the system is configured as would be for normal field use.
TP-CBG-2018-01.02 Rev. A 24

Accuracy Test 

An Accuracy Test will be performed to ensure the ClearVote 1.4.5 correctly captures, stores, 

consolidates, and reports the specific ballot selections, and absence of selections, for each ballot 

position. Due to the limited scope of testing, Accuracy testing will be performed on the 

ClearCount component only; results from the previous certification campaign will be used to 

satisfy this requirement for the unmodified system components.

4.1.1 Rationale for ‘Not Applicable’ Requirements

All requirements that were excluded from the previous test campaign (ClearVote 1.4) were also 

deemed not applicable to this test campaign due to the submitted modifications not impacting the 

specific requirements.

4.2 Hardware Configuration and Design

The hardware configuration and design for the modification are unchanged from the baseline 

system. The ClearVote 1.4.5 Voting System is a paper-based voting system that consists of the 

following major components: ClearDesign, ClearAccess, ClearCast, and ClearCount. ClearVote 

is comprised of one proprietary hardware component (ClearCast) and two COTS hardware 

components (ClearCount) and (ClearAccess). All ClearDesign functions are managed by 

proprietary software running on COTS PCs/laptops/servers, which is excluded from hardware 

testing.

ClearCast – The ClearCast component has been previously subjected to the full suite of 

hardware and electrical testing required by the EAC 2005 VVSG as well as Usability, Security, 

Maintainability, Availability, Safety, and Accuracy Testing, the results of which are contained in 

the final certification test report for the baselined system.

ClearAccess – The ClearAccess component has been previously subjected to the full suite of 

hardware and electrical testing required by the EAC 2005 VVSG as well as Usability, Security, 

Maintainability, Availability, and Accuracy Testing, the results of which are contained in the final 

certification test report for the baselined system.

ClearCount – The ClearCount component is unmodified COTS equipment and is exempt from 

non-operational hardware testing. ClearCount was previously subjected to Temperature/Power 

Variation Testing in conjunction with the ClearCast and ClearAccess in addition to being utilized 

in functional and system level testing such as accuracy, volume and stress, and system integration 

during the test campaign, the results of which are contained in the final certification test report for 

the baselined system.

4.3 Software System Functions

The ClearVote 1.4.5 Election Management System (EMS) consists of a set of applications 

responsible for all pre-voting and post-voting activities used in election definition and 

management process. The ClearVote 1.4.5 EMS applications are as follows:

 ClearDesign

 ClearCount
TP-CBG-2018-01.02 Rev. A 25

4.4 Test Case Design

Test cases are designed based on the manufacturer’s design specifications and the relevant 

technical requirements set forth by the VVSG. Test cases shall be examined based on the 

following aspects of the voting system:

 Hardware qualitative examination design

 Hardware environmental test case design

 Software module test case design and data

 Software functional test case design

 System level test case design

Test cases shall provide information regarding the sequence of actions to be performed for the 

execution of a test, the requirements being met, the test objective, test configuration, equipment 

needed, special requirements, assumptions, and pass/fail criteria. Once the test cases are finalized, 

they will be validated and published for use in the test campaign. The validation of the test case 

will be accomplished by Technical Review and Approval. This validation will include the 

following: confirmation of adequate test coverage of all requirements; confirmation that test case 

results are not ambiguous and gave objective pass/fail criteria; and conformation that any 

automated test suites will produce valid results.

Prior to execution of the required test cases, the system under test will undergo testing 

initialization. The testing initialization will seek to establish the baseline for testing and ensure 

that the testing candidate matches the expected testing candidate and that all equipment and 

supplies are present.

The following will be completed during the testing initialization:

 Ensure proper system of equipment. Check network connections, power cords, keys, etc. 

 Check version numbers of (system) software and firmware on all components. 

 Verify the presence of only the documented COTS. 

 Ensure removable media is formatted and does not contain any data.

 Ensure batteries are fully charged. 

 Inspect supplies and test decks. 

 Record protective counter on all tabulators.

 Review physical security measures of all equipment. 

 Record basic observations of the testing setup and review. 

 Record serial numbers of equipment.

 Retain proof of version numbers.
TP-CBG-2018-01.02 Rev. A 26

4.4.1 Hardware Qualitative Examination Design

Previous hardware examinations were performed on the certified baseline system (ClearVote 

1.4). The updates to the modified system (ClearVote 1.4.5) require no additional hardware testing 

to be performed.

4.4.2 Hardware Environmental Test Case Design

Previous hardware examinations were performed on the certified baseline system (ClearVote 

1.4). The updates to the modified system (ClearVote 1.4.5) require no additional hardware testing 

to be performed. 

4.4.3 Software Module Test Case Design and Data

Pro V&V shall review the manufacturer’s program analysis, documentation, and module test case 

design and shall evaluate the test cases for each module with respect to flow control parameters 

and entry/exit data. As needed, Pro V&V shall design additional test cases to satisfy the coverage 

criteria specified in Volume II, Section 7.2.1. 

Component Level Testing will be implemented during the FCA for each component and 

subcomponent. During the Source Code Review and Compliance Builds, Pro V&V will utilize 

limited structural-based techniques (white-box testing). Additionally, specification-based 

techniques (black-box testing) will be utilized for the individual software components.

Pro V&V shall define the expected result for each test and the ACCEPT/REJECT criteria for 

certification. If the system performs as expected, the results will be accepted. If the system does 

not perform as expected, an analysis will be performed to determine the cause. The test will be 

repeated in an attempt to reproduce the results. If the failure can be reproduced and the expected 

results are not met, the system will have failed the test. If the results cannot be reproduced, the 

test will continue. All errors encountered will be documented and tracked through resolution.

4.4.4 Software Functional Test Case Design and Data

Pro V&V shall review the manufacturer-submitted test plans and data to verify that the individual 

performance requirements specified in the EAC 2005 VVSG and the TDP are reflected in the 

software. As part of this process, Pro V&V shall review the manufacturer’s test case design and 

prepare a detailed matrix of system functions and the test cases that exercise them. During this 

review, emphasis shall be placed on those functions where the manufacturer data on module 

development reflects significant debugging problems, and on functional tests that resulted in high 

error rates.

Pro V&V shall also prepare a test procedure describing all test ballots, operator procedures, and 

the data content of output reports. Pro V&V shall define abnormal input data and operator actions 

and then design test cases to verify that the system is able to handle and recover from these 

abnormal conditions. Pro V&V shall define the expected result for each test and the 

ACCEPT/REJECT criteria for certification. If the system performs as expected, the results will be 

accepted. If the system does not perform as expected, an analysis will be performed to determine 

the cause. The test will be repeated in an attempt to reproduce the results. If the failure can be 

reproduced and the expected results are not met, the system will have failed the test. If the results 

cannot be reproduced, the test will continue. All errors encountered will be documented and 

tracked through resolution.
TP-CBG-2018-01.02 Rev. A 27

4.4.5 System-Level Test Case Design

System Level testing will be implemented to evaluate the complete system. This testing will 

include all proprietary components (software, hardware, and peripherals) and COTS components 

(software, hardware, and peripherals) in a configuration of the system’s intended use. For 

software system tests, the tests shall be designed according to the stated design objective without 

consideration of its functional specification. The system level hardware and software test cases 

shall be prepared independently to assess the response of the hardware and software to a range of 

conditions.

4.5 Security Functions

The system security functions for the modification remain unchanged from the previously 

certified system.

4.6 TDP Evaluation

In order to determine compliance of the modified TDP documents with the EAC 2005 VVSG, a 

limited TDP review shall be conducted. This review will focus on TDP documents that have been 

modified since the certification of the baseline system. The review will consist of a compliance 

review to determine if each regulatory, state, or manufacturer-stated requirement has been met 

based on the context of each requirement. Results of the review of each document will be entered 

on the TDP Review Checklist and reported to the manufacturer for disposition of any anomalies. 

This process will be ongoing until all anomalies are resolved. Any revised documents during the 

TDP review process will be compared with the previous document revision to determine changes 

made, and the document will be re-reviewed to determine whether subject requirements have 

been met.

A listing of all documents contained in the ClearVote 1.4.5 TDP is provided in Table 4-1.

Table 4-1. TDP Documents

Document 

Number Description Version

ClearVote Documents

100101 ClearVote 1.4.5 Approved Parts List 1.1.2

100067 ClearVote 1.4.5 Ballot Stock and Printing Specification 1.0.7

100057 ClearVote 1.4.5 Configuration Management Plan 1.0.13

100069 ClearVote 1.4.5 Glossary 1.0.8

100058 ClearVote 1.4.5 Personnel Deployment and Training Plan 1.0.9

100059 ClearVote 1.4.5 Quality Assurance Program 1.0.10

100086 ClearVote 1.4.5 Security Policy 1.0.10

100071 ClearVote 1.4.5 System Overview 1.0.13

100073 ClearVote 1.4.5 Test and Verification Specification 1.0.10
TP-CBG-2018-01.02 Rev. A 28

Table 4-1. TDP Documents (continued)

Document 

Number Description Version

100128 ClearVote 1.4.5 Change Notes 1.0.1

ClearDesign Documents

100011 ClearDesign 1.4.5 Acceptance Test Checklist 1.0.5

100062 ClearDesign 1.4.5 Administration Guide 1.0.6

100083 ClearDesign 1.4.5 Build Procedures 1.0.5

100103 ClearDesign 1.4.5 Database Specifications 1.0.9

100046 ClearDesign 1.4.5 Functionality Description 1.0.4

100098 ClearDesign 1.4.5 Hardware Specification 1.0.8

100063 ClearDesign 1.4.5 Installation Guide 1.0.20

100082 ClearDesign 1.4.5 Maintenance Guide 1.0.9

100045 ClearDesign 1.4.5 Security Specification 1.0.10

100072 ClearDesign 1.4.5 Software Design and Specification 1.0.10

100074 ClearDesign 1.4.5 System Identification Guide 1.0.5

100043 ClearDesign 1.4.5 System Overview 1.0.12

100041 ClearDesign 1.4.5 User Guide 1.0.12

ClearCount Documents

100102 ClearCount 1.4.5 Acceptance Test Checklist 1.0.7

100009 ClearCount 1.4.5 Build Procedures 1.4.5

100005 ClearCount 1.4.5 Database Specification 1.0.6

100004 ClearCount 1.4.5 Election Administration Guide 1.0.13

100006 ClearCount 1.4.5 Election Preparation and Installation Guide 1.2.4

100021 ClearCount 1.4.5 Functionality Description 1.0.9

100022 ClearCount 1.4.5 Hardware Specification 1.0.10

100023 ClearCount 1.4.5 Maintenance Guide 1.0.11

100070 ClearCount 1.4.5 Reporting Guide 1.0.8

100013 ClearCount 1.4.5 Scanner Operator Guide 1.1.4

100026 ClearCount 1.4.5 Security Specification 1.0.10

100019 ClearCount 1.4.5 Software Design and Specification 1.0.11

100047 ClearCount 1.4.5 System Identification Guide 1.0.6

100024 ClearCount 1.4.5 System Operations Procedures 1.0.9
TP-CBG-2018-01.02 Rev. A 29

Table 4-1. TDP Documents (continued)

Document 

Number Description Version

100025 ClearCount 1.4.5 System Overview 1.0.10

100130 ClearCount 1.4.5 XML Report Conversion Tool 1.0

ClearCast Documents

100095 ClearCast 1.4.5 Acceptance Test Checklist 1.2.2

100096 ClearCast 1.4.5 Approved Parts List 1.1.1

100094 ClearCast 1.4.5 Build Procedures 1.0.2

100079 ClearCast 1.4.5 Functionality Description 1.4.2

100080 ClearCast 1.4.5 Installation Guide 1.1.4

100081 ClearCast 1.4.5 Hardware Specification 1.3.2

100089 ClearCast 1.4.5 Maintenance Guide 1.5.3

100090 ClearCast 1.4.5 Poll Worker Guide 1.5.3

100084 ClearCast 1.4.5 Security Specification 1.3.2

100093 ClearCast 1.4.5 Software Design and Specification 1.3.3

100100 ClearCast 1.4.5 Supervisor Guide 1.6.3

100097 ClearCast 1.4.5 System Identification Guide 1.0.3

100078 ClearCast 1.4.5 System Overview 1.3.3

100027 ClearCast 1.4.5 Tabulator Quick Guide (poster) 1.0

100068 ClearCast 1.4.5 ClearCast Troubleshooting Instructions (poster) 1.0

ClearAccess Documents

100109 ClearAccess 1.4.5 Acceptance Test Checklist 1.0.3

100051 ClearAccess 1.4.5 Build Procedures 1.1

100049 ClearAccess 1.4.5 Functionality Description 1.5.2

100085 ClearAccess 1.4.5 Hardware Specification 1.4.2

100053 ClearAccess 1.4.5 Installation Guide 1.6

100052 ClearAccess 1.4.5 Maintenance Guide 1.7

100054 ClearAccess 1.4.5 Poll Worker Guide 1.7

100035 ClearAccess Poll Worker Instructions (poster) 1.0

100033 ClearAccess Poll Worker Instructions-Multi Day Voting (poster) 1.0

100050 ClearAccess 1.4.5 Security Specification 1.4.4

100099 ClearAccess 1.4.5 Software Design and Specification 1.5
TP-CBG-2018-01.02 Rev. A 30

Table 4-1. TDP Documents (continued)

Document

Number Description Version

100055 ClearAccess 1.4.5 Supervisor Guide 1.9

100038 ClearAccess 1.4.5 System Identification Guide 1.0.5

100044 ClearAccess 1.4.5 System Overview 1.5.3

100056 ClearAccess 1.4.5 Voter Guide 1.1.2

100126 ClearAccess Equipment Modification 1.0

4.7 Source Code Review

Pro V&V will review the submitted source code to the EAC 2005 VVSG and the manufacturersubmitted coding standards. Prior to initiating the software review, Pro V&V shall verify that the 

submitted documentation is sufficient to enable:

(1) A review of the source code and (2) Pro V&V to design and conduct tests at every level of the 

software structure to verify that design specifications and performance guidelines are met.

4.8 QA & CM System Review

The Clear Ballot Group Quality and Configuration Management Manuals shall be reviewed for 

their fulfillment of Volume I, Sections 8 and 9, and the requirements specified in Volume II, 

Section 2. 

The requirements for these sections establish the quality assurance and configuration standards 

for voting systems to which manufacturers must conform and require voting system 

manufacturers to implement a quality assurance and configuration management program that is 

conformant with recognized ISO standards. As part of the review process, the Clear Ballot Group

TDP documents will be reviewed to determine if the stated policies are being followed.

4.9 PCA

The physical configuration audit compares the voting system components submitted for 

qualification to the manufacturer’s technical documentation, and shall include the following 

activities: 

• Establish a configuration baseline of software and hardware to be tested; confirm whether 

manufacturer’s documentation is sufficient for the user to install, validate, operate, and maintain 

the voting system 

• Verify software conforms to the manufacturer’s specifications; inspect all records of 

manufacturer’s release control system; if changes have been made to the baseline version, verify 

manufacturer’s engineering and test data are for the software version submitted for certification 

• If the hardware is non-COTS, Pro V&V shall review drawings, specifications, technical data, 

and test data associated with system hardware to establish system hardware baseline associated 

with software baseline 
TP-CBG-2018-01.02 Rev. A 31

• Review manufacturer’s documents of user acceptance test procedures and data against system’s 

functional specifications; resolve any discrepancy or inadequacy in manufacturer’s plan or data 

prior to beginning system integration functional and performance tests 

• Subsequent changes to baseline software configuration made during testing, as well as system 

hardware changes that may produce a change in software operation are subject to re-examination

4.10 FCA

The functional configuration audit encompasses an examination of manufacturer’s tests, and the 

conduct of additional tests, to verify that the system hardware and software perform all the 

functions described in the manufacturer’s documentation submitted in the TDP. 

In addition to functioning according to the manufacturer’s documentation tests will be conducted 

to insure all applicable EAC 2005 VVSG requirements are met.

4.11 Accuracy

The accuracy test ensures that each component of the voting system can each process 1,549,703 

consecutive ballot positions correctly within the allowable target error rate. The Accuracy test is 

designed to test the ability of the system to “capture, record, store, consolidate and report” 

specific selections and absences of a selection. The required accuracy is defined as an error rate. 

This rate is the maximum number of errors allowed while processing a specified volume of data. 

For paper-based voting systems the ballot positions on a paper ballot must be scanned to detect 

selections for individual candidates and contests and the conversion of those selections detected 

on the paper ballot converted into digital data.

4.12 System Integration

System Level test for the integrated operation of both hardware and software. Compatibility of 

the voting system software components or subsystems with one another, and with other 

components of the voting system environment, shall be determined through functional tests 

integrating the voting system software with the remainder of the system. 

Additionally, the system shall be configured exactly as it would for normal field use. This 

includes connecting all supporting equipment and peripherals including ballot boxes, voting 

booths (regular and accessible), and any physical security equipment such as locks and ties. 

Pro V&V personnel shall properly configure and test the system by following the procedures 

detailed in the ClearVote 1.4.5 voting system technical documentation.

5.0 TEST DATA

The following subsections provide information concerning test data recording, criteria, and 

reduction.

5.1 Data Recording

All equipment utilized for test data recording shall be identified in the test data package. The 

output test data shall be recorded in an appropriate manner as to allow for data analysis. For 

source code and TDP reviews, results shall be compiled in reports and submitted to Clear Ballot 

Group for resolution.
TP-CBG-2018-01.02 Rev. A 32

5.2 Test Data Criteria

The ClearVote 1.4.5 Voting System shall be evaluated against all applicable requirements 

contained in the EAC 2005 VVSG. The acceptable range for system performance and the 

expected results for each test case shall be derived from the manufacturer-submitted technical 

documentation and the EAC 2005 VVSG.

5.3 Test Data Reduction

Test data shall be processed and recorded in the test log book and the relevant Test Cases.

 

6.0 TEST PROCEDURE AND CONDITIONS

The following subsections detail the facility requirements, test setup conditions, and sequence of 

testing.

6.1 Facility Requirements

Unless otherwise annotated, all testing shall be conducted at the Pro V&V test facility located in 

Huntsville, AL, by personnel verified by Pro V&V to be qualified to perform the test. 

Unless otherwise specified herein, testing shall be performed at the following standard ambient 

conditions and tolerances:

 Temperature: 68-75◦ F (±4◦F)

 Relative Humidity: Local Site Humidity

 Atmospheric Pressure: Local Site Pressure

 Time Allowable Tolerance: ±5%

Testing performed at third-party laboratories will be subject to the test parameters and tolerances 

defined by VVSG. If not specified in VVSG, the test facilities’ standard parameters and 

tolerances will be used. These will be reported in the final Test Report.

6.2 Test Set-up

All voting system equipment shall be received and documented using Pro V&V proper QA 

procedures. Upon receipt of all hardware, an inspection will be performed to verify that the 

equipment received is free from obvious signs of damage and/or degradation that may have 

occurred during transit. If present, this damage shall be recorded, photographed, and reported to 

the Clear Ballot Group Representative. Additionally, a comparison shall be made between the 

recorded serial numbers/part numbers and those listed on shipper’s manifest and any 

discrepancies shall be reported to the Clear Ballot Group Representative. TDP items and all 

source code received shall be inventoried and maintained by Pro V&V during the test campaign.

During test performance, the system shall be configured as would be for normal field use. This 

includes connecting all supporting equipment and peripherals.
TP-CBG-2018-01.02 Rev. A 33

6.3 Test Sequence

The ClearVote 1.4.5 Voting System will be evaluated against all applicable requirements in the 

EAC 2005 VVSG. There is no required sequence for test performance.

6.4 Test Operations Procedure

Pro V&V will identify PASS/FAIL criteria for each executed test case. The PASS/FAIL criteria 

will be based on the specific expected results of the system. In the case of an unexpected result 

that deviates from what is considered standard, normal, or expected, a root cause analysis will be 

performed. 

Pro V&V will evaluate every applicable EAC 2005 VVSG requirement. Any deficiencies noted 

will be reported to the EAC and the manufacturer. If it is determined that there is insufficient data 

to determine compliance, this test plan will be altered and additional testing will be performed.
TP-CBG-2018-01.02 A-1

APPENDIX A

PROJECT SCHEDULE
TP-CBG-2018-01.02 A-2

Task Name Start Date End Date Assigned To Duration Predecessors Status

TDP 07/16/18 08/10/18 20d In Progress

Initial Review 07/16/18 07/20/18 Alan 5d In Progress

Compliance Review 07/23/18 08/03/18 Alan 10d 2 Not Started

Final review 08/06/18 08/10/18 Alan 5d 3 Not Started

Test Plan 07/16/18 09/14/18 44d In Progress

Test Plan Creation 07/16/18 07/20/18 Wendy 5d In Progress

Vendor Review & Comments 07/23/18 07/24/18 Wendy 2d 6 Not Started

EAC Submission and Review 07/25/18 08/21/18 Wendy 20d 7 Not Started

EAC Comment Review & Update 08/22/18 08/28/18 Wendy 5d 8 Not Started

EAC Submission & Review of Revision 08/29/18 09/12/18 Wendy 10d 9 Not Started

EAC Approved Test Plan 09/13/18 09/14/18 Wendy 2d 10 Not Started

Source Code 07/02/18 07/17/18 10d Complete

Automated Review 07/02/18 07/03/18 Jack 2d Complete

Source Code Review 07/06/18 07/11/18 Jack 4d 13 Complete

Source Code Re-Review 07/12/18 07/12/18 Jack 1d 14 Complete

Document Review 07/13/18 07/13/18 Jack 1d 15 Complete

Compliance Build 07/16/18 07/17/18 Stephen 2d 16 Complete

System Delivery & Setup 07/09/18 07/17/18 7d Complete

PCA 07/09/18 07/10/18 Stephen 2d Complete

System Setup 07/11/18 07/13/18 Stephen 3d 19 Complete

System Loads & Hardening 07/16/18 07/17/18 Stephen 2d 20 Complete

System Level Testing 07/18/18 08/13/18 19d In Progress

FCA 07/18/18 07/24/18 Stephen 5d 21 In Progress

Usability 07/25/18 07/26/18 Stephen 2d 23 Not Started

Accessibility 07/25/18 07/26/18 Stephen 2d 23 Not Started

Accuracy 07/27/18 07/31/18 Stephen 3d 25 Not Started

Regression Testing 08/01/18 08/02/18 Stephen 2d 26 Not Started

Trusted Build 08/01/18 08/02/18 Jack 2d 26 Not Started

System Loads & Hardening 08/03/18 08/06/18 Stephen 2d 28 Not Started

System Integration 08/07/18 08/13/18 Stephen 5d 29 Not Started

Test Report 08/03/18 11/02/18 65d Not Started

Test Report Creation 08/03/18 08/16/18 Wendy 10d 28 Not Started

Vendor Review & Comments 08/17/18 08/20/18 Wendy 2d 32 Not Started

EAC Submission & Review 09/17/18 10/12/18 Wendy 20d 11 Not Started

EAC Comment Review & Update 10/15/18 10/17/18 Wendy 3d 34 Not Started

EAC Submission & Review of Revision 10/18/18 10/31/18 Wendy 10d 35 Not Started

EAC Approved Test Report 11/01/18 11/02/18 Wendy 2d 36 Not Started