TP-01-01-EOC-2016.02

700 Boulevard South

Suite 102

Huntsville, AL 35802

Phone (256)-713-1111

Fax (256) – 713-1112

Test Plan for EAC VVSG 1.1 Certification Testing

Everyone Counts eLect Quad Audit v.4.4.0

EAC Project ID: ECI1601

Version: Initial Release

Date: November 23, 2016

TP-01-01-EOC-2016.02

REVISIONS

Revision Description Date

NR Initial Release 11/23/16
TP-01-01-EOC-2016.02

1.0 INTRODUCTION ......................................................................................................................1

1.1 References.....................................................................................................................................1

1.2 Terms, Abbreviations, and Definitions.........................................................................................2

1.3 Testing Responsibilities...............................................................................................................3

1.3.1 Project Schedule...........................................................................................................................3

1.3.1.1 Owner Assignments......................................................................................................................3

1.3.1.2 Test Case Development ................................................................................................................3

1.3.1.3 Test Procedure Development and Validation ...............................................................................3

1.3.1.4 Third Party Tests...........................................................................................................................4

1.3.1.5 EAC and Manufacturing Dependencies........................................................................................4

1.4 Target of Evaluation Description.................................................................................................4

1.4.1 System Overview..........................................................................................................................4

1.4.2 Block Diagram............................................................................................................................15

1.4.3 System Limits .............................................................................................................................15

1.4.4 Supported Languages..................................................................................................................17

1.4.5 Supported Functionality..............................................................................................................17

1.4.5.1 Standard (VVSG) Functionality..................................................................................................18

1.4.5.2 Manufacturer Extensions ............................................................................................................18

2.0 PRE-CERTIFICATION TESTING AND ISSUES................................................................18

2.1 Evaluation of Prior VSTL Testing..............................................................................................19

2.2 Evaluation of Prior Non-VSTL Testing......................................................................................19

2.3 Known Field Issues.....................................................................................................................19

3.0 MATERIALS REQUIRED FOR TESTING..........................................................................20

3.1 Software ......................................................................................................................................20

3.2 Equipment...................................................................................................................................20

3.3 Test Materials..............................................................................................................................25

4.0 TEST SPECIFICATIONS .......................................................................................................26

4.1 Requirements ..............................................................................................................................26

4.1.1 Mapping of Requirements to Equipment Type and Features......................................................27

4.1.2 Rationale NA Requirements .......................................................................................................27

4.2 Hardware Configuration and Design ..........................................................................................27

4.3 Software System Functions.........................................................................................................28
TP-01-01-EOC-2016.02

4.4 Test Case Design.........................................................................................................................29

4.4.1 Hardware Qualitative Examination Design ................................................................................29

4.4.1.1 Mapping of Requirements to Specific Interfaces........................................................................30

4.4.2 Hardware Environmental Test Case Design ...............................................................................30

4.4.3 Software Module Test Case Design and Data.............................................................................30

4.4.4 Software Functional Test Case Design and Data........................................................................31

4.4.5 System-Level Test Case Design .................................................................................................31

4.5 Security Functions ......................................................................................................................31

4.6 TDP Evaluation...........................................................................................................................31

4.7 Source Code Review...................................................................................................................35

4.8 QA & CM System Review..........................................................................................................36

5.0 TEST DATA..............................................................................................................................36

5.1 Test Data Recording ...................................................................................................................36

5.2 Test Data Criteria........................................................................................................................36

5.3 Test Data Reduction....................................................................................................................36

6.0 TEST PROCEDURES AND CONDITIONS..........................................................................36

6.1 Facility Requirements...............................................................................................................366

6.2 Test Setup....................................................................................................................................37

6.3 Test Sequence .............................................................................................................................37

7.0 TEST OPERATIONS PROCEDURES ..................................................................................37

7.1 Proprietary Data .......................................................................................................................37

ATTACHMENT A – PROJECT SCHEDULE....................................................................A-1

ATTACHMENT B – IMPLEMENTATION STATEMENT..............................................B-1
1

TP-EOC-2016-01.02

1.0 INTRODUCTION

The purpose of this Test Plan is to document the procedures that Pro V&V, Inc. will follow to 

perform certification testing of the Everyone Counts eLect Quad Audit v.4.4.0 to the 

requirements set forth by voting systems in the U.S. Election Assistance Commission (EAC) 

2015 Voluntary Voting System Guidelines (VVSG) Version 1.1. Prior to submitting the voting 

system for testing, Everyone Counts submitted an application package to the EAC for 

certification of the elect Quad Audit v.4.4.0 to the requirements of the EAC VVSG Version 1.1. 

The application was accepted by the EAC and assigned the unique application number of 

ECI1601. In the application, Everyone Counts selected Pro V&V as the lead VSTL for the test 

engagement. 

Pro V&V received its Voting System Testing Accreditation from The National Institute of 

Standards and Technology’s (NIST) National Voluntary Laboratory Accreditation Program 

(NVLAP) on April 2, 2012 for core test methods. Additionally, Pro V&V was audited by the 

Election Assistance Commission (EAC) and successfully met all requirements for the EAC 

Voting System Testing Laboratory (VSTL) accreditation. As a result, Pro V&V was granted 

accreditation as a VSTL on February 24, 2015.

At test conclusion, the results of all testing performed as part of this test campaign will be 

submitted to the EAC in the form of a national certification test report.

The implementation statement provided by the manufacturer for this test campaign is presented in 

Attachment B.

1.1 References

This subsection lists all documents relevant to the preparation of this Test Plan.

 Election Assistance Commission 2015 Voluntary Voting System Guidelines (VVSG) Version 

1.1, Volume I, “Voting System Performance Guidelines”, and Volume II, “National 

Certification Testing Guidelines”

 Election Assistance Commission Testing and Certification Program Manual, Version 2.0

 Election Assistance Commission Voting System Test Laboratory Program Manual, Version 

2.0

 National Voluntary Laboratory Accreditation Program NIST Handbook 150, 2006 Edition, 

“NVLAP Procedures and General Requirements (NIST Handbook 150)”, dated February 

2006

 National Voluntary Laboratory Accreditation Program NIST Handbook 150-22, 2008 

Edition, “Voting System Testing (NIST Handbook 150-22)”, dated May 2008

 United States 107th Congress Help America Vote Act (HAVA) of 2002 (Public Law 107-

252), dated October 2002

 Pro V&V, Inc. Quality Assurance Manual, Version 7.0

 Election Assistance Commission “Approval of Voting System Testing Application Package” 

letter dated October 6, 2016

 EAC Requests for Interpretation (RFI) (listed on www.eac.gov)
2

TP-EOC-2016-01.02

 EAC Notices of Clarification (NOC) (listed on www.eac.gov)

 Everyone Counts eLect Quad Audit v.4.4.0 Technical Data Package (A listing of the eLect 

Quad Audit v.4.4.0 documents submitted for this test campaign is listed in Section 4.6 of this 

Test Plan)

1.2 Terms, Abbreviations, and Definitions

This subsection lists terms and abbreviations relevant to the hardware, the software, or this Test 

Plan.

“ADA” – Americans with Disabilities Act 1990

“CM” – Configuration Management

“COTS” – Commercial Off-The-Shelf

“DRE” – Direct Record Electronic

“EAC” – United States Election Assistance Commission

“EAC VVSG 1.1” – United States Election Assistance Commission 2015 Voluntary Voting 

System Guidelines, Version 1.1

“EBM” – Electronic Ballot Marker

“EOC” – Everyone Counts

“EMS” – Election Management System

“FCA” – Functional Configuration Audit

“HAVA” – Help America Vote Act

“MPSM” – Master Poll Station Machine

“NIST” – National Institute of Standards and Technology 

“NOC” – Notice of Clarification

“PCA” – Physical Configuration Audit

“QA” – Quality Assurance

“RFI” – Request for Interpretation

“TDP” – Technical Data Package

“UPS” – Uninterruptible Power Supply

“NVLAP” – National Voluntary Laboratory Accreditation Program
3

TP-EOC-2016-01.02

“VSTL” – Voting System Test Laboratory

“VVSG” – Voluntary Voting Systems Guidelines

1.3 Testing Responsibilities

All testing will be conducted under the guidance of Pro V&V by personnel verified by Pro V&V 

to be qualified to perform the testing. 

1.3.1 Project Schedule

The Project Schedule for the test campaign is located in Attachment A – Project Schedule. The 

dates on the schedule are not firm dates but planned estimates based on the anticipated project 

work flow.

1.3.1.1 Owner Assignments

This information is contained in the Project Schedule presented in Attachment A.

1.3.1.2 Test Case Development

Pro V&V will utilize baseline test cases for the Functional Configuration Audit (FCA), Usability, 

Security, and System Integration Tests. These test cases will be augmented with specially 

designed test cases tailored to the specific design of the eLect Quad Audit v.4.4.0. Additionally, 

specific election definitions will be designed for the Operational Status Check. 

1.3.1.3 Test Procedure Development and Validation

Pro V&V will utilize VSTL Test Procedures during the duration of the test campaign. These 

procedures are developed to the EAC 2015 VVSG 1.1 standards. The validation of the VSTL 

Test Procedures is accomplished by Technical Review and Approval. If necessary and where 

practical, a validation might include execution to attempt to achieve the expected results using the 

selected tool of the testing methodology. Test Plans, Test Suites, Test Specifications, and Test 

Cases will be validated prior to execution. This validation will include the following:

 Confirmation of adequate test coverage of all requirements.

 Confirmation that test case results are not ambiguous and have objective pass/fail criteria.

 Confirmation that any automated test suites will produce valid results.
4

TP-EOC-2016-01.02

1.3.1.4 Third Party Tests

Pro V&V will be utilizing third party testing during the performance of hardware testing. All of 

this testing will be witnessed by Pro V&V personnel at the third party test site.

1.3.1.5 EAC and Manufacturing Dependencies

This information is contained in the Project Schedule presented in Attachment A.

1.4 Target of Evaluation Description

The following sections contain a product description and an overview of the design methodology 

of the eLect Quad Audit v.4.4.0 Voting System, as taken from the Everyone Counts technical 

documentation.

1.4.1 System Overview

The eLect Quad Audit 4.4.0 System is a voting system encompassing all aspects of election 

management, including election definition and configuration, ballot creation, voting, data 

transmission, vote data management, reporting, and auditing. The eLect Quad Audit 4.4.0

incorporates polling location ballot delivery, ballot marking, vote capture, and vote storage 

capabilities that operate on commercial off-the-shelf (COTS) hardware located at a Central Office 

and Poll Station. 

The eLect Quad Audit 4.4.0 System consists of the major components listed below:

Software Components:

eLect Quad Audit

The eLect Quad Audit is the software platform that provides election administrators with the tools 

to manage elections. The software is accessed by means of COTS hardware devices. The eLect 

Quad Audit user interface (the EMS) is called eLect Administration and it includes the following: 

 eLect Central Scanning module

 eLect Quad Audit Voting module

 eLect Security Suite utility application

eLect Administration

The eLect Administration dashboard is the voting system’s user interface for the Admin User. It

contains the various system modules for all aspects of election creation and management. eLect 

Administration is used by election administrators to define election criteria, create and set up an 

election configuration, create ballots, generate reports, and manage the election life cycle.

eLect Security Suite

eLect Security Suite is a utility application that provides encryption and decryption services. 

There are two modes of operation:
5

TP-EOC-2016-01.02

1. Standalone - runs as a desktop application on the eLect Quad Audit solution’s Air-Gapped 

Machine. The election official/administrator creates public and private security keys for 

vote data encryption in the system.

2. Embedded - The automated utility is directly called by eLect Quad Audit to encrypt cast 

ballots during the voting session and decrypt encrypted ballots after the voting session

eLect Quad Audit Voting

The eLect Quad Audit voting module is the core software module of the voting system. A voting 

session is initiated at the EBM once the QR code is scanned; the voter is then served the assigned 

ballot style. Ballot submissions are saved, transmitted, confirmed, scanned, encrypted, and stored 

in the Master Poll Station Machine (MPSM) database. 

eLect Central Scanning

eLect Central Scanning is a scanning and imaging module incorporated within eLect

Administration .eLect Central Scanning supports write-in ballot scanning, imaging and

adjudication processes.

eLect Poll Station Management Console

The eLect Poll Station Management Console is the user interface that poll workers access through 

the Poll Station EBM. 

Hardware Components:

Central Server

The Central Server runs on a Linux operating system and is the primary system device that hosts 

the eLect Quad Audit voting system, including eLect Administration (the EMS) and eLect Voting 

(for tabulation). The Central Server is located in the Central Office.

eLect Administration Access Machine

The eLect Administration Access Machine (EAAM) runs on a Windows operating system and is 

connected to the Central Server, the Central Scanner, and a printer. Election management activity 

is executed using the eLect Administration interface. The EAAM is located in the Central Office.

Air-Gapped Machine

The Air-Gapped Machine is a dedicated COTS laptop that is isolated from the voting system and 

any network connection. The Air-Gapped Machine hosts eLect Security Suite, which provides

ballot encryption and decryption services. The Air-Gapped Machine is located in the Central

Office.

Imaging Server

The Imaging Server provides dedicated image processing and character mark recognition for 

scanned paper ballots only. The Imaging Server runs on a Windows operating system and is used 

for eLect Central Scanning. The Imaging Server is located in the Central Office.
6

TP-EOC-2016-01.02

Central Scanner

The Central Scanner is a COTS imaging scanner that is connected to the eLect Administration 

Access Machine and performs eLect Central Scanning functions. The Central Scanner is located 

in the Central Office.

Master Poll Station Machine (MPSM)

The Master Poll Station Machine (MPSM) is a laptop that runs on a Linux operating system and 

is used to manage the voting process. The election configuration from the Central Server is 

replicated to each MPSM prior to Poll Station delivery. The MPSM is then wired to the 

Electronic Ballot Markers (EBMs) using a local network switch. MPSMs are located in the Poll 

Stations.

Electronic Ballot Marker (EBM)

The Electronic Ballot Marker (EBM) is a COTS tablet that runs on a Windows operating system. 

The EBM is used to mark the ballot and transmit digital vote data to and from the MPSM. The 

EBM does not store vote data. EBMs are located in the Poll Stations. The EBM can function in 

Optical Scan or DRE mode.

Ballot Box

The Ballot Box provides physical security for completed ballots. The Ballot Box includes the 

secure container and the COTS Ballot Box scanner. The Ballot Box is located in the Poll Station 

and is used in Optical Scan mode only.

The follow tables provide the software and hardware components of the eLect Quad Audit 4.4.0

System to be tested, identified with version numbers.

Table 1-1: Firmware/Software Versions

eLect Quad Audit 4.4.0 System 

Component Manufacturer Version

Air-Gapped Machine

OS: Ubuntu Canonical 12.04.5 LTS

eLect Security Suite (Stand Alone) Everyone Counts 1.3.0

Chef Chef 12.4.3

NodeJS Node.js Foundation v.4.2.3

OpenSSL The OpenSSL Project OPenSSL 1.0.1p-fips 9 Jul 

2015
7

TP-EOC-2016-01.02

Table 1-1: Firmware/Software Versions (continued)

Central Server

OS: Ubuntu Canonical 12.04.5 LTS

eLect Security Suite (Embedded) Everyone Counts 1.3.0

eLect Everyone Counts e546052b2ff7093e1b8eb63

df2dG10itL6a9b3e4178830

eLect Admin Everyone Counts a743f0307f727360229edd6

7499Sata9sbh0d6b0bf877

MariaDB MariaDB Corp. 10.0.26

Tomcat Apache 6.0.35

Apache Webserver Apache 2.2.22

Perl Perl 5.14.2

Chef Chef 12.4.3

Java Oracle 1.8.0_65

LAME LAME 3.99.3

Festival

Centre for Speech 

Technology Research 

(CSTR), University of 

Edinburgh, UK

2.1

eLect Administration Access Machine

OS: Windows Microsoft 8.1 Pro (64 bit)

Dynamic Web TWAIN HTML5 

edition Dynamsoft 10.2.312

eLect Secure Transfer Installer Everyone Counts 1.2.1

Chrome Google 47.0.2526.106

iP110 Printer Driver Canon 6.3.9600.16384
8

TP-EOC-2016-01.02

Table 1-1: Firmware/Software Versions (continued)

eLect Quad Audit 4.4.0 System 

Component Manufacturer Version

eLect Administration Access Machine (continued)

Management Engine Components Intel 10.0.30.1060

Network Connections Drivers Intel 19.2

Kodak i4250 Driver Kodak 1.2

Kodak i3400 Driver Kodak 3.0

Kodak i2820 Driver Kodak 4.5

E5450 Drivers: Chipset Driver Dell

Chipset_Driver_YVGN8_

WN_ 

10.0.30.1060_A01.EXE

E5450 BIOS Dell A12

Central Scan Imaging Server

OS: Windows Microsoft 8.1 Pro (64 bit)

Visual C++ 2005 Redistributable 

KB246775 Microsoft 8.0.51011

Visual C++ 2010 x86 

Redistributable Microsoft 10.0.30319

Microsoft.NET Framework Microsoft v4.0.30319

Recostar OpenText 7.0.2

Recostar Patch MSP, Service Pack 

1

OpenText

revision #: {7F534C39-

D036-42A1-BF60-

12971ACF811C}

E1C eLect Central Scanning: 

Imaging Service exe Everyone Counts 1.4
9

TP-EOC-2016-01.02

Table 1-1: Firmware/Software Versions (continued)

eLect Quad Audit 4.4.0 System 

Component Manufacturer Version

Imaging Server (continued)

Power Chute Business Edition 

Agent Schneider Electric 9.1.1.604

Local GPO Microsoft 3.0.60.0

Enhanced Mitigation Experience 

Toolkit Microsoft 5.2

Master Poll Station Machine (MPSM)

OS: Ubuntu Canonical 12.01.5 LTS

eLect Security Suite Everyone Counts 1.3.0

eLect Everyone Counts e546052b2ff7093e1b8eb63

df2dG10itL6a9b3e4178830

Apache Webserver Apache 2.2.22

Perl Perl 5.14.2

Chef Chef 12.4.3

Java Oracle 1.8.0_65

MariaDB MariaDB Corp 10.0.26

LAME LAME 3.99.3

Festival

Centre for Speech 

Technology Research 

(CSTR), University of 

Edinburgh, UK

2.1

Electronic Ballot Marker (EBM)

OS: Windows Microsoft 8.1 Pro (64 bit)

5130 Chipset Driver Dell 603.9600.1948.34944
10

TP-EOC-2016-01.02

Table 1-1: Firmware/Software Versions (continued)

eLect Quad Audit 4.4.0 System 

Component Manufacturer Version

Electronic Ballot Marker (continued)

Dell System BIOS update Dell A04

Chrome Google 47.0.2526.106

i110 Printer Driver Canon 6.3.9600.16384

EMET Microsoft 5.2

Local GPO Microsoft 3.0.60.0

Intel Graphics Driver Intel 10.18.10.4176

JoyToKey JoyToKey 5.7

Ballot Box Scanner

OS: Windows Microsoft

Microsoft Windows

Embedded 8.1 Industry Pro

6.3.9600 Build 9600 

Chrome Google 47.0.2526.106

Everyone Counts Ballot Box 

Software Everyone Counts 1.0.0

Twain TWAIN Working

Group 10.5

Pillow PIL: Python Imaging

Library 3.0.60.0

Python Python software

foundation 2.7.10

zbar

Licensed under the

GNU Lesser General

Public License

0.10
11

TP-EOC-2016-01.02

Table 1-2: Hardware Versions

eLect Quad Audit 4.4.0

System Component Manufacturer Model Number

Air-Gapped Machine

Laptop Dell Latitude E5450

Removable Storage Device: 

128GB USB Flash Drive PNY P-FD128TBOP-GE

Surge Protector Tripp Lite ISOBAR6ULTRAHG

Central Server

Server SuperMicro SC813MTQ-R400CB

Monitor Dell SE2416HX

Keyboard Logitech MK120

Mouse Logitech MK120

Uninterruptible Power Supply 

(UPS) with USB connectivity APC SMC1500

Local Network Switch Netgear JGS516

Uninterruptible Power Supply 

(UPS) Eaton 5S700

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT

Removable Storage Device: 

VEX 128GB USB Flash Drive Patriot PSF128GVEX3USB

Removable Storage Device: 

128GB USB Flash Drive PNY P-FD128TBOP-GE
12

TP-EOC-2016-01.02

Table 1-2: Hardware Versions (continued)

eLect Quad Audit 4.4.0

System Component Manufacturer Model Number

eLect Administration Access Machine

Laptop Dell Latitude E5450

Printer Canon iP110

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT

Central Scan Imaging Server

Server SuperMicro SC813MTQ-R400CB

Uninterruptible Power Supply 

(UPS) with USB connectivity APC SMC1500

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT

Monitor Dell SE2416HX

Keyboard Logitech MK120

Mouse Logitech MK120

Master Poll Station Machine (MPSM)

Laptop Dell Latitude E5450

Removable Storage Device: 

128GB USB Flash Drive PNY P-FD128TBOP-GE

Laptop Security Bracket Computer Security 

Products, Inc. CSP-4020

Uninterruptible Power Supply 

(UPS) Eaton 5S700
13

TP-EOC-2016-01.02

Table 1-2: Hardware Versions (continued)

eLect Quad Audit 4.4.0

System Component Manufacturer Model Number

Master Poll Station Machine (MPSM)(continued)

Local Network Switch Netgear JGS516

Surge Protector Tripp Lite ISOBAR6ULTRAHG

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT

Electronic Ballot Marker (EBM)

Tablet Dell Venue 11 Pro

5130

Dell Venue 11 Pro POS Stand 

Enclosure iPadSimple IBS-002-1004

Activation Token 4-inch Tray Everyone Counts 002

LED Multi SELCA light Pickogen B013U47A5A

USB 3.0 Gigabit Ethernet and 3-

port Hub J5Create JUH470

1 ft. Extension USB Blue Right 

Angle USB 3.0 Type A Male to 

Female Adapter

Smays GAZEWALL19810035

Headphone Jack Extension Cable iMBAPrice iMBA-PS-06MF

Printer Canon iP110

Surge Protector Tripp Lite ISOBAR6ULTRAHG

Cat6 Ethernet Cable 20' Cable Exchange CAX-C6-SHLBUSR-20FT

Hub Enclosure Everyone Counts PVV-EOC001
14

TP-EOC-2016-01.02

Table 1-2: Hardware Versions (continued)

eLect Quad Audit 4.4.0

System Component Manufacturer Model Number

EBM - Accessibility

ATI Storm Interface EZ05-222013

Sip/Puff Breeze with Headset USB 

Interface Origin Instruments AC-0313-MU

Headphones Koss KPH7

Ballot Magnifier MakExpress B01AL8J2NQ

Ballot Box

Ballot Box Container PakFlatt Fortress Lander

Model: 1.0

Ballot Box Scanner Kodak Scan Station

730EX

Uninterruptible Power Supply 

(UPS) Eaton 5S700

Cat6 Ethernet Cable 50' Cable Exchange CAX-C6-SHLBUSR-50FT

Ballot Bag

Provisional Ballot Bag Election Source BA-51

Central Scanner

Large Scanner Kodak i4250

Medium Scanner Kodak i3400

Small Scanner Kodak i2820

Uninterruptible Power Supply 

(UPS) APC SMC1500
15

TP-EOC-2016-01.02

1.4.2 Block Diagram

The following diagram provides an overview of the eLect Quad Audit system operations:

Figure 1.1: eLect Quad Audit System Operations Overview

1.4.3 System Limits

The system limits that Everyone Counts has stated to be supported by the eLect Quad Audit 

v.4.4.0 are listed in the table below.

Table 1-3. System Limits

Device/Component Limits

eLect 

Administration

Candidate names: A limit of 100 characters per candidate name applies. 

Lengthy candidate names limit ballot style configuration. To accommodate 

this limitation, users should adjust the candidate name length. Visual 

inspection and confirmation to verify this is performed during the ballot 

configuration process on both the digital ballots and the mail-in paper 

ballots.

Central Server

 Active Elections: The Central Server has the capability to run one 

active election. The Central Server cannot run multiple elections.

 Name cannot exceed a maximum of two (2) digits when naming the 

Central Server: For example, c01, c20, c99 are acceptable
16

TP-EOC-2016-01.02

Table 1-3. System Limits (continued)

Device/Component Limits

Master Poll Station 

Machine

 Cannot alter election data after the configuration is defined and pulled 

from the Central Server.

 Cannot exceed a maximum of 1000 devices names

 Cannot exceed a maximum of two mpsm (2) digits when naming the 

MPSM after naming the central with two (2) digits: For example, 

cXXpsYY.

Electronic Ballot 

Marker (EBM)

 Two EBMs per MPSM

 One surge protector for every two EBM tablets

 Cannot store any election data on device

 Cannot display the ballot without connection to MPSM

 Cannot mark the ballot without connection to MPSM

 Cannot review the ballot without connection to MPSM

 Cannot alter any election data or settings beyond a single ballot voting 

session

EBM Printer Paper 

Limits

Paper Size: US Standard 8.5x11 inches

Paper Color: White

Paper Weight: 20 lb. minimum to 24 lb. maximum

Capacity: 50 sheets in feeder tray

Ballot Box Limits

 Cannot reverse roll a paper ballot once accepted from the ballot box 

scanner

 Does not maintain the ballot counter without connection to MPSM 

(maintained on the MPSM, and displays on ballot box display screen)

 Does not store election data or vote data on the device

 Cannot operate as a ballot box without connection to the MPSM

 Will not duplicate scanning of individual ballots

Central Scan Paper 

Ballot Limits

Paper Size in inches:

- U.S. Letter (8.5 x 11)

- U.S. Legal (8.5 x 14)

- U.S. Ledger (11 x 17) not supported by the Kodak i2820 scanner or 

i3400 scanner

Paper Color: White

Paper Weight: 20 lb. minimum to 24 lb. maximum

Printing Accuracy: Anchors are expected to be 5mm from the edge of the 

paper. The system can manage an error of +/10 mm, and automatically 

scales the scanned images. It is recommended not to exceed an error of 

more than +/5mm.

Central Scan Limits

 Depending on the size of the scanner, Central Scan supports 

maximum batch sizes of up to 500 ballots, and deck loads of 100 

sheets or less on the scanner. Batches can be paused between scanner 

reloads, when image processing continues for the batch, allowing for 

the display of scanned images to be added to the images in the batch.
17

TP-EOC-2016-01.02

Table 1-3. System Limits (continued)

Device/Component Limits

Central Scan Limits 

(continued)

 Additionally, in the creation of the paper ballot, it is important to note 

that every candidate name and contest name must be uniquely 

different from all others on the ballot or in the election. For example, 

if two candidates each have the name John Smith, a differentiator 

such as middle initial, party affiliation, etc., must be placed with the 

candidate name to ensure the proper display of tabulation results at 

the close of the election.

Ballot Pens

Recommended and tested ballot pens for optimal scanning and imaging 

include:

 Pilot Precise V7 Rolling Ball Point Pens Black 0.7mm

 Papermate InkJoy 300 Ballpoint Pens Black 0.7mm

1.4.4 Supported Languages

The following languages have been stated by Everyone Counts to be supported by the eLect Quad 

Audit v.4.4.0:

 English

 Spanish

 Bengali

 Korean

 Chinese

 Vietnamese

 Non-written language audio support

Support for all stated languages will be verified; however, only English and Spanish language 

ballots will be cast during the performance of functional testing. Additionally, one character 

based language (Chinese) will be tested during System Integration Testing.

1.4.5 Supported Functionality

The eLect Quad Audit v.4.4.0 is designed to support the following voting variations:

 General Elections

 Closed Primary

 Open Primary

 Early Voting

 Partisan/Non-Partisan Offices

 Write-In Voting

 Primary Presidential Delegation Nominations

 Straight Party Voting
18

TP-EOC-2016-01.02

 Split Precincts

 Vote for N of M

 Ballot Rotation

 By Precinct

 Weighted by Number of Registered Voters per Precinct

 Cumulative Voting

 Provisional or Challenged Ballots

As stated in the Implementation Statement, the eLect Quad Audit v.4.4.0 does not include support 

for Ranked Order Voting, Cross-party Endorsement, Recall Issues with options, or Proportional 

Representation.

1.4.5.1 Standard (VVSG) Functionality

The eLect Quad Audit v.4.4.0 will be evaluated to all applicable EAC 2015 VVSG 1.1 

requirements. 

1.4.5.2 Manufacturer Extensions

There are no manufacturer extensions on testing for this test campaign.

2.0 Pre-Certification Testing and Issues

The following pre-certification testing has been performed on an “At-Risk” basis:

 

Source Code Review

A first pass review has been performed for all source code submitted by Everyone Counts for the 

eLect Quad Audit v.4.4.0. Any issues discovered were compiled and reported to Everyone Counts 

for resolution. Subsequent submissions will be reviewed by comparing newly submitted code 

against previously reviewed code to ensure all documented issues are resolved.

Technical Date Package (TDP) Review

An initial TDP review was performed on the documents submitted for this test campaign. The 

initial review is an abbreviated review performed in order to determine if the documents contain 

sufficient information. It was determined that various documents were lacking in information. 

Additionally, inconsistencies were noted throughout the TDP package. The results of the review 

were reported to Everyone Counts for resolution. Everyone Counts has subsequently revised and 

resubmitted the TDP and Pro V&V is currently reviewing these submissions. Specific documents 

have been reviewed multiple times in order to facilitate performance of the FCA. The findings of 

each review performed are tracked in the TDP review spreadsheet.
19

TP-EOC-2016-01.02

Physical Configuration Audit (PCA)

An initial PCA has been performed to baseline the system prior to test campaign commencement. 

The PCA was performed by documenting each hardware and software component of the voting 

system by name, model, serial number, major component, and any other relevant information 

needed for identification. Photographs of each hardware component were also taken. 

Hardware Testing

Pro V&V has initiated Hardware Testing on an at-risk basis. This testing is being conducted at an

off-site third party laboratory.

For the FCA, Test Suites are developed to cover the pre-voting, voting, and post-voting phases of 

the election process including the usability of the system during each phase.

Functional Configuration Audit (FCA)

Pro V&V has begun conducting the FCA for the eLect Quad Audit v.4.4.0. This includes multiple 

reviews of specific TDP documents to clarify and expand upon information concerning the 

system functions and design methodology. During the FCA, assigned test personnel mapped the 

voting system functional options, input data to exercise all system functions, and took note of all 

program logic and data processing errors (this step will be performed throughout the test 

campaign in parallel with other examinations as long as all functions are exercised and 

independently tracked).

Assigned test personnel also reviewed the requirements matrix for completeness, developed and 

performed system-specific Test Cases for any functions not exercised during other areas of the 

test campaign or for functions judged to need individual examination, reviewed the 

manufacturer’s test procedures and test results to determine if the voting systems functional 

capabilities have been adequately tested, and analyzed the results of all testing and verify that all 

functionality and requirements not satisfied during previous testing have been met.

2.1 Evaluation of Prior VSTL Testing

The eLect Quad Audit v.4.4.0 is a new voting system that has not previously been tested to 

applicable federal standards in the EAC Program. 

2.2 Evaluation of Prior Non-VSTL Testing

Everyone Counts submitted the following summative usability report for review: “Usability Test 

Report of eLect Quad Audit v. 4.4.0 with 53 Voter Participants and 16 Poll Worker Participants 

for Requirement VVSG 1.1 Volume I, Section 3, and Volume 2, Section 7” prepared by Mile7, 

dated November 2, 2015.

2.3 Known Field Issues

This system has never been fielded; therefore, there are no known field issues to consider.
20

TP-EOC-2016-01.02

3.0 Materials Required for Testing

The following sections list all materials needed to enable the test engagement to occur.

The materials required for testing of the eLect Quad Audit v.4.4.0 include all materials to enable 

the test campaign to occur. This includes the applicable software and equipment as well as the 

TDP, test support materials, and deliverable materials. 

3.1 Software

This subsection lists the proprietary and COTS software to be provided by the manufacturer as 

part of the test campaign. This includes all software required for the performance of hardware, 

software, telecommunications, security, and system integration tests.

Table 3-1. Voting System Software

Firmware/Software Manufacturer Version

Air-Gapped Machine

OS: Ubuntu Canonical 12.04.5 LTS

eLect Security Suite (Stand Alone) Everyone Counts 1.3.0

Chef Chef 12.4.3

NodeJS Node.js Foundation v4.2.3

OpenSSL The OpenSSL Project OpenSSL 1.0.1p-fips 9 

Jul 2015

Central Server

OS: Ubuntu Canonical 12.04.5 LTS

eLect Security Suite (Embedded) Everyone Counts 1.3.0

eLect Everyone Counts e546052b2ff7093e1b8eb

63df2

eLect Admin Everyone Counts a743f0307f727360229ed

d6749

MariaDB MariaDB Corp. 10.0.26

Tomcat Apache 6.0.35

Apache Webserver Apache 2.2.22

Perl Perl 5.14.2

Chef Chef 12.4.3

Java Oracle 1.8.0_65

LAME LAME 3.99.3

Festival N/A 2.1
21

TP-EOC-2016-01.02

Table 3-1. Voting System Software (continued)

Firmware/Software Manufacturer Version

eLect Administration Access Machine

OS: Windows Microsoft 8.1 Pro (64 bit)

Dynamic Web TWAIN HTML5 edition Dynamsoft 10.2.312

eLect Secure Transfer Installer Everyone Counts 1.1.2

Chrome Google 47.0.2526.106

iP110 Printer Driver Canon 6.3.9600.16384

Management Engine Components Intel 10.0.30.1060

Network Connections Drivers Intel 19.2

Kodak i4250 Driver Kodak 1.2

Kodak i3400 Driver Kodak 3.0

Kodak i2820 Driver Kodak 4.5

E5450 Drivers: Chipset Driver Dell

Chipset_Driver_YVGN8

_WN_ 

10.0.30.1060_A01

.EXE

E5450 BIOS Dell A12

Central Scan Imagine Server

OS: Windows Microsoft 8.1 Pro (64 bit)

Visual C++ 2005 Redistributable 

KB246775 Microsoft 8.0.51011

Visual C++ 2010 x86 Redistributable Microsoft 10.0.30319

Microsoft.NET Framework Microsoft v4.0.30319

Recostar OpenText 7.0.2

Recostar Patch MSP, Service Pack 1 OpenText

revision #: {7F534C39-

D036-42A1-BF60-

12971ACF811C}

E1C eLect Central Scanning: Imaging 

Service exe Everyone Counts 1.4

Power Chute Business Edition Agent Schneider Electric 9.1.1.604

Local GPO Microsoft 3.0.60.0

Enhanced Mitigation Experience Toolkit Microsoft 5.2

MPSM

OS: Ubuntu Canonical 12.04.5 LTS

eLect Security Suite Everyone Counts 1.3.0
22

TP-EOC-2016-01.02

Table 3-1. Voting System Software (continued)

Firmware/Software Manufacturer Version

MPSM (continued)

eLect Everyone Counts e546052b2ff7093e1b8eb

63df2

Apache Webserver Apache 2.2.22

Perl Perl 5.14.2

Chef Chef 12.4.3

Java Oracle 1.8.0_65

MariaDB MariaDB Corp 10.0.26

LAME LAME 3.99.3

Festival --- 2.1

EBM (DRE and/or OpScan)

OS: Windows Microsoft 8.1 Pro (64 bit)

5130 Chipset Driver Dell 603.9600.1948.34944

Dell System BIOS update Dell A04

Chrome Google 47.0.2526.106

i110 Printer Driver Cannon 6.3.9600.16384

EMET Microsoft 5.2

Local GPO Microsoft 3.0.60.0

Intel Graphics Driver Intel 10.18.10.4176

JoyToKey JoyToKey 5.7

Ballot Box Scanner (OpScan Mode Only)

OS: Windows Microsoft

Microsoft Windows 

Embedded

8.1 Industry Pro 6.3.9600

Build 9600

Google Chrome Google 47.0.2526.106

Everyone Counts Ballot Box Software Everyone Counts 1.0.0

Twain TWAIN Working

Group 1.0.5

Pillow PIL: Python Imaging

Library 2.7.0

Python Python software

foundation 2.7.10

zbar

Licensed under the

GNU Lesser

General Public

License

0.10
23

TP-EOC-2016-01.02

3.2 Equipment

This subsection lists the proprietary and COTS equipment to be provided by the manufacturer as 

part of the test campaign. This includes all equipment required for the performance of hardware, 

software, telecommunications, security, and system integration tests.

For COTS equipment, every effort will be made to verify that the COTS equipment has not been 

modified for use. This will be accomplished by performing research using the COTS equipment 

manufacturer’s websites based on the serial numbers and service tag numbers for each piece of 

equipment. Assigned test personnel will evaluate COTS hardware, system software and 

communications components for proven performance in commercial applications other than 

voting. For PCs, laptops, and servers, the service tag information will be compared to the system 

information found on each machine. Physical external and internal examination will also be 

performed when the equipment is easily accessible without the possibility of damage. Hard 

drives, RAM memory, and other components will be examined to verify that the components 

match the information found on the COTS equipment manufacturers’ websites.

Table 3-2. Voting System Equipment

Component Manufacturer Model #

Air-Gapped Machine

Laptop Dell Latitude E5450

Removable Storage Device: 

128GB USB Flash Drive PNY P-FD128TBOPGE

Surge Protector Tripp Life ISOBAR6ULTRA

HG

Central Server

Server SuperMicro SC813MTQ-R400CB

Smart-UPS 1000 

Watts/1500VA LCD 230V APC SMC1500

Local Network Switch Netgear JGS516

UPS Eaton 5S700

Monitor Dell SE2416HX

Keyboard Logitech MK120

Mouse Logitech MK120

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT

Removable Storage Device: 

VEX 128GB USB Flash Drive Patriot PSF128GVEX3 USB

Removable Storage Device: 

128GB USB Flash Drive PNY P-FD128TBOPGE

eLect Administration Access Machine

Laptop Dell Latitude E5450

Printer Canon iP110

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT
24

TP-EOC-2016-01.02

Table 3-2. Voting System Equipment (continued)

Component Manufacturer Model #

Central Scan Imaging Server

Server SuperMicro SC813MTQ-R400CB

Smart-UPS 1000 

Watts/1500VA LCD 230V APC SMC1500

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT

Monitor Dell SE2416HX

Keyboard Logitech MK120

Mouse Logitech MK120

MPSM

Laptop Dell Latitude E5450

Removable Storage Device: 

128GB USB Flash Drive PNY P-FD128TBOPGE

Laptop Security Bracket Computer Security Products, 

Inc CSP-4020

5S700 Eaton 5S700

Local Network Switch Netgear JGS516

Surge Protector Tripp-Lite ISOBAR6ULTRAHG

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT

EBM (DRE and/or OpScan)

Tablet Dell Venue 11 Pro 5130

Dell Venue 11 Pro POS Stand 

Enclosure iPadSimple IBS-002-1004

Activation 4-inch Token Tray Everyone Counts 002

LED Light Pickogen B013U47A5A

USB 3.0 Gigabit Ethernet and 

3-port Hub J5Create JUH470

1 ft Extension USB Blue Right 

Angle USB 3.0 Type A Male to 

Female Adapt

Smays GAZEWALL19810035

Headphone Extension Cable iMBAPrice iMBA-PS-06MF

Printer Canon iP110

Surge Protector Tripp-Lite ISOBAR6ULTRAHG

Cat6 Ethernet Cable 10' Cable Exchange CAX-C6-SHLBUSR-10FT

Hub Enclosure Everyone Counts PVV-EOC001

EBM-Accessibility

ATI Storm Interface EZ05-222013

Sip/Puff Device Origin Instruments AC-0313-MU

Headphones Koss KPH7

USB/Ethernet Hub J5Create JUH470
25

TP-EOC-2016-01.02

Table 3-2. Voting System Equipment (continued)

Component Manufacturer Model #

Ballot Box

Ballot Box Container PakFlatt Fortress Lander

Model: 1.0

Ballot Box Scanner Kodak Scan Station

730EX

Uninterruptible Power Supply 

(UPS) Eaton 5S700

Cat6 Ethernet Cable 50' Cable Exchange CAX-C6-SHLBUSR-50FT

Ballot Bag

Provisional Ballot Bag Election Source BA-51

Central Scanning Scanner

Scanner Kodak i4250

Scanner Kodak i3400

Scanner Kodak i2820

Uninterruptible Power Supply 

(UPS) APC SMC1500

3.3 Test Materials

This subsection lists all test materials required for the performance of the test campaign.

The materials required for testing of the eLect Quad Audit v.4.4.0 include all materials to enable the test 

campaign to occur. This includes the applicable software and equipment as well as the TDP, test support 

materials, and deliverable materials. A detailed list of the consumables required for each implementation 

of the eLect Quad Audit v.4.4.0 system in contained in the Everyone Counts document RF-1505 

Consumables. 

3.4 Deliverable Materials

This subsection lists all materials delivered as part of the voting system to the users.

All materials delivered to the end user as part of the eLect Quad Audit v.4.4.0 System are detailed in the 

Everyone Counts Configuration Items Register (Document No. DOC-1101). This document provides 

detailed information for all software, hardware, and TDP documents. 
26

TP-EOC-2016-01.02

4.0 Test Specifications

Certification testing of the eLect Quad Audit v.4.4.0 submitted for evaluation will be performed 

to ensure the applicable requirements of the EAC 2015 VVSG 1.1 and the EAC Testing and 

Certification Program Manual are met. Additionally, all EAC Request for Interpretations (RFI) 

and Notices of Clarification (NOC) relevant to the system under test will be incorporated in the 

test campaign. A complete listing of the EAC RFIs and NOCs is available on the EAC website.

4.1 Requirements

To evaluate the eLect Quad Audit v.4.4.0 test requirements, each section of the EAC 2015 VVSG 

1.1 will be analyzed to determine the applicable tests. The EAC 2015 VVSG 1.1 Volume I 

Sections, along with the strategy of evaluation, are described below:

Section 2: Functional Requirements

The requirements in this section shall be tested during the FCA and System Integration Test. 

This evaluation will utilize baseline test cases as well as specifically designed test cases and will 

include predefined election definitions for the input data.

Section 3: Usability and Accessibility Requirements

The requirements in this section shall be tested during the Usability and Accessibility Testing. 

This evaluation will utilize baseline test cases as well as specifically designed test cases and will 

include predefined election definitions for the input data. Pro V&V will review the results of 

Summative Usability Testing performed on the system.

Section 4: Hardware Requirements

The requirements in this section shall be tested and/or evaluated by personnel verified by Pro 

V&V to be qualified to perform the testing.

Section 5: Software Requirements

The requirements in this section shall be tested utilizing a combination of review and functional 

testing during the Source Code Review, TDP Review, and FCA. 

Section 6: Telecommunications Requirements

The requirements in this section shall be tested utilizing baseline test cases as well as specifically 

designed test cases.

Section 7: Security Requirements

The requirements in this section shall be tested during the Source Codec Review, Security Tests, 

and FCA.

Section 8: Quality Assurance and Configuration Management

The requirements in this section shall be tested throughout the test campaign. This testing will 

utilize a TDP Review in conjunction with the Source Code Review and PCA to determine 

compliance to the EAC 2015 VVSG 1.1 requirements and the requirements stated in the 

Everyone Counts technical documentation. The review of the Quality Assurance and 

Configuration Management documentation will focus on Everyone Counts’ adherence to its 

stated QA and CM processes.
27

TP-EOC-2016-01.02

Throughout the test campaign, Pro V&V personnel shall maintain a test log identifying the 

system and equipment under test and any records of deviations to the test plan along with the 

rationale for performing the deviations. Pro V&V shall also utilize an internal bug tracking 

system to record and track all issues and/or discrepancies noted during the test campaign. 

4.1.1 Mapping of Requirements to Equipment Type and Features

This information shall be included in the EAC online matrix tool (VRT).

4.1.2 Rationale NA Requirements

All EAC 2015 VVSG 1.1 requirements, with the exceptions listed below, will be evaluated as 

part of this test campaign.

 Volume I, Section 7.6 (Use of Public Networks)

 Volume I, Section 7.7 (Wireless Communications)

 Volume I, Section 7.8 (Voter Verifiable Paper Audit Trail Requirements)

The rationale for not evaluating the eLect Quad Audit v.4.4.0 to the indicated sections is 

described in following table. Specific requirements that are excluded from this test campaign are 

identified in the EAC online matrix tool.

Table 4-1. Not Applicable Requirements

EAC 2015 VVSG Version 1.1

Volume I, Section Rationale for ‘Not Applicable’

7.6 This system does not support transmission over public 

networks.

7.7 No wireless technology is utilized in this system.

7.8 This system does not contain a Voter Verifiable Paper Audit 

Trail (VVPAT).

4.2 Hardware Configuration and Design

The eLect Quad Audit v.4.4.0 system operates in two environments/locations: the central location 

and the polling place. Each location type utilizes a distinct hardware set. Each location 

incorporates servers or laptops containing versions of eLect Quad Audit and associated election 

settings/data.

The following six devices run different functions within the voting system:

 Central Server

The Central Server at the central location is the primary system device and runs eLect 

Quad Audit (Voting) and eLect Administration. All official tabulation occurs on the 

Central Server. The Central Server is comprised of COTS hardware and is connected to 

COTS UPS devices for soft shutdown during power interruptions.
28

TP-EOC-2016-01.02

 Central Scan Imaging Server

The Central Scan Imaging Server is a COTS rack server attached to the Central Server 

and is used solely for software image processing of centrally scanned mail-in ballots.

 Master Poll Station Machine (MPSM)

MPSM laptops are located at the polling place and contain the same version of the 

election configuration, but are not configured with eLect Administration; therefore, they 

do not contain the full eLect Quad Audit voting system.

 Air-Gapped Encryption/Decryption Device

An air-Gapped COTS laptop running eLect Security Suite only on Ubuntu desktop is 

used to generate encryption key pairs during the pre-election setup stage and used to 

decrypt vote data during post-election processing

 eLect Administration Access Machine

The eLect Administration Access Machine is the COTS laptop that provides access to the 

eLect Administration and is the user interface for all election management functionality 

in the eLect Quad Audit voting system. 

 Local Network Switch

The Local Network Switch is a COTS network switch that provides Ethernet 

communications between the Central Server, Central Scan Imaging Server and the eLect 

Administration Access Machine.

4.3 Software System Functions

The eLect Quad Audit v.4.4.0 system includes the following proprietary software components:

 eLect Quad Audit 

The eLect Quad Audit voting system is the platform that provides election administrators 

with the tools to manage elections. 

 eLect Administration Dashboard

eLect Administration is the user interface for the Admin User. It contains various system 

modules for election creation and management. 

 Poll Station Management Console

The eLect Poll Station Management Console is the user interface that is accessed through 

the poll station EBM to the poll worker.

 eLect Central Scanning

eLect Central Scanning is incorporated within eLect Administration.

 eLect Quad Audit

eLect Quad Audit is the core software module of the voting system.

 eLect Security Suite

eLect Security Suite is a utility application that provides encryption and decryption 

services.
29

TP-EOC-2016-01.02

The eLect Quad Audit v.4.4.0 system also includes modified and unmodified third-party 

components, and external library items, as previously detailed in Table 3-1.

4.4 Test Case Design

Test cases are designed based on the manufacturer’s design specifications and the relevant 

technical requirements set forth by the VVSG. Test cases shall be examined based on the 

following aspects of the voting system:

 Hardware qualitative examination design

 Hardware environmental test case design

 Software module test case design and data

 Software functional test case design

 System level test case design

Test cases shall provide information regarding the sequence of actions to be performed for the 

execution of a test, the requirements being met, the test objective, test configuration, equipment 

needed, special requirements, assumptions, and pass/fail criteria. Once the test cases are finalized, 

they will be validated and published for use in the test campaign. The validation of the test case 

will be accomplished by Technical Review and Approval. This validation will include the 

following: confirmation of adequate test coverage of all requirements; confirmation that test case 

results are not ambiguous and gave objective pass/fail criteria; and conformation that any 

automated test suites will produce valid results.

4.4.1 Hardware Qualitative Examination Design

The voting system hardware with the exception of COTS Central Count shall be subjected to the 

tests specified below. Testing will be performed by personnel verified by Pro V&V to be 

qualified to perform the test. Pro V&V will utilize a third-party test facility for performance of 

the majority of the electrical and environmental tests. All pre/post-tests shall be conducted by Pro 

V&V personnel.

Electrical Tests:

 Electrical Power Disturbance 

 Electromagnetic Emissions (Radiated & Conducted)

 Electrostatic Disruption 

 Electromagnetic Susceptibility 

 Electrical Fast Transient 

 Lightning Surge 

 Conducted RF Immunity 

 Magnetic Fields Immunity 

 Electrical Supply
30

TP-EOC-2016-01.02

Environmental Tests:

 Bench Handling 

 Vibration 

 Low Temperature Non-Operational

 High Temperature Non-Operational

 Low Temp Operational

 High Temp Operational

 Humidity Non-Operational

 Humidity Operational

 Acoustic 

4.4.1.1 Mapping of Requirements to Specific Interfaces

This information shall be contained in the EAC online matrix tool.

4.4.2 Hardware Environmental Test Case Design

No previous examinations have been performed on the eLect Quad Audit v.4.4.0. The voting 

system hardware shall be subjected to the tests specified in section 4.4.1. Testing will be 

performed by personnel verified by Pro V&V to be qualified to perform the test. Pro V&V will 

utilize a third-party test facility for performance of the electrical and environmental tests. All 

pre/post-tests shall be conducted by Pro V&V personnel. 

4.4.3 Software Module Test Case Design and Data

Pro V&V shall review the manufacturer’s program analysis, documentation, and module test case 

design and shall evaluated the test cases for each module with respect to flow control parameters 

and entry/exit data. As needed, Pro V&V shall design additional test cases to satisfy the coverage 

criteria specified in Volume II, Section 7.2.1. 

Component Level Testing will be implemented during the FCA for each component and 

subcomponent. During the Source Code Review, Compliance Builds, and Security Testing, Pro 

V&V will utilize limited structural-based techniques (white-box testing). Additionally, 

specification-based techniques (black-box testing) will be utilized for the individual software 

components.

Pro V&V shall define the expected result for each test and the ACCEPT/REJECT criteria for 

certification. If the system performs as expected, the results will be accepted. If the system does 

not perform as expected, an analysis will be performed to determine the cause. The test will be 

repeated in an attempt to reproduce the results. If the failure can be reproduced and the expected 

results are not met, the system will have failed the test. If the results cannot be reproduced, the 

test will continue. All errors encountered will be documented and tracked through resolution. 
31

TP-EOC-2016-01.02

4.4.4 Software Functional Test Case Design and Data

Pro V&V shall review the manufacturer-submitted test plans and data to verify that the individual 

performance requirements specified in the EAC 2015 VVSG 1.1 and the TDP are reflected in the 

software. As part of this process, Pro V&V shall review the manufacturer’s test case design and 

prepare a detailed matrix of system functions and the test cases that exercise them. Pro V&V shall 

also prepare a test procedure describing all test ballots, operator procedures, and the data content 

of output reports. Pro V&V shall define abnormal input data and operator actions and design test 

cases to verify that the system is able to handle and recover from these abnormal conditions.

During this review, emphasis shall be placed on those functions where the manufacturer data on 

module development reflects significant debugging problems, and on functional tests that resulted 

in high error rates. 

Pro V&V shall define the expected result for each test and the ACCEPT/REJECT criteria for 

certification. If the system performs as expected, the results will be accepted. If the system does 

not perform as expected, an analysis will be performed to determine the cause. The test will be 

repeated in an attempt to reproduce the results. If the failure can be reproduced and the expected 

results are not met, the system will have failed the test. If the results cannot be reproduced, the 

test will continue. All errors encountered will be documented and tracked through resolution.

4.4.5 System-Level Test Case Design

System Level testing will be implemented to evaluate the complete system. This testing will 

include all proprietary components (software, hardware, and peripherals) and COTS components 

(software, hardware, and peripherals) in a configuration of the system’s intended use. For 

software system tests, the tests shall be deigned according to the stated design objective without 

consideration of its functional specification. The system level hardware and software test cases 

shall be prepared independently to assess the response of the hardware and software to a range of 

conditions. These tests include: Volume, Stress, Usability, Accessibility, Security, Performance, 

and Recovery.

4.5 Security Functions

The objective of the Security Testing is to evaluate the effectiveness of the voting system in 

detecting, preventing, recording, reporting, and recovering from security threats. To evaluate the 

integrity of the system, Pro V&V shall develop specifically designed test cases in an attempt to 

defeat the access controls and security measures documented in the system TDP. A threat matrix 

shall be created to determine the risks and vulnerabilities. An evaluation of the system shall be 

accomplished by utilizing a combination of functional testing, source code review, and Fortify 

SCA. All findings will be reported to the EAC and Everyone Counts.

 

4.6 TDP Evaluation

In order to determine full compliance with the EAC 2015 VVSG 1.1, three phases of TDP review 

shall be conducted: 
32

TP-EOC-2016-01.02

 Initial TDP Review: The first review is performed to determine whether the TDP submitted is 

complete enough to perform TDP review. This is an abbreviated review. Each document is 

read to determine whether it provides enough description of the submitted voting system 

components and whether it at least generically addresses VVSG requirements. The results of 

the review are used in determining contractual requirements for the test campaign. 

 Compliance Review: This review is conducted on a document-by-document basis to 

determine if every regulatory, state, or manufacturer-stated requirement has been met based 

on the context of each requirement. This review does not address consistency or 

completeness of documents. The review is more complex than the initial TDP review. 

Results of the review of each document are entered on the TDP Review Checklist and are 

reported to the manufacturer for disposition of any anomalies. This process is ongoing until 

all anomalies are resolved. Any revised documents during the TDP review process are 

compared with the previous document revision to determine changes made, and the document 

is re-reviewed to determine whether subject requirements have been met.

 Consistency/Completeness Review: The third TDP review is completed after the review for 

compliance has been performed (the Consistency/Completeness TDP Review may overlap 

the Compliance Review in part). This review is to ensure the information included in the TDP 

documents is consistent across documents, especially in component naming, software and 

firmware versioning, and the hardware, software, and firmware included with the voting 

system submitted for testing. Any revisions to a document during the TDP review process 

which affect other documents must also be revised. As with the other TDP reviews, the TDP 

Review Checklist is utilized to report any anomalies to the manufacturer for resolution, if 

required. The TDP review continues until all anomalies have been satisfactorily resolved.

A listing of all documents contained in the eLect Quad Audit v.4.4.0 TDP is provided in Table 4-

2.

Table 4-2. eLect Quad Audit v.4.4.0 TDP Documents

Title Document Rev.

DOC-1101 Quad Audit 4.4.0 Configuration Items Register v4.4.0, r1.8

ST-1202 System Setup Overview v.4.4.0, r1.1

ST-1205 eLect Administration Access Machine Setup v.4.4.0, r1.0

ST-1206-OPS Dell Venue 11 EBM Setup Instructions v.4.4.0, r1.1

ST-1206-DRE Dell Venue 11 EBM Setup Instructions v.4.4.0, r1.0

ST-1207-OPS Ballot Box Setup v.4.4.0, r1.0

ST-1208 Building the Imaging and Transfer Service Files v.4.4.0, r1.0

ST-1209 Building an Imaging Server on Supermicro v.4.4.0, r1.0

ST-1210 Installing Ubuntu v.4.4.0, r1.0

ST-1212 Building an Ubuntu Linux ISO v.4.4.0, r1.0

ST-1214 Deploying the Air-Gapped Machine with eLect Security Suite v.4.4.0, r1.0

ST-1215 Deploying the Master Poll Station Machine v.4.4.0, r1.0

ST-1216 Deploying the Central Server v.4.4.0, r1.0
33

TP-EOC-2016-01.02

Table 4-2. eLect Quad Audit v.4.4.0 TDP Documents (continued)

Title Document Rev.

OP-1312 System Operations v.4.4.0, r1.0

UG-1400 User Guide: Central Office Hardware Setup v.4.4.0, r1.0

UG-1402 User Guide: Using the Correct Documentation Set v.4.4.0, r1.0

UG-1405 User Guide: Pre-Voting eLect Administration v.4.4.0, r1.0

UG-1406 User Guide: Appendix to User Guide: Pre-Voting eLect

Administration v.4.4.0, r1.0

UG-1410 User Guide: Pre-Voting Central Scan Configuration v.4.4.0, r1.0

UG-1415-DRE User Guide: Poll Station Management v.4.4.0, r1.0

UG-1415-OPS User Guide: Poll Station Management v.4.4.0, r1.0

UG-1420 User Guide: Central Scan Ballot Processing v.4.4.0, r1.0

UG-1425-DRE User Guide: Voter Manual v.4.4.0, r1.0

UG-1425-OPS User Guide: Voter Manual v.4.4.0, r1.0

UG-1430-DRE User Guide: Post-Voting Poll Station Management v.4.4.0, r1.0

UG-1430-OPS User Guide: Post-Voting Poll Station Management v.4.4.0, r1.0

UG-1435 User Guide: Post-Voting eLect Administration v.4.4.0, r1.0

UG-1440 User Guide: Quick Reference for System Reports v.4.4.0, r1.0

UG-1445 User Guide: Accessing Audit Events v.4.4.0, r1.0

UG-1450 User Guide: Backing Up and Restoring MPSM Data and the

Election Server v.4.4.0, r1.0

RF-1500 eLect Security Manual v.4.4.0, r1.0

RF-1505 Consumables v.4.4.0, r1.0

RF-1510 System Maintenance Manual v.4.4.0, r1.0

RF-1511 System Personnel and Training Manual v.4.4.0, r1.0

RF-1515 COTS Manuals v.4.4.0

RF-1520 Environmental Data Sheets v.4.4.0

QA-1605 Quality Manual v.4.4.0, r1.0

QA-1610 Configuration Management Plan v.4.4.0, r1.1

QA-1615 System Test Specification v.4.4.0, r1.0

QA-1616 System Test Reports v.4.4.0, r1.0

DS-1705 Implementation Statement v.4.4.0, r1.0

DS-1705a Implementation Statement Supplemental: Accuracy Attestation v.4.4.0, r1.0

DS-1710 System Overview v.4.4.0, r1.3

DS-1715 System Functionality Description v.4.4.0, r1.0
34

TP-EOC-2016-01.02

Table 4-2. eLect Quad Audit v.4.4.0 TDP Documents (continued)

Title Document Rev.

DS-1720 Usability Testing Report v.4.4.0, r1.0

SP-1805 System Hardware Specifications v.4.4.0, r1.1

SP-1806 System Hardware Supplement v.4.4.0, r1.0

SP-1810 Software Design Specifications v.4.4.0, r1.0

SP-1811 Software Design Specifications: General Appendix A v.4.4.0, r1.0

SP-1811 Software Design Specifications Appendix B: Interface 

Description v.4.4.0, r1.0

SP-1811 Software Design Specifications Appendix C : Database Entities v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix D: Introduction v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix E: eLect Security Suite v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix F: ASYNC v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix G: Interface Audit Log

Service v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix H: Interface Bridge 

Web Service v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix I: Interface 

Configuration Service v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix J: Ballot Builder 

Module v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix K: Administrative Web

Service v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix L: Batch Manager v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix M: Notification 

Services v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix N: Storage Web 

Service v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix O: Interface User

Manager v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix P: Core: General v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix Q: Core: System v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix R: Core: 

Authentication v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix S: Core: Election

Content v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix T: Core - Vote Data v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix U: Management 

Election v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix V: Voting: Ballot

Storage And Backup v.4.4.0, r1.0
35

TP-EOC-2016-01.02

Table 4-2. eLect Quad Audit v.4.4.0 TDP Documents (continued)

Title Document Rev.

SP-1812 Software Design Specifications Appendix W: Results Auditing 

and Logging v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix X: Results Tabulation v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix Y: Imaging and 

Transfer Service Program Specification Part 1 of 2 v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix Y: Imaging and 

Transfer Service Program Specification Part 2 of 2 v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix Z Management: Poll

Station v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AA: Voting Session

Management v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AB: Voting: Ballot

Rendering And Navigation v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AC: Voting 

Accessibility v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AD: Javascript Code v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AE: Perl Code v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AF: Voting: Ballot 

Box (Optical scan only) v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AG: Voting Ballot

Printing (opscan only) v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AH - Voting - EBM

Barcode Scanner v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AI: eLect 

Administration JavaScript Interfaces v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AJ: eLect 

Administration Architecture Overview v.4.4.0, r1.0

SP-1812 Software Design Specifications Appendix AK: eLect

Administration Groovy Interfaces v.4.4.0, r1.0

SP-1813 Third Party Software License Agreements v.4.4.0

SP-1815 System Security Specifications v.4.4.0, r1.0

4.7 Source Code Review

Pro V&V will review the submitted source code to the EAC 2015 VVSG 1.1 and the 

manufacturer-submitted coding standards. Prior to initiating the software review, Pro V&V shall 

verify that the submitted documentation is sufficient to enable: (1) a review of the source code 

and (2) Pro V&V to design and conduct tests at every level of the software structure to verify that 

design specifications and performance guidelines are met. 
36

TP-EOC-2016-01.02

4.8 QA & CM System Review

The Everyone Counts Quality and Configuration Management Manual shall be reviewed for its 

fulfillment of Volume I, Section 8.1.a and the requirements specified in Volume II, Section 2.1. 

The requirements for these sections establish the quality assurance and configuration standards 

for voting systems to which manufacturers must conform. The requirements also require voting 

system manufacturers to implement a quality assurance and configuration management program 

that is conformant with recognized ISO standards.

As part of the review process, the Everyone Counts TDP documents will be reviewed to 

determine if the stated policies are being followed. 

5.0 Test Data

The following subsections provide information concerning test data recording, criteria, and 

reduction.

5.1 Test Data Recording

All equipment utilized for test data recording shall be identified in the test data package. The 

output test data shall be recorded in an appropriate manner as to allow for data analysis. For 

source code and TDP reviews, results shall be compiled in reports and submitted to Everyone 

Counts for resolution.

5.2 Test Data Criteria

The eLect Quad Audit v.4.4.0 shall be evaluated against all applicable requirements contained in 

the EAC 2015 VVSG 1.1. The acceptable range for system performance and the expected results 

for each test case shall be derived from the manufacturer-submitted technical documentation and 

the EAC 2015 VVSG 1.1. 

5.3 Test Data Reduction

Test data shall be processed and recorded in the test log book and the relevant Test Cases. 

6.0 Test Procedure and Conditions

The following subsections detail the facility requirements, test setup conditions, and sequence of 

testing.

6.1 Facility Requirements

Unless otherwise annotated, all testing shall be conducted at the Pro V&V test facility located in 

Huntsville, AL, by personnel verified by Pro V&V to be qualified to perform the test. Unless 

otherwise specified herein, testing shall be performed at the following standard ambient 

conditions and tolerances:
37

TP-EOC-2016-01.02

 Temperature: 68-75◦ F (±3.6◦F)

 Relative Humidity: Local Site Humidity

 Atmospheric Pressure: Local Site Pressure

 Time Allowable Tolerance: ±5%

Testing performed at third-party laboratories will be subjected to the test parameters and 

tolerances defined by the test facility and will be reported in the final Test Report. 

6.2 Test Setup

All voting system equipment shall be received and documented using Pro V&V proper QA 

procedures. Upon receipt of all hardware, an inspection will be performed to verify that the 

equipment received is free from obvious signs of damage and/or degradation that may have 

occurred during transit. If present, this damage shall be recorded, photographed, and reported to 

the Everyone Counts Representative. Additionally, a comparison shall be made between the 

recorded serial numbers/part numbers and those listed on shipper’s manifest and any 

discrepancies shall be reported to the Everyone Counts Representative. TDP items and all source 

code received shall be inventoried and maintained by Pro V&V during the test campaign.

During test performance, the system shall be configured as would be for normal field use. This 

includes connecting all supporting equipment and peripherals. 

6.3 Test Sequence

The eLect Quad Audit v.4.4.0 will be evaluated against all applicable requirements in the EAC 

2015 VVSG 1.1. There is no required sequence for test performance.

7.0 TEST OPERATIONS PROCEDURES

Pro V&V will identify PASS/FAIL criteria for each executed test case. The PASS/FAIL criteria 

will be based on the specific expected results of the system. In the case of an unexpected result 

that deviates from what is considered standard, normal, or expected, a root cause analysis will be 

performed. 

Pro V&V will evaluate every EAC 2015 VVSG 1.1 requirement applicable to the eLect Quad 

Audit v.4.4.0 voting system. Any deficiencies noted will be reported to the EAC and the 

manufacturer. If it is determined that there is insufficient data to determine compliance, this test 

plan will be altered and additional testing will be performed.

7.1 Proprietary Data

All data and documentation considered by the manufacturer to be proprietary will be identified 

and documented in an independent submission along with a Notice of Protected Information. 
A 1

TP-EOC-2016-01.01

Attachment A – Project Schedule
A

2

TP-EOC-2016-01.01
B 1

TP-EOC-2016-01.01

Attachment B – Implementation Statement
B 

2

TP-EOC-2016-01.01
B 

3

TP-EOC-2016-01.01
B 

4

TP-EOC-2016-01.01
B 

5

TP-EOC-2016-01.01
B 

6

TP-EOC-2016-01.01
B 

7

TP-EOC-2016-01.01
B 

8

TP-EOC-2016-01.01
B 

9

TP-EOC-2016-01.01
B 10

TP-EOC-2016-01.01
B 11

TP-EOC-2016-01.01
B 12

TP-EOC-2016-01.01
B 13

TP-EOC-2016-01.01
B 14

TP-EOC-2016-01.01
B 15

TP-EOC-2016-01.01
B 16

TP-EOC-2016-01.01
B 17

TP-EOC-2016-01.01
B 18

TP-EOC-2016-01.01
B 19

TP-EOC-2016-01.01
B 20

TP-EOC-2016-01.01
B 21

TP-EOC-2016-01.01
B 22

TP-EOC-2016-01.01
B 23

TP-EOC-2016-01.01
B 24

TP-EOC-2016-01.01
B 25

TP-EOC-2016-01.01
B 26

TP-EOC-2016-01.01
B 27

TP-EOC-2016-01.01
B 28

TP-EOC-2016-01.01
B 29

TP-EOC-2016-01.01