U. S. ELECTION ASSISTANCE COMMISSION 

VOTING SYSTEM TESTING AND CERTIFICATION PROGRAM

1225 New York Avenue, NW, Suite 1100 

Washington, DC. 20005 

2/12/2009 

Carolyn Coggins, QA Director - Voting 

iBeta Quality Assurance 

3131 S Vaughn Way, Suite 650 

Aurora, CO 80014 

Re: Reuse of prior testing conducted by SysTest Laboratories

Dear Ms. Coggins, 

This letter is regarding the reuse of certain portions of testing conducted by SysTest 

Laboratories as part of the testing campaign for the ES&S Unity 3.2.0.0. As the EAC 

indicated in its January 3, 2009 letter (attached) to ES&S and consistent with section 

2.10.6.of the EAC’s Testing and Certification Program Manual, the EAC has conducted 

a review and audit of certain portions of testing conducted by SysTest prior to SysTest’s 

suspension. This review and audit was conducted in an effort to preserve any prior 

testing that could be relied upon as meeting the EAC’s rigorous program requirements. 

Based on this review and audit the EAC approves the following reuse of functional

accessibility, maintainability, accuracy, and reliability testing conducted for the Unity 

3.2.0.0. test campaign: 

Functional, Accessibility, Maintainability, Accuracy, and Reliability Test Review

In order to determine the scope of the possible reuse of the functional, accessibility, 

maintainability, accuracy, and reliability testing conducted for the Unity 3.2.0.0. EAC 

technical reviewers conducted a full audit of all test plans, test methods, test cases, and 

test results related to the Unity 3.2.0.0. test campaign. This included a review of a 

document created by SysTest Labs that summarized all related testing conducted to date 

for the Unity 3.2.0.0. test campaign and provided the test results for that testing. As a 

result of this review and audit the EAC reached the following conclusions regarding the 

reuse of this testing: 

 All functional, accessibility, maintainability, accuracy, and reliability testing 

outlined in the approved SysTest Unity 4.0 test plan is approved for reuse in the 

Unity 3.2.0.0 test campaign. 
 As part of the remaining testing the EAC is tasking iBeta with testing and 

verifying that the Unity 3.2.0.0 system is in compliance with EAC RFI 2008-07 “ 

0’ count to start the election”. This testing should be reflected in the test plan 

being developed by iBeta for the Unity 3.2.0.0 system. 

 iBeta is also tasked with testing and resolving the discrepancies listed by SysTest 

under the following tests: 

 GEN 02 – Straight Party

 GEN 03 – Add Languages

 PR101 – Pick-a-Party tests

Please let me know if you have any questions regarding this decision regarding 

reuse of testing or the Unity 3.2.0.0 test campaign. 

Sincerely, 

Brian Hancock 

Director, Testing and Certification 

US Election Assistance Commission 

attachment: 1.3.2009 ltr. From EAC to ES&S 

cc: Election Systems and Software (ES&S) 
 U. S. ELECTION ASSISTANCE COMMISSION VOTING SYSTEM TESTING AND CERTIFICATION PROGRAM

 1225 New York Avenue, NW, Suite 1100 

 Washington, DC. 20005 

February 3, 2009 

Ms. Sue L. Munguia 

Director of Certification 

Election Systems and Software (ES&S) Sent via e-mail

11208 John Galt Boulevard 

Omaha, NE 68137 

Ms. Munguia: 

This letter is being sent to address ES&S’s questions regarding the reuse of testing by iBeta 

Quality Assurance (iBeta) that was conducted by SysTest Laboratories (SysTest) on the Unity 

3.2.0.0. voting system prior to the suspension of SysTest’s accreditation as an EAC Voting 

System Test Laboratory (VSTL). As you are aware, and as indicated in our letter to all EAC 

registered manufacturers (attached), section 2.10.6. of the EAC’s Voting System Test Laboratory 

Program Manual allows for the reuse of prior testing at the EAC’s discretion. 

The EAC recognizes the unique circumstances the SysTest suspension has created and the need 

for the EAC to be diligent in reviewing and deciding on the reuse of this testing. Given these 

circumstances the EAC has corresponded on several occasions with both iBeta and ES&S to 

discuss the testing that was conducted by SysTest and its possible reuse. Per an email from Brian 

Hancock, Director, EAC Testing and Certification Program to Steve Pearson of ES&S dated 

November 21, 2008 (attached) the following plan is approved for the review of prior testing 

conducted by SysTest for possible reuse by iBeta in the testing of the Unity 3.2.0.0. voting 

system: 

 The EAC has authorized the reuse of the hardware testing conducted by SysTest subcontractors. iBeta will review the reports to confirm any failures resulting in engineering 

changes are documented and the reports document that all hardware ultimately passed. 

 iBeta will audit a sample of the Technical Data Package (TDP) submitted to and 

reviewed by SysTest and provide a recommendation to the EAC regarding the need to 

conduct a more comprehensive review of the TDP. The EAC shall issue a decision 

regarding reuse of the PCA Document Review. 

 iBeta will conduct a 3% review of the ES&S source code. This review will focus on 

important functional sections of the code in order to determine the depth and focus of 

source review conducted by SysTest. iBeta will provide a recommendation to the EAC 
regarding the reuse of the source code review conducted by SysTest. The EAC will then 

issue a decision regarding the reuse of the source code review conducted by SysTest. 

 The EAC Technical Reviewers will review and assess the Functional, Accessibility, 

Maintainability, Accuracy, and Reliability test summary reports provided by SysTest on 

the DS-200, M650, AutoMARK VATs, Ballot on Demand printer, and Unity EMS 

software. The EAC will issue a decision regarding the reuse of this testing. 

 The Volume, Stress, Error Recovery and Security testing was not completed by SysTest; 

iBeta will perform this testing on the DS-200, M650, AutoMARK VATs, and Unity EMS 

software. 

 While applicable areas from the Unity v.4.0.0.0 Test Plan may be used, iBeta must issue 

a Unity v.3.2.0.0 test plan. The EAC will review and approve a full test plan provided by 

iBeta. 

 SysTest shall provide the appropriate test summaries for all items that are accepted for 

reuse. 

If you should have any questions regarding these decisions or the testing to be conducted at iBeta 

please do not hesitate to contact me. Thank you for your time in resolving these issues. 

Sincerely, 

Brian J. Hancock 

Director, Testing and Certification 

Attachment: 10.29.08 EAC ltr. to all EAC registered manufactures 

 11.21.08 email from Brian Hancock to Steve Pearson 

cc: iBeta Quality Assurance 

 Steve Pearson, Election Systems and Software 
 U. S. ELECTION ASSISTANCE COMMISSION VOTING SYSTEM TESTING AND CERTIFICATION PROGRAM

 1225 New York Avenue, NW, Suite 1100 

 Washington, DC. 20005 

October 29, 2008 

To: Registered Voting System Manufacturers 

From: Brian Hancock, Director 

 United States Election Assistance Commission 

 Testing and Certification Program 

RE: EAC Issuance of Notice of Intent to Suspend SysTest, Laboratories Inc. 

The U.S. Election Assistance Commission today notified (attached) SyStest Laboratories 

Inc. of its intent to suspend the laboratory based upon the suspension of its accreditation 

by NIST/NVLAP. 

As a result of the notice, SysTest has three days to respond to EAC’s action. If SysTest 

cannot refute the fact that NVLAP has suspended the laboratory the EAC will suspend 

SysTest and all testing under the EAC’s program must be halted immediately. 

Those manufacturers currently using SysTest as their lead VSTL for testing under the 

EAC’s program should be aware of their options as provided for in the EAC’s Voting 

System Testing and Certification Program Manual and the Voting System Test 

Laboratory Program Manual. Per section 4.3.1.2. of the EAC’s Testing and 

Certification Program Manual, the EAC Program Director may, at his discretion, allow a 

manufacturer to change VSTL’s provided the manufacturer shows good cause for the 

change. A manufacturer may request to change its VSTL by providing in writing: 

1. A statement indicating the current VSTL conducting testing of their voting 

system. 

2. The reasoning for the request to change VSTL (good cause). 

3. A statement indicating the new VSTL the manufacturer wishes to test the voting 

system. 

4. A proposed amended Voting System Certification Application reflecting the 

proposed VSTL change. 
Upon receipt of this information, the EAC Program Director will issue written notice to 

the manufacturer regarding the proposed change of VSTL. Upon receipt of expressed 

written permission from the Program Director to change VSTLs the manufacturer may 

begin testing at the new VSTL in conformance with the EAC’s program requirements. 

Manufacturers may also choose to halt testing until such time as SysTest Laboratories 

may become eligible to recommence testing of their voting system. Please be aware that 

SysTest MAY NOT recommence testing until such time as the EAC provides written 

notice to SysTest of their ability to begin testing again under the EAC’s program. 

Many of you may have questions regarding the testing already conducted by SysTest and 

its use by a new VSTL. Per section 2.10.6. of the EAC’s Voting System Test Laboratory 

Program Manual a VSTL may accept prior testing conducted by another VSTL or third 

party laboratory provided certain conditions are met. These conditions are: 

1. The discrete hardware or software component previously tested is demonstrably 

identical to that presently offered for testing. 

2. The voting system standards and relevant EAC interpretations applicable to the 

prior and current testing are identical. 

3. The test methods used are equivalent or identical to current test methods approved 

by the EAC. 

4. The prior testing has been reviewed by the VSTL and no errors or omissions are 

apparent. 

5. The adoption and use of prior testing is noted in the test plan and test report. 

Please be aware that the lead VSTL is responsible for ensuring that the prior testing has 

met these requirements. Like all testing under the EAC’s program, all prior testing 

remains subject to EAC technical review and approval. 

If you have any questions regarding the possible suspension of SysTest Labs, the process 

for the changing of a lead VSTL, or the process for approval of prior testing please do not 

hesitate to contact myself or my staff. 

Sincerely, 

Brian J. Hancock 

Director, Testing and Certification 

 
From: bhancock@eac.gov [mailto:bhancock@eac.gov] 

Sent: Friday, November 21, 2008 8:59 AM

To: smpearson@essvote.com; slmunguia@essvote.com

Cc: Carolyn Coggins; Gail Audette; lotero@eac.gov; mmasterson@eac.gov; 

stephen.berger@suddenlink.net; tapernew@earthlink.net; dawn.mehlhaff@sbcglobal.net; 

txtwatson@yahoo.com; tcaddy@sbcglobal.net

Subject: Reuse of Previous Testing for Unity 3.2.0.0 

 

Steve, 

As promised, this email responds to the questions posed by ES&S on the November 6th telecon, 

and the 11/19 follow-up email regarding reuse of testing done by SysTest. 

 The EAC recognizes the unique circumstances the SysTest suspension has created and the 

need for the EAC to be diligent in reviewing and deciding on the reuse of this testing. As a 

result of this discussions with both iBeta and the EAC’s Technical Reviewers the EAC has 

reached the following decisions regarding the reuse of prior testing by SysTest : 

• All hardware testing conducted by SysTest sub-contractor Percept Technology Labs is 

accepted and may be reused. Testing done by Wyle Labs is, of course, also accepted. 

• iBeta will conduct an audit of the Technical Data Package (TDP) submitted to and 

reviewed by SysTest and make a recommendation to the EAC regarding the need to 

conduct a more comprehensive review of the TDP. Our hope and expectation is that this 

TDP audit will find that the prior review was valid, but given the circumstances 

surrounding the SysTest suspension, our need for assurance that the work was done 

correctly, and the need for iBeta to feel confident in accepting the results of the TDP 

review, we feel that the audit is necessary. 

• iBeta will conduct a 3% review of the ES&S source code. This review will focus on 

important functional sections of the code in order to determine the depth and focus of 

source review conducted by SysTest. iBeta will then make a recommendation to the 

EAC regarding the reuse of the source code review conducted by SysTest. The EAC will 

then issue a decision regarding the reuse of the source code review conducted by 

SysTest. 

• Functional Test Cases: It is our understanding that iBeta has been provided the test 

cases for review. iBeta will review the test cases, modify as they see a weakness or 

need, and present their test cases to EAC for review. Obviously we worked long and 

hard with SysTest on these test cases, so again, our hope is that the majority of the 

cases can be reused. • Unity 3.2 Test Plan: Although the Test Plan for Unity 4.0 has now been approved, the 

EAC will not automatically accept and approve a Unity 3.2 Test Plan by default. While 

applicable areas from the Unity 4.0 Test Plan may be used, the EAC will review and 

approve a full test plan provided by iBeta. 

We can discuss these and any other issues you may have on our conference call later today. 

Thanks. 

Brian Hancock 

Director of Voting System Testing & Certification 

U.S. Election Assistance Commission 

1225 New York Avenue, NW, Ste. 1100 

Washington, DC 20005 

202-566-3100 

www.eac.gov