U. S. ELECTION ASSISTANCE COMMISSION 

VOTING SYSTEM TESTING AND CERTIFICATION PROGRAM

1225 New York Avenue, NW, Suite 1100 

Washington, DC. 20005 

2/3/2009 

Carolyn Coggins, QA Director - Voting 

iBeta Quality Assurance 

3131 S Vaughn Way, Suite 650 

Aurora, CO 80014 

Re: Reuse of prior testing conducted by SysTest Laboratories

Dear Ms. Coggins, 

This letter is in response to iBeta Quality Assurance’s recommendations (attached) to the 

Election Assistance Commission (EAC) regarding the reuse of certain portions of testing 

conducted by SysTest Laboratories as part of the testing campaign for the ES&S Unity 

3.2.0.0. EAC has instructed (attached) iBeta, per section 2.10.6.of the EAC’s Testing and 

Certification Program Manual, to conduct a review and audit of certain portions of 

testing conducted by SysTest prior to SysTest’s suspension. This review and audit was 

conducted in an effort to preserve any prior testing that could be relied upon as meeting 

the EAC’s rigorous program requirements. 

Based on iBeta’s recommendations the EAC approves the following reuse of prior 

testing: 

PCA Document Review

In its letter to the EAC dated January 14, 2009 regarding the reuse of the PCA document 

review iBeta recommends, “Based upon the audit and review documented herein iBeta 

deems that the results of the SysTest PCA Document Review are adequate for reuse in 

the Unity v.3.2.0.0 Certification test effort”. iBeta additionally recommends further 

review of the TDP for the Volume, Stress, Error Recover and Security testing still to be 

conducted on the voting system. 

After careful review of iBeta’s recommendation as well as the documentation submitted 

by SysTest for the ES&S Unity 3.2.0.0.voting system, the EAC approves iBeta’s 

recommendation for the reuse of the PCA document review conducted by SysTest. In 

approving this prior review and testing the EAC expects iBeta to document any further 

review conducted on the TDP and to clearly identify any discrepancies found during this 

additional review. 
Source Code Review 

In its letter dated January 16, 2009 to the EAC regarding reuse of the source code review 

conducted by SysTest on the Unity 4.0. voting system for the Unity 3.2.0.0. testing 

engagement iBeta recommended reuse of the SysTest source code review. Before 

making this recommendation, and per the EAC’s instruction, iBeta conducted a 3% audit 

of the Unity 3.2.0.0. source code. As a result of this audit and the existence of possible 

discrepancies that could impact the source code, iBeta conducted two additional analyses 

of the source code: 

1. Confirmed that the results of the iBeta review of the 3% of code are 

consistent with the previous results (not identical but consistent): This 

confirmation was reached by reviewing the types of discrepancies 

generated by SysTest in the 100% review against those generated by 

iBeta. 

2. Reviewed the severity of the discrepancies discovered: The number of 

discrepancies potentially impacting the source code is considered very 

low versus the overall number of discrepancies (as is consistent with a 

100% review). The severity of the discrepancies and the vendor 

responses do indicate that the majority of those 21 potential logic 

discrepancies would be resolved without source code modifications. 

In addition to the audit and reviews conducted by iBeta the EAC Technical Reviewers 

have conducted a review of the recommendation made by iBeta and the audit and reviews 

conducted. Based on iBeta’s audit and reviews and this additional technical review the 

EAC approves iBeta’s recommendation for the reuse of the source code review 

previously conducted by SysTest. 

If you should have any questions regarding this approval or the impact it has on the Unity 

3.2.0.0. testing engagement please do not hesitate to contact me. Thank you. 

Sincerely, 

Brian Hancock 

Director, Testing and Certification 

US Election Assistance Commission 
 U. S. ELECTION ASSISTANCE COMMISSION VOTING SYSTEM TESTING AND CERTIFICATION PROGRAM

 1225 New York Avenue, NW, Suite 1100 

 Washington, DC. 20005 

February 3, 2009 

Ms. Sue L. Munguia 

Director of Certification 

Election Systems and Software (ES&S) Sent via e-mail

11208 John Galt Boulevard 

Omaha, NE 68137 

Ms. Munguia: 

This letter is being sent to address ES&S’s questions regarding the reuse of testing by iBeta 

Quality Assurance (iBeta) that was conducted by SysTest Laboratories (SysTest) on the Unity 

3.2.0.0. voting system prior to the suspension of SysTest’s accreditation as an EAC Voting 

System Test Laboratory (VSTL). As you are aware, and as indicated in our letter to all EAC 

registered manufacturers (attached), section 2.10.6. of the EAC’s Voting System Test Laboratory 

Program Manual allows for the reuse of prior testing at the EAC’s discretion. 

The EAC recognizes the unique circumstances the SysTest suspension has created and the need 

for the EAC to be diligent in reviewing and deciding on the reuse of this testing. Given these 

circumstances the EAC has corresponded on several occasions with both iBeta and ES&S to 

discuss the testing that was conducted by SysTest and its possible reuse. Per an email from Brian 

Hancock, Director, EAC Testing and Certification Program to Steve Pearson of ES&S dated 

November 21, 2008 (attached) the following plan is approved for the review of prior testing 

conducted by SysTest for possible reuse by iBeta in the testing of the Unity 3.2.0.0. voting 

system: 

 The EAC has authorized the reuse of the hardware testing conducted by SysTest subcontractors. iBeta will review the reports to confirm any failures resulting in engineering 

changes are documented and the reports document that all hardware ultimately passed. 

 iBeta will audit a sample of the Technical Data Package (TDP) submitted to and 

reviewed by SysTest and provide a recommendation to the EAC regarding the need to 

conduct a more comprehensive review of the TDP. The EAC shall issue a decision 

regarding reuse of the PCA Document Review. 

 iBeta will conduct a 3% review of the ES&S source code. This review will focus on 

important functional sections of the code in order to determine the depth and focus of 

source review conducted by SysTest. iBeta will provide a recommendation to the EAC 
regarding the reuse of the source code review conducted by SysTest. The EAC will then 

issue a decision regarding the reuse of the source code review conducted by SysTest. 

 The EAC Technical Reviewers will review and assess the Functional, Accessibility, 

Maintainability, Accuracy, and Reliability test summary reports provided by SysTest on 

the DS-200, M650, AutoMARK VATs, Ballot on Demand printer, and Unity EMS 

software. The EAC will issue a decision regarding the reuse of this testing. 

 The Volume, Stress, Error Recovery and Security testing was not completed by SysTest; 

iBeta will perform this testing on the DS-200, M650, AutoMARK VATs, and Unity EMS 

software. 

 While applicable areas from the Unity v.4.0.0.0 Test Plan may be used, iBeta must issue 

a Unity v.3.2.0.0 test plan. The EAC will review and approve a full test plan provided by 

iBeta. 

 SysTest shall provide the appropriate test summaries for all items that are accepted for 

reuse. 

If you should have any questions regarding these decisions or the testing to be conducted at iBeta 

please do not hesitate to contact me. Thank you for your time in resolving these issues. 

Sincerely, 

Brian J. Hancock 

Director, Testing and Certification 

Attachment: 10.29.08 EAC ltr. to all EAC registered manufactures 

 11.21.08 email from Brian Hancock to Steve Pearson 

cc: iBeta Quality Assurance 

 Steve Pearson, Election Systems and Software 
3131 South Vaughn Way Suite 650 Aurora, CO 80014 Phone (303) 627-1110 Fax (303) 627-1233

January 14, 2009

Mr. Brian Hancock Sent via E-mail

U.S. Election Assistance Commission

Voting System Testing and Certification Program

1225 New York Avenue, NW, Suite 110

Washington, DC 20005

Dear Mr. Hancock, 

The purpose of this letter is to document the audit of the ES&S Unity v.3.2.0.0 Technical Data Package (TDP) in 

accordance with your November 21, 2008 email to Steve Pearson of ES&S providing instructions on the reuse 

of SysTest Labs' Unity v.4.0.0.0 testing for the ES&S Unity v.3.2.0.0 . EAC Certification test effort. iBeta has 

completed the audit of the Unity v.3.2.0.0 TDP. This letter provides iBeta's findings and opinion to the EAC 

regarding the reuse of the PCA Document Review conducted by SysTest Labs from the Unity v.4.0.0.0 test 

effort. 

Scope of the Unity v.3.2.0.0 EAC Certification Test Effort

The scope of Unity v.3.2.0.0 is a subset of Unity v.4.0.0.0 . It is restricted to the applications of the ES&S 

election management system, the DS200 and M650 optical scanners and the AutoMARK Voter Assistant 

Terminals (VAT) A100 and A200 ballot markers. It is our understanding that over the course of the Unity

v.4.0.0.0 test effort ES&S made changes to the scope of Unity v.4.0.0.0 . These scope changes included the 

incorporation of the VAT ballot markers and the AIMS application. These had previously been in the Unity 

3.0.1.0-3.0.1.1 EAC Certification test effort. Results of the PCA Document Review performed by SysTest Labs, 

including open discrepancies were delivered in: 

Unity v.4.0.0.0 PCA Document Review results for the ES&S election management system applications, 

the DS200and M650; and

Unity 3.0.1.0-3.0.1.1 PCA Document Review results for the VATs and AIMS.

ES&S delivered Unity v.3.2.0.0 TDP which consisted of:

The rebranded Unity v.4.0.0.0 TDP incorporating the "NOTICE OF UNCERTIFIED FUNCTIONALITY"

disclaimer; and

The AutoMARK VAT and AIMS TDP.

iBeta sampled the ES&S Unity v.3.2.0.0 documents. The sample selection included the documents identified in 

the SysTest Labs issued discrepancies and documents needed to complete the Unity v.3.2.0.0 trusted builds, a 

sample 3% source code review, test planning and test execution. Criteria for the review were to confirm that the 

Unity v.3.2.0.0 documents addressed any document discrepancies within the scope of the Unity v.3.2.0.0 test 

effort and the content provided sufficient information in order to complete the test tasks list above. 

Documentation of the Audit of the TDP 

Due to the change of scope many discrepancies issued by SysTest Labs were outside the scope of Unity

v.3.2.0.0 . iBeta reviewed every open discrepancy. Issues which were identified as all or partially relevant to 

the Unity v.3.2.0.0 scope were transferred to iBeta's Unity v.3.2.0.0 Discrepancy Report. Issues or parts of 

issues, outside this scope were excluded. Scope assessment was recorded in a review disposition document. 

The transferred discrepancies identified location of the issue, SysTest Labs discrepancy number, and detail of 

the initial description from the SysTest Labs discrepancy report. iBeta confirmed the issues were valid and 

traced to an appropriate 2002 VSS requirement. iBeta reviewed the SysTest Labs description history from the 

original SysTest Labs discrepancy report and the Unity v.3.2.0.0 documents submitted by ES&S to validate 
Page 2 of 4

resolution of the issue. In some instances discrepancies are being incorporated into Unity v.3.2.0.0 Functional 

Configuration Audit (FCA). 

The review of documents necessary to complete Unity v.3.2.0.0 trusted builds, sample code review, test 

planning and test execution was incorporated into these tasks and recorded in the daily status. Missing content 

or discrepancies were reported in the Unity v.3.2.0.0 Discrepancy Report. 

Review Results

The Unity v.3.2.0.0 TDP submitted by ES&S was sufficient to close the majority of the document discrepancies 

deemed inside the scope of Unity v.3.2.0.0. 

One document issue remains open for additional clarification of the ES&S response;

One document issue remains open for incorporation into the iBeta Security Review; and 

Four issues did not have a response from ES&S. As these were the last items on the list it is possible 

that they had not been submitted to ES&S by SysTest Labs. 

Review of documents necessary to perform Unity v.3.2.0.0 trusted builds, sample 3% code review and test 

planning has generally found them to contain the information needed to perform these task. Four issues were 

noted in the review. 

Document discrepancy #10 identified a gap in the Win650 build procedure;

Document discrepancy #50 identified the System Overview and System Limitations do not reflect the 

language scope of Unity v.3.2.0.0;

Document discrepancy #52 identified System Overview contained a typo with an incorrect hardware 

version for the DS200; and

Document discrepancy #53 identified the absence of the VATs and AIMS from the System Limitations.

The attached Tables 1 and 2 provide the results and disposition of all SysTest Labs Unity v.4.0.0.0 issued 

discrepancies. Note: Functional discrepancies which remain open for validation in the FCA are also listed in 

Table 1.

Recommendation on Reuse of the SysTest Labs PCA Document Review

Based upon the audit and review documented herein iBeta deems that the results of the SysTest PCA 

Document Review are adequate for reuse in the Unity v.3.2.0.0 Certification test effort. Use of the TDP in 

development of the Volume, Stress, Error Recovery and Security testing shall incorporate additional review. Any 

documentation issues encountered shall be reported in the Unity v.3.2.0.0 discrepancy report. We do not 

recommend a more comprehensive review of the TDP. 

Best regards,

Carolyn Coggins

QA Director - Voting

Attachments: Table 1 & 2
Draft v.0.1 Unity v.3.2.0.0 PCA Document Review Assessment - Attachment - Tables 1 & 2 Page 3 of 4

Table 1 Unity v.3.2.0.0 Applicable Discrepancies

SysTest# DS200 M650 VAT EMS Other 

Doc

iBeta# Disposition Part Excluded fm Unity v.3.2.0.0 Out of Scope Disposition

6 X X 12 Closed M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

23 X 13 Closed ABCR, Test Plan Not reviewed, remains open in Unity 

v.4.0.0.0 

24 X 14 Closed ABCR, Test Plan Not reviewed, remains open in Unity 

v.4.0.0.0 

26 X X 15 Closed M100, lVIM, IVO, ABCR Not reviewed, remains open in Unity 

v.4.0.0.0 

27 X X X 16 Closed IVIM, DAM, IVO, M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

43 X 17 Closed ABCR Not reviewed, remains open in Unity 

v.4.0.0.0 

284 X X 18 Closed PEB Reader/ Writer, DAM, IVIM Not reviewed, remains open in Unity 

v.4.0.0.0 

297 X 19 Closed

317 X 20 Closed

318 X X 21 Closed IVIM, M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

339 X 22 Closed

348 X X 23 Closed ABCR, IVIM, DAM, M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

355 X 24 Closed ABCR , Voyager hand scanner, 4.0 

Test Plan

Not reviewed, remains open in Unity 

v.4.0.0.0 

359 X 25 Closed ABCR , Voyager hand scanner, Test 

Plan

Not reviewed, remains open in Unity 

v.4.0.0.0 

361 X 26 Closed Test Plan Not reviewed, remains open in Unity 

v.4.0.0.0 

372 X X X 27 Closed M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

411 X 28 Open, FCA

429 X 30 Open

435 X 31 Open, FCA

453 X 32 Open, FCA

454 X 33 Open, FCA

473 X 34 Closed

475 X 35 Open, FCA

479 X 36 Closed

480 X 37 Closed

492 X 38 Closed

493 X 39 Closed

495 X 40 Closed

496 X 41 Closed

497 X 42 Closed
Draft v.0.1 Unity v.3.2.0.0 PCA Document Review Assessment - Attachment - Tables 1 & 2 Page 4 of 4

SysTest# DS200 M650 VAT EMS Other 

Doc

iBeta# Disposition Part Excluded fm Unity v.3.2.0.0 Out of Scope Disposition

549 X 43 Closed

550 X 44 Closed

553 X 45 Open

554 X 46 Open

555 X 47 Open

556 X 48 Open

557 X 49 Closed

Table 2 Out of Scope & Non Issues 

SysTest # Finding Disposition

190, 191, 196, 198, 235, 238, 

245, 369, 382, 388, 390, 401, 

428, 434, 437, 441, 442, 445, 

446, 450, 451, 452, 458, 461, 

463, 464, 466, 467, 468, 469, 

474, 478, 483, 485, 486, 487, 

488, 490, 491, 494, 498, 503, 

511, 512, 513, 514, 515, 516, 

517, 518, 519, 520, 521, 522, 

523, 524, 525, 545, 547, 548, 

551, 552

The following are excluded from Unity v.3.2.0.0 :

System Hardware

Automated Bar Code Reader

iVotronic DRE Precinct Tabulator

Model 100 Precinct Ballot Counter

Voyager Hand Scanner (COTS)

System Software

Unity Data Acquisition Manager

Unity iVotronic Ballot Image Manager

Uncertified System Features

Network Data Transmission Including remote transmission of vote data 

and/or consolidated results data

Not reviewed, remains open in Unity v.4.0.0.0 

459, 510, 538 Closed or Informational Issues

Comments in the report identified these issues as closed or informational 

typographic errors 

Not reviewed, non- significant issue

499, 500, 501, 502, 504, 505, 

506, 507, 508, 509, 526, 527, 

528, 529, 530, 531, 532, 533, 

534, 535, 536, 537, 539, 540, 

541, 542, 543, 544, 546

Issues Written Against System Change Notes

Changes occurring during the Unity v.4.0.0.0 testing were reported in the 

System Change Notes. The role of the VSTL in the FCA process is to 

determine if the changes were tested appropriately and determine how 

they should be incorporated into functional testing. These discrepancies 

identify test or other documentation as lacking. The VSS instructs the 

VSTL to test if testing is inadequate. In iBeta's opinion, as written, these 

are not documentation discrepancies, but findings applicable to the FCA.

As these are findings for functional test scope they 

remain open in Unity v.4.0.0.0 ; iBeta shall examine the 

change notes as part of the FCA Document Review for 

relevance to the Unity v.3.2.0.0 test scope 
3131 South Vaughn Way Suite 650 Aurora, CO 80014 Phone (303) 627-1110 Fax (303) 627-1221

January 16, 2009

Mr. Brian Hancock Sent via E-mail

U.S. Election Assistance Commission

Voting System Testing and Certification Program

1225 New York Avenue, NW, Suite 110

Washington, DC 20005

Mr. Hancock,

The purpose of this letter is to document the 3% review of the Unity 3.2.0.0 source code in 

accordance with your 21 November 2008 email providing instruction on the reuse of testing for 

the ES&S certification effort. This letter also provides the iBeta recommendation to the EAC 

regarding the reuse of the source code review conducted by SysTest.

Documentation of the Review Process

To conduct the review, iBeta used our PCA Source Code Review Procedure. The source code 

was delivered from SysTest Labs and configuration managed in the iBeta Source Code 

Repository. With the exception of Cobol, the coding languages submitted for review had been 

previously reviewed on other certification test efforts therefore the previously used interpretation 

of the generic VSS requirements to the language specific review criteria were utilized 

unmodified. For the Cobol review, iBeta provided the interpretation of each VSS requirement to 

ES&S prior to initiating the source code review task. The language specific review criteria for 

each of the five languages is not attached to this letter and will be provided if deemed necessary 

for the EAC review. The VSS requirements applicable to the source code review task are: 

VSS

Vol. # Section(s) #

1 4.2.2

1 4.2.3

1 4.2.4

1 4.2.5

1 4.2.6

1 4.2.7

1 6.2

1 6.4.2

2 2.5.4d

2 5.4.2
3131 South Vaughn Way Suite 650 Aurora, CO 80014 Phone (303) 627-1110 Fax (303) 627-1221

To select the 3% for review, iBeta conducted an analysis by first using a library of static analysis 

tools to parse each application source code base and provide a list of the files and functions as 

well as the Lines of Code (LOC) count. iBeta uses executable LOCs only and does not include 

comment, blank, or continued lines in our metrics. An exception to this process was the Cobol 

applications as our library of static analysis tools do not address Cobol source code. For those 

two applications, the number of files and files sizes were used to determine the volume of code 

in order to select 3%.

Once the spreadsheets were populated for each application, a selection of files/functions was 

made based on the file header information documenting the file purpose. iBeta focused the 

review by selecting source code files and functions that process vote data, audit logs, and 

reporting. 

The ES&S AutoMARK source code submitted was compared against the source code submitted 

with the Premier certification effort as the code is similar. The differences between those two 

source code bases were then reviewed as part of the ES&S 3% source code review. The 

unique as well as the shared application discrepancies are reported herein. 

The peer review of each Source Code Review was conducted by experienced reviewers who 

had reviewed source code to the VSS requirements on a minimum of two VSTL test efforts. 

Based on the instruction in your 21 November 2008 email "This review will focus on important 

functional sections of the code in order to determine the depth and focus of source review 

conducted by SysTest", the peer review analyzed each instance of non-compliance with the 

VSS requirements and assessed if the issue impacted source code logic. Discrepancies that 

dealt with comments, headers, formatting, and style were accepted as non-logic issues and 

color coded as green. Potential logic issues were flagged as needing an EAC decision and 

color coded as yellow. Confirmed logic issues were to be flagged as red (no confirmed logic 

issues were identified).

The matrix of the source code reviewed is provided as Attachment 1 and each individual 

discrepancy spreadsheet is provided as a separate confidential compressed file delivered on 

CD subsequent to the email delivery of this letter. 

Summary of 3% Source Code Review Results

A total of 330 discrepancies were identified with the majority, 307 or 93%, categorized as nonlogic issues. The summary of discrepancies categorized as EAC Decision Discrepancies as 

well as the vendor responses are provided as Attachment 2 to this letter. 

Of the 21 of 23 potential logic discrepancies, ES&S has provided in their response their 

justification for non complying with the requirement or their disagreement of the iBeta 

interpretation of the VSS requirements. Precedence for the iBeta interpretation has been 

established with testing for other clients and these established interpretations must be applied 

consistently to all manufacturers under test with iBeta. We do acknowledge that in some 

instances another interpretation may be possible and that alternative interpretation may be 

acceptable to the EAC reviewers. 
3131 South Vaughn Way Suite 650 Aurora, CO 80014 Phone (303) 627-1110 Fax (303) 627-1221

The remaining 2 potential logic discrepancies remain under investigation by both Premier and 

ES&S and are expected to be addressed within the Premier source code review letter. 

Recommendation Regarding the Reuse of the SysTest Source Code Review

In order to provide a recommendation, iBeta evaluated the results of the 3% source code review 

whereas the results would be recommended as accepted if no significant discrepancies were 

found, this includes the less critical requirement which were not addressed, not recorded or 

interpretations are inconsistent with documenting industry accepted practices. As there were 

discrepancies written that potentially impact the source code, two other analyses were 

conducted:

1. Confirmed that the results of the iBeta review of the 3% of code are consistent with 

the previous results (not identical but consistent): This confirmation was reached by 

reviewing the types of discrepancies generated by SysTest in the 100% review against 

those generated by iBeta.

2. Reviewed the severity of the discrepancies discovered: The number of discrepancies 

potentially impacting the source code is considered very low versus the overall number 

of discrepancies (as is consistent with a 100% review). The severity of the 

discrepancies and the vendor responses do indicate that the majority of those 21 

potential logic discrepancies would be resolved without source code modifications.

Based on the limited impact (or perhaps no impact) on the source code as a result of these 

discrepancies, iBeta recommends reuse of the results of the SysTest source code.

 

Sincerely,

Gail Audette

iBeta Quality Manager

Attachment 1: Matrix of Source Code Reviewed

Attachment 2: Summary of Discrepancies

Enclosure: CONFIDENTIAL CD Source Code Review Discrepancies 1-16-09.zip

cc: Steve Pearson, ES&S

Sue Munguia, ES&S