3131 South Vaughn Way Suite 650 Aurora, CO 80014 Phone (303) 627-1110 Fax (303) 627-1233

January 14, 2009

Mr. Brian Hancock Sent via E-mail

U.S. Election Assistance Commission

Voting System Testing and Certification Program

1225 New York Avenue, NW, Suite 110

Washington, DC 20005

Dear Mr. Hancock, 

The purpose of this letter is to document the audit of the ES&S Unity v.3.2.0.0 Technical Data Package (TDP) in 

accordance with your November 21, 2008 email to Steve Pearson of ES&S providing instructions on the reuse 

of SysTest Labs' Unity v.4.0.0.0 testing for the ES&S Unity v.3.2.0.0 . EAC Certification test effort. iBeta has 

completed the audit of the Unity v.3.2.0.0 TDP. This letter provides iBeta's findings and opinion to the EAC 

regarding the reuse of the PCA Document Review conducted by SysTest Labs from the Unity v.4.0.0.0 test 

effort. 

Scope of the Unity v.3.2.0.0 EAC Certification Test Effort

The scope of Unity v.3.2.0.0 is a subset of Unity v.4.0.0.0 . It is restricted to the applications of the ES&S 

election management system, the DS200 and M650 optical scanners and the AutoMARK Voter Assistant 

Terminals (VAT) A100 and A200 ballot markers. It is our understanding that over the course of the Unity

v.4.0.0.0 test effort ES&S made changes to the scope of Unity v.4.0.0.0 . These scope changes included the 

incorporation of the VAT ballot markers and the AIMS application. These had previously been in the Unity 

3.0.1.0-3.0.1.1 EAC Certification test effort. Results of the PCA Document Review performed by SysTest Labs, 

including open discrepancies were delivered in: 

Unity v.4.0.0.0 PCA Document Review results for the ES&S election management system applications, 

the DS200and M650; and

Unity 3.0.1.0-3.0.1.1 PCA Document Review results for the VATs and AIMS.

ES&S delivered Unity v.3.2.0.0 TDP which consisted of:

The rebranded Unity v.4.0.0.0 TDP incorporating the "NOTICE OF UNCERTIFIED FUNCTIONALITY"

disclaimer; and

The AutoMARK VAT and AIMS TDP.

iBeta sampled the ES&S Unity v.3.2.0.0 documents. The sample selection included the documents identified in 

the SysTest Labs issued discrepancies and documents needed to complete the Unity v.3.2.0.0 trusted builds, a 

sample 3% source code review, test planning and test execution. Criteria for the review were to confirm that the 

Unity v.3.2.0.0 documents addressed any document discrepancies within the scope of the Unity v.3.2.0.0 test 

effort and the content provided sufficient information in order to complete the test tasks list above. 

Documentation of the Audit of the TDP 

Due to the change of scope many discrepancies issued by SysTest Labs were outside the scope of Unity

v.3.2.0.0 . iBeta reviewed every open discrepancy. Issues which were identified as all or partially relevant to 

the Unity v.3.2.0.0 scope were transferred to iBeta's Unity v.3.2.0.0 Discrepancy Report. Issues or parts of 

issues, outside this scope were excluded. Scope assessment was recorded in a review disposition document. 

The transferred discrepancies identified location of the issue, SysTest Labs discrepancy number, and detail of 

the initial description from the SysTest Labs discrepancy report. iBeta confirmed the issues were valid and 

traced to an appropriate 2002 VSS requirement. iBeta reviewed the SysTest Labs description history from the 

original SysTest Labs discrepancy report and the Unity v.3.2.0.0 documents submitted by ES&S to validate 
Page 2 of 4

resolution of the issue. In some instances discrepancies are being incorporated into Unity v.3.2.0.0 Functional 

Configuration Audit (FCA). 

The review of documents necessary to complete Unity v.3.2.0.0 trusted builds, sample code review, test 

planning and test execution was incorporated into these tasks and recorded in the daily status. Missing content 

or discrepancies were reported in the Unity v.3.2.0.0 Discrepancy Report. 

Review Results

The Unity v.3.2.0.0 TDP submitted by ES&S was sufficient to close the majority of the document discrepancies 

deemed inside the scope of Unity v.3.2.0.0. 

One document issue remains open for additional clarification of the ES&S response;

One document issue remains open for incorporation into the iBeta Security Review; and 

Four issues did not have a response from ES&S. As these were the last items on the list it is possible 

that they had not been submitted to ES&S by SysTest Labs. 

Review of documents necessary to perform Unity v.3.2.0.0 trusted builds, sample 3% code review and test 

planning has generally found them to contain the information needed to perform these task. Four issues were 

noted in the review. 

Document discrepancy #10 identified a gap in the Win650 build procedure;

Document discrepancy #50 identified the System Overview and System Limitations do not reflect the 

language scope of Unity v.3.2.0.0;

Document discrepancy #52 identified System Overview contained a typo with an incorrect hardware 

version for the DS200; and

Document discrepancy #53 identified the absence of the VATs and AIMS from the System Limitations.

The attached Tables 1 and 2 provide the results and disposition of all SysTest Labs Unity v.4.0.0.0 issued 

discrepancies. Note: Functional discrepancies which remain open for validation in the FCA are also listed in 

Table 1.

Recommendation on Reuse of the SysTest Labs PCA Document Review

Based upon the audit and review documented herein iBeta deems that the results of the SysTest PCA 

Document Review are adequate for reuse in the Unity v.3.2.0.0 Certification test effort. Use of the TDP in 

development of the Volume, Stress, Error Recovery and Security testing shall incorporate additional review. Any 

documentation issues encountered shall be reported in the Unity v.3.2.0.0 discrepancy report. We do not 

recommend a more comprehensive review of the TDP. 

Best regards,

Carolyn Coggins

QA Director - Voting

Attachments: Table 1 & 2
Draft v.0.1 Unity v.3.2.0.0 PCA Document Review Assessment - Attachment - Tables 1 & 2 Page 3 of 4

Table 1 Unity v.3.2.0.0 Applicable Discrepancies

SysTest# DS200 M650 VAT EMS Other 

Doc

iBeta# Disposition Part Excluded fm Unity v.3.2.0.0 Out of Scope Disposition

6 X X 12 Closed M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

23 X 13 Closed ABCR, Test Plan Not reviewed, remains open in Unity 

v.4.0.0.0 

24 X 14 Closed ABCR, Test Plan Not reviewed, remains open in Unity 

v.4.0.0.0 

26 X X 15 Closed M100, lVIM, IVO, ABCR Not reviewed, remains open in Unity 

v.4.0.0.0 

27 X X X 16 Closed IVIM, DAM, IVO, M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

43 X 17 Closed ABCR Not reviewed, remains open in Unity 

v.4.0.0.0 

284 X X 18 Closed PEB Reader/ Writer, DAM, IVIM Not reviewed, remains open in Unity 

v.4.0.0.0 

297 X 19 Closed

317 X 20 Closed

318 X X 21 Closed IVIM, M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

339 X 22 Closed

348 X X 23 Closed ABCR, IVIM, DAM, M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

355 X 24 Closed ABCR , Voyager hand scanner, 4.0 

Test Plan

Not reviewed, remains open in Unity 

v.4.0.0.0 

359 X 25 Closed ABCR , Voyager hand scanner, Test 

Plan

Not reviewed, remains open in Unity 

v.4.0.0.0 

361 X 26 Closed Test Plan Not reviewed, remains open in Unity 

v.4.0.0.0 

372 X X X 27 Closed M100 Not reviewed, remains open in Unity 

v.4.0.0.0 

411 X 28 Open, FCA

429 X 30 Open

435 X 31 Open, FCA

453 X 32 Open, FCA

454 X 33 Open, FCA

473 X 34 Closed

475 X 35 Open, FCA

479 X 36 Closed

480 X 37 Closed

492 X 38 Closed

493 X 39 Closed

495 X 40 Closed

496 X 41 Closed

497 X 42 Closed
Draft v.0.1 Unity v.3.2.0.0 PCA Document Review Assessment - Attachment - Tables 1 & 2 Page 4 of 4

SysTest# DS200 M650 VAT EMS Other 

Doc

iBeta# Disposition Part Excluded fm Unity v.3.2.0.0 Out of Scope Disposition

549 X 43 Closed

550 X 44 Closed

553 X 45 Open

554 X 46 Open

555 X 47 Open

556 X 48 Open

557 X 49 Closed

Table 2 Out of Scope & Non Issues 

SysTest # Finding Disposition

190, 191, 196, 198, 235, 238, 

245, 369, 382, 388, 390, 401, 

428, 434, 437, 441, 442, 445, 

446, 450, 451, 452, 458, 461, 

463, 464, 466, 467, 468, 469, 

474, 478, 483, 485, 486, 487, 

488, 490, 491, 494, 498, 503, 

511, 512, 513, 514, 515, 516, 

517, 518, 519, 520, 521, 522, 

523, 524, 525, 545, 547, 548, 

551, 552

The following are excluded from Unity v.3.2.0.0 :

System Hardware

Automated Bar Code Reader

iVotronic DRE Precinct Tabulator

Model 100 Precinct Ballot Counter

Voyager Hand Scanner (COTS)

System Software

Unity Data Acquisition Manager

Unity iVotronic Ballot Image Manager

Uncertified System Features

Network Data Transmission Including remote transmission of vote data 

and/or consolidated results data

Not reviewed, remains open in Unity v.4.0.0.0 

459, 510, 538 Closed or Informational Issues

Comments in the report identified these issues as closed or informational 

typographic errors 

Not reviewed, non- significant issue

499, 500, 501, 502, 504, 505, 

506, 507, 508, 509, 526, 527, 

528, 529, 530, 531, 532, 533, 

534, 535, 536, 537, 539, 540, 

541, 542, 543, 544, 546

Issues Written Against System Change Notes

Changes occurring during the Unity v.4.0.0.0 testing were reported in the 

System Change Notes. The role of the VSTL in the FCA process is to 

determine if the changes were tested appropriately and determine how 

they should be incorporated into functional testing. These discrepancies 

identify test or other documentation as lacking. The VSS instructs the 

VSTL to test if testing is inadequate. In iBeta's opinion, as written, these 

are not documentation discrepancies, but findings applicable to the FCA.

As these are findings for functional test scope they 

remain open in Unity v.4.0.0.0 ; iBeta shall examine the 

change notes as part of the FCA Document Review for 

relevance to the Unity v.3.2.0.0 test scope