Software ITA Testing t

2002 FEC Voting System S

First NIST Symposium on Building T

Confidence in Voting System

National Institute of Standards and Techn

Gaithersburg, MD

December 10-11, 2003 

Presented by Carolyn Coggins
NASED Voting ITA S

NASED requires all voting systems to su

by both the hardware and software ITA

•

Hardware ITA Scope

3⁄4 Functional Testing: Polling Place 3⁄4 Environmental Hardware Testing: Polling Place 3⁄4 Security, Accuracy & Reliability: Polling Place3⁄4 Accessibility: Polling Place

•

Software ITA Scope

3⁄4 Functional Testing: Ballot Definition & Central 3⁄4 Source Code Review: Ballot Definition & Cent3⁄4 Documentation Review: Ballot Definition & Ce3⁄4 Security, Accuracy & Reliability: Ballot Defini3⁄4 Syste

m Level Tests: Ballot Definition Polling P

•COTS exemptions:

3⁄4 Environmental Hardware Testing3⁄4 Source Code Review
Software ITA Test Standards 

SysTest Labs uses the following to perform 

qualification testing:

•

Federal Election Com

m

ission Voting System 

April 2002

3⁄4 Effective on voting systems submitted for test 

•

SysTest Labs Quality Manual & Standard Pro

3⁄4 Sub

mitted to NASED as part of our accreditati

3⁄4 Documents interpretation of the VSS & softwa3⁄4 Updated for the 2002 VSS & on-going proces

•

Various review/report forms, data bases and t

3⁄4 Specifically designed for testing to the require3⁄4 Assuring version control, complete documenta

standardized/repeatable testing.

3⁄4 Updated for on-going process improvement
Physical Configuration Aud

(FECVSS V

ol. 2 Section 2, 3, 5.3, 6.6, & 6.7)

PCA Docum

ent Review of the Technical 

•

The reviewer the completes the PCA Docume

for each submitted document against the requirVSS.

3⁄4 Syste

m Functionality, Hardware/Software/Des

Security, Training, Maintenance, User ManualsManagement and QA Program

•

Documented in the Pre-Qualification Report 

the vendor.

Verification of Software & Hardware Con•

Examine and set up the syste

m hardw

components for all documented components.

•

Verify documentation is consistent with config

hardware ITA.
Physical Configuration Aud

continued (FECVS

S V

ol. 2. Section 5.4)

Source Code Review•

Identify code to review (New code vs. Changed 

•

Review vendor coding standards and customize 

necessary) for the specific progra

m

m

ing langua

•

Manual source code review, examine the softwa

3⁄4 Integrity- No external m

odification of code

• Follow each function looking for wor

ms, viruses 

• Check for buffer overflows that can lead to s

ecuri

execution of m

alicious code.

3⁄4 Absolute logical correctness, 

modularity, overall 

conform

ance with VSS requirem

ents and vendor 

3⁄4 Review the system architecture for use of system

•

Automated source code review (if possible).

3⁄4 Use an automated tool to obtain metrics of the L3⁄4 Use security tools run to checks for constructs kn
Functional Configuration Au

(FECVSS V

ol. 2 Section 2, 3, 5.3, 6, 6.6, & 6.7 Appen

FCA document review of the System Test & Specification and all vendor testing. 

•

Complete the FCA Vendor Testing Review 

coverage of:

3⁄4Required Ballot Preparation and Central Count Fu3⁄4Optional Features3⁄4Syste

m Level Testing

•

Document the results of the FCA Testing 

qualification Report, sent to NASED and the ve

3⁄4Assess the overall adequacy of the vendor’s testin3⁄4Identify any gaps in testing3⁄4Identify the scope of functional testing3⁄4Identify the test tasks and predecessor tasks.
Functional Configuration Au

continued (Vol. 2 Section 6, Appendix A )

Define scope of testing based upon results •

Create the Test Plan, sent to NASED and the ve

•

Custom

ize System Level Tests to the voting sys

3⁄4 Voting Variations: Pri

m

ary/General Election, Str

3⁄4 Security: 

• Access control policies • Unauthorized changes to ballot for

mats, cast vote

• Alteration of voting s

ystem audit trails

• Access to individual votes (

maintaining ballot sec

• Test election dates 0 to 8 

years out, including the 

3⁄4 Accuracy & Reliability: Meets expected results 

number of votes and tim

e

•

Assess any additional risks for the specific syste

3⁄4 Aug

ment tests for the specific voting system.
Functional Configuration A

continued (Vol. 2 Section 6, Appendix C)

Test Execution:•

Observe the build of the executable from review

•

Test environ

ment is setup per the PCA -

System 

•

Execute test cases and record results

3⁄4 Any discovered issues are logged on the project 

• Discrepancies: user documentation or claim

s abo

per

formance.

• Defects: issue prevents the s

yste

m

fr

o

m functionin

•

Vendors must address all issues and resol

qualification. 3⁄4 Fixes defect or discrepancy that impacts qualific

• Sends new code, it’s reviewed, regression tested a• Decides not to support 

functionality or addres

Documentation changes are reviewed, verified an

3⁄4 Logs a bug for a future release or chooses not t

qualification.

3⁄4 All vendor responses are noted in the Qualificati
Qualification Test Re

(Vol. 2. Appendix B)

The Qualification Test Report consists of:•

Introduction: The vendor’s s

ystem, any changes 

•

Qualification Test Background: Test Process and 

•

System Identification: S

ystem, Version, Test Env

•

System Overview: Syste

m Design and Operation

•

Qualification Test Results and Recommendation

Deficiencies and Recommendation 

•

Appendix –

Test Operations , Findings and Data 

3⁄4 Qualification Test Requirem

ents, Source C

ode Rev

Test Results and Discrepancy Report 

• NASED Signatory reviews and signs the report 

the vendor and the NASED Technical Com

m

ittee

3⁄4

C

o

m

mittee has five days to question the report. 

3⁄4 Report is revised with NASE

D Certification Nu

mb

3⁄4 Report is distributed to states or jurisdiction upon r