EAC Decision on Request for Interpretation 2015-01

(Testing Breadth)

VVSG 1.1 Volume 2: 7.2.1 Testing Breadth

Date: 

July 15, 2015

Question: 

How much functional testing is required under VVSG 1.1 Volume 2 

requirement 7.2.1?

Section of Guidelines:

VVSG 1.1 V2: 7.2.1

7.2.1 Testing Breadth

The VSTL shall execute tests that provide coverage of every accessible instruction 

and branch outcome in application logic and border logic. This is not exhaustive 

path testing, but testing of paths sufficient to cover every accessible instruction and 

every accessible branch outcome. There should be no inaccessible code in 

application logic and border logic other than defensive code (including exception 

handlers) that is provided to defend against the occurrence of failures and "can't 

happen" conditions that cannot be reproduced and should not be reproducible by a 

VSTL. Full coverage of third-party logic is not mandated because it might include a 

large amount of code that is never used by the voting application. 

Discussion: 

A key statement in this requirement leads the discussion: “This is not exhaustive 

path testing, but testing of paths sufficient to cover every accessible 

instruction and every accessible branch outcome.” The intent of this requirement is 

not to exhaustively test every possible path within the voting system, but to test a 

practical amount of entry and exit points. Exhaustive testing, or full path coverage, 
is not a common practice as it can be extremely costly and can often be impossible 

depending upon the complexity of the system under test. 

The EAC interprets accessible instruction to mean user accessible instruction. A 

user accessible instruction includes any input the user (pollworker, election official, 

etc.) can access through the user interface. Any path that can be activated through 

the user interface shall be tested to validate the expected outcome. 

Voting system components may have a nearly infinite combination of inputs; 

therefore the Testing and Certification Program requires that the manufacturer 

define system limits in addition to all allowable potential user inputs. Ideally this 

documentation includes the system response/handling of invalid inputs. This should 

include given user inputs in given system environmental conditions, which could 

result in different system responses. An example of this might include where a 

polling place media has been read into the tabulation application/device, and then 

tried to be read in again, once with the same data, then another time with different 

data.

It is the responsibility of the VSTL to accurately test all system limits, and inputs,

as per the manufacturer’s definition, such that adequate path coverage is provided. 

The manufacturer may also provide internal test cases and results to the VSTL for 

assistance in developing the certification test plan. 

Conclusion: 

When testing to requirement V2:7.2.1, testing all possible branches is not required. 

All user accessible entry points and their outputs must be tested to validate 

expected outcomes for both positive and negative scenarios, accounting for 

environmental variances that result in different outcomes. Test cases shall be 

reviewed by the manufacturer and the EAC to validate these paths are covered 

within each campaign. 

Applicability:

Immediate- for all voting system applications submitted to VVSG 1.1.