EAC Decision on Request for Interpretation 2008- 11 

(Software branch analysis) 

2002 VSS Vol. II Section 6.2.1 

2005 VVSG Vol. II Section 6.2.1 

Date: 

December 19, 2008 

Question(s): 

Should the VSTLs' testing include software branch analysis? 

Section of Standards or Guidelines: 

2002 VSS Vol. II 6.2.1 Testing Breadth 

2005 VVSG Vol. II 6.2.1 Testing Breadth 

Background: 

Branch analysis is a process to coordinate source code review with system and functional 

testing. It can be a powerful technique to help the test designer demonstrate that the 

voting system standards' requirements highlighted below have been tested. The analysis 

allows a reviewer to determine if the current tests evaluate the key code branches. 

There are a variety of objectives for branch analysis. These objectives include: 

Function Coverage - This metric reports percentage of functions or procedures 

covered during testing. 

Component Entry/Exit – This metric reports the percentage of entry and exit 

paths tests for each component of a voting system and then the total voting 

system. 

Condition coverage – Measure of the percentage of evaluation points (such as a 

true/false decision) executed during testing. 
Path coverage – Measure of the percentage of possible route through a given part 

of the code that have been executed. 

Methods of branch analysis range from use of commercial products to automate the 

analysis, the use of code with logging statements included (sometimes called 

instrumented code), and manual analysis. 

2002 VSS Volume II 

ITAs shall design and perform procedures that test the voting system capabilities for the system as a whole. 

These procedures follow the testing of the systems hardware and software, and address voting system 

requirements defined in Volume I, Sections 2, 5, 6 and 8. These procedures shall also address the 

requirements for testing system functionality provided in Volume II, Section 3. Where practical, the ITA 

will perform coverage reporting of the software branches executed in the functional testing. The 

selection of the baseline test cases will follow an operational profile of the common procedures, 

sequencing, and options among the shared state requirements and those that are specifically recognized 

and supported by the vendor. The ITA will use the coverage report to identify any portions of the source 

code that were not covered and determine: 

a. The additional functional tests that are needed; 

b. Where more detailed source code review is needed; or 

c. Both of the above. 

The specific procedures to be used shall be identified in the Qualification Test Plan prepared by the ITA. 

These procedures may replicate testing performed by the vendor and documented in the vendor's TDP, but 

shall not rely on vendor testing as a substitute for testing performed by the ITA. 

Recognizing variations in system design and the technologies employed by different vendors, the IT As 

shall design test procedures that account for these variations. 

2005 VVSG Vol. II 

The accredited test lab shall design and perform procedures that test the voting system capabilities for the 

system as a whole. These procedures follow the testing of the systems hardware and software, and address 

voting system requirements defined in Volume I, Sections 2, 4, 5 and 6. These procedures shall also 

address the requirements for testing system functionality provided in Section 3. Where practical, the 

accredited test lab will perform coverage reporting of the software branches executed in the functional 

testing. The selection of the baseline test cases will follow an operational profile of the common 

procedures, sequencing, and options among the shared state requirements and those that are specifically 

recognized and supported by the vendor. The accredited test lab will use the coverage report to identify any 

portions of the source code that were not covered and determine: 

a. The additional functional tests that are needed 

b. Where more detailed source code review is needed 

c. Both of the above 

The specific procedures to be used shall be identified in the National Certification Test Plan. These 

procedures may replicate testing performed by the vendor and documented in the vendor’s TDP, but shall 

not rely on vendor testing as a substitute for testing performed by the accredited test lab. Recognizing 

variations in system design and the technologies employed by different vendors, the accredited test lab 

shall design test procedures that account for these variations. 

2
3

Conclusion: 

To address the requirement in VSS/VVSG Section 6.2.1, the VSTL’s are expected to 

analyze their test campaign (plan, test cases, implementation, and report) with respect to 

the system design and provide that analysis as part of the test report. 

To address the requirement regarding branch (coverage) analysis is to use the analysis to 

help develop and execute a comprehensive test campaign and provide assurance that the 

testing has covered all relevant sections in the system. This means that the testing should 

exercise the normal and alternative data paths as well as exception paths. The testing 

includes operational, code review and document review in the optimum mix to provide 

assurance of compliance. 

Branch (coverage) analysis (and source code review) should also include database objects 

such as: triggers, functions, and stored procedures, it should also include logic and code 

in ROM and Programmable Logic Devices whenever they are employed by the voting 

system. 

Effective Date: 

Immediately for all voting systems not having a test report submitted to the EAC for 

review as of the date of this document. 

Useful Links: 

http://www.testingfaqs.org/t-eval.html – list of coverage analysis tools 

http://bullseye/coverage.html - article explaining code testing analysis 

http://en.wikipedia.org/wiki/Code_coverage – Wikipedia article on code coverage testing 

http://doi.ieeecomputersociety.org/10.1109/COMPSAC.2005.150 - ieee article “Testing 

Coverage Analysis for Software Component Validation”