TGDC Meeting, July 2011

VVSG 1.1 Test Suite Status

Mary Brady

Manager, NIST Information Systems Group, Software and Systems Division, ITL

http://vote.nist.gov 

*

TGDC Meeting, July 2011

Page *

Background
        - Status quo:  Labs have been testing to VVSG 1.0 (2005) using proprietary, custom tooling and review processes
        - In 2007-08, NIST developed a set of public test suites for VVSG 2.0, to be used as part of the EAC Testing and Certification Program
        - In 2009, to support VVSG 1.1, test methods for new and changed material was back-ported from 2.0 test suites; status quo prevails for everything else

TGDC Meeting, July 2011

Page *

Test Validation
        - NIST test suites have been rigorously traced to VVSG requirements
        - Reviewed by independent parties, including the EAC, VSTL labs, experts, and the public
        - Procedural validation has been on-going
        - Operating temperature and humidity -- complete
        - Usability and accessibility – ongoing
        - Core functionality, security under consideration
        - Pursued obtaining in-house voting systems to aid in the evaluation of NIST test suites

TGDC Meeting, July 2011

Page *

Recent Activities
        - NIST met with each accredited VSTL to discuss possible integration of core functionality tests
        - Recent activities regarding changes to VVSG 1.1 will require changes to the NIST developed test suites; delayed status of VVSG 2.0 warrants additional focus on improving testing for VVSG 1.1, including existing methods for VVSG 1.0
        - NIST and EAC met with both accredited VSTL’s to discuss ways to improve testing for the evolving VVSG 1.1 and to integrate NIST-developed tests into the certification process 

TGDC Meeting, July 2011

Page *

Preliminary Findings
        - To ensure consistency across testing laboratories, agreed upon test methods must be developed for all of VVSG 1.1
        - Face-to-face meetings revealed differences in interpretation of the term “readiness test”.  Labs read as “acceptance test” whereas NIST definition was a “simple health check”
        - Some NIST tests require new election definitions, which are problematic for the labs
        - Labs define test data separately from the test case; NIST defines them together

TGDC Meeting, July 2011

Page *

Additional Findings
        - Some NIST tests should be reworked to take into account practical test methods, with temperature and humidity cited as an example
        - Source code analysis represents a large percentage of the cost of testing; tools remain difficult to use
        - Reference ballots, including guidance on how ballots should be marked (pre-marked, hand-marked), election definitions/voting patterns to be used, and complexity of precincts/ballot styles, would be useful
        - Should identify priority areas outside of VVSG 1.1

TGDC Meeting, July 2011

Page *

Action Plan
        - Monthly meetings among EAC, NIST, VSTL’s, with bi-annual face-to-face meetings:
        - Focus on developing common test methods for new VVSG 1.1, starting with information from EAC matrix, NIST test suites, and existing VSTL definitions
        - Validate NIST test suites and integrate into certification process
        - Expand effort on source code analysis tools and work with the VSTL’s to develop rules for VVSG 1.1
        - Explore creating a set of reference ballots to be used by all labs

TGDC Meeting, July 2011

Discussion/Questions

Page *