Source: http://www.eac.gov/vvsg/part_3_testing_requirements/chapter_5_test_methods.aspx
Timestamp: 2015-11-25 23:42:39
Document Index: 92133187

Matched Legal Cases: ['art 1', 'art 2', 'art 1', 'art 1', 'art 1', 'art 1', 'art 15', 'art 1', 'art 1', 'art 3', 'art 2', 'art 2', 'art 1', 'art 1', 'art 3', 'art 3', 'art 3', 'art 1', 'art 1', 'art 1', 'art 1', 'art 1', 'art 3', 'art 3', 'art 3', 'art 3', 'art 1', 'art 1', 'art 3', 'art 1', 'art 1', 'art 3', 'art 1', 'art 3', 'art 1', 'art 2', 'art 2', 'arts 1', 'art 4', 'art 3', 'art 3', 'art 3']

Chapter 5: Test Methods
HomeChapter 5: Test Methods
The accredited test lab must design and perform procedures to test a voting system against the requirements outlined in Part 1. Test procedures must be designed and performed that address:
Overall system capabilities;
Pre-voting functions;
Voting functions;
Post-voting functions;
System maintenance; and
The specific procedures to be used must be identified in the test plan prepared by the accredited test lab (see Part 2: Chapter 5: "Test Plan (test lab)"). These procedures must not rely on manufacturer testing as a substitute for independent testing.
5.1. It is not clear in this section whether each electronic device that comprises the voting system is to be separately tested, or if the entire system is to be tested as a whole. 5.1.1.2. "the product" is not defined. 5.1.1.2-A.1. "it is recommended" has no place. The limit must be specified exactly. 5.1.1.2-A.2. "it is recommended" has no place. The limit must be specified exactly. 5.1.1.2-B. The "industry-recognized standards" need to be cited. This section needs to acknowledge that not all devices have telephone ports. 5.1.3.2-A. "Voting system" or "voting device"? 5.2.2-A. This is not possible. Earlier VVSG sections (1: 6.4.1.8-B and 3: 4.6) acknowledge that not all paths can be verified. Other sections, such as 5.2.3-B.1, also are counter to this requirement. 5.2.3. Using "system" instead of "voting system". 5.2.3-F.1 and 5.2.3-F.2. How is "...tests to verify..." different from "...tests to check..."? 5.3.2. Far too many significant figures in the calculation results. You can't use 7 or 8 significant digits when the input is only 2 or 3. 5.4.2-B. "expert" is subjective. 5.4.2-C vs 5.4.2-D. Why is the experience requirement higher for the election management "expert" than for the other "experts"? 5.4.2-E.a. "complete knowledge" of anything is impossible. 5.4.2-E. These requirements are heavily biased towards the election field. For instance, why would the Information Security expert be required to have designed a voting system? 5.4.3-B. "system model" is not defined, nor used in any other place in the VVSG. 5.4.3-C. "threat model" is not defined, nor used in any other place in the VVSG. 5.4.4-A, 5.4.4-B, 5.4.4-C. Sections switch from "voting system" to "voting device". Is this intentional? The use of "voting device" implies that parts of a system may pass and others may fail. 5.4.6. VSTL is used, not test lab. Is this intentional under the Program?
5.1.1 Electromagnetic compatibility (EMC) immunity
Testing of voting systems for EMC immunity will be conducted using the black-box testing approach, which "ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions" (from [IEEE00]). It will be necessary to subject voting systems to a regimen of tests including most, if not all, disturbances that might be expected to impinge on the system, as recited in the requirements of Part 1.
Note: Some EMC immunity requirements have been established by Federal Regulations or for compliance with authorities having jurisdiction as a condition for offering equipment to the US market. In such cases, part of the requirements include affixing a label or notice stating that the equipment complies with the technical requirements, and therefore the VVSG does not suggest performing a redundant test.
Comment by Al Backlund (Voting System Test Laboratory)
Does this mean that the VSTLs can accept FCC class B testing performed outside the certification timeframe?
5.1.1.1 Steady-state conditions
Testing laboratories that perform conformity assessments can be expected to have readily available a 120 V power supply from an energy service provider and access to a landline telephone service provider that will enable them to simulate the environment of a typical polling place.
5.1.1.2 Conducted disturbances immunity
Immunity to conducted disturbances will be demonstrated by appropriate industry-recognized tests and criteria for the ports involved in the operation of the voting system.
Adequacy of the product is demonstrated by satisfying specific "pass criteria" as outcome of the tests, which include not producing failure in the functions, firmware, or hardware.
The test procedure, test equipment, and test sequences will be based on some benchmark tests, and observation of the voltage and current waveforms during the tests, including (if relevant) detection of a "walking wounded" condition resulting from a severe but not immediately lethal stress that would produce a hardware failure some time later on.
5.1.1.2-A Power port disturbances
Testing SHALL be conducted in accordance with the power port stress testing specified in IEEE Std C62.41.2™-2002 [IEEE02a] and IEEE Std C62.45™-2002 [IEEE02b].
Applies to:	Electronic device
Both the IEEE and the IEC have developed test protocols for immunity of equipment power ports. In the case of a voting system intended for application in the United States, test equipment tailored to perform tests according to these two IEEE standards is readily available in tests laboratories, thus facilitating the process of compliance testing.
Comment by E Smith/B Pevzner (Manufacturer)
There are two IEEE specs are listed. Does it mean that testing for requirements of either of them is satisfactory?
5.1.1.2-A.1 Combination wave
Testing SHALL be conducted in accordance with the power port stress of "Category B" to be applied by a Combination Waveform generator, in the powered mode, between line and neutral as well as between line and equipment grounding conductor.
To satisfy this requirement, it is recommended that voting systems be capable of withstanding a 1.2/50 – 8/20 Combination Wave of 6 kV open-circuit voltage, 3 kA short-circuit current, with the following application points:
Three surges, positive polarity at the positive peak of the line voltage;
Three surges, negative polarity at the negative peak of the line voltage, line to neutral;
Three surges, positive polarity at the positive peak of the line voltage, line to equipment grounding conductor; and
Three surges, negative polarity at the negative peak of the line voltage, line to equipment grounding conductor.
The requirement of three successive pulses is based on the need to monitor any possible change in the equipment response caused by the application of the surges.
Source:	[IEEE02a] Table 3
5.1.1.2-A.2 Ring wave
Testing SHALL be conducted in accordance with the power port stress of "Category B" to be applied by a "Ring Wave" generator, in the powered mode, between line and neutral as well as between line and equipment grounding conductor and neutral to equipment grounding conductor, at the levels shown below.
Two different levels are recommended:
6 kV open-circuit voltage per Table 2 of [IEEE02a], applied as follows:
Three surges, positive polarity at the positive peak of the line voltage, line to neutral;
3 kV open circuit voltage, per Table 5 of [IEEE02a], applied as follows:
Three surges, positive polarity at the positive peak of the line voltage, neutral to equipment grounding conductor; and
Three surges, negative polarity at the negative peak of the line voltage, neutral to equipment grounding conductor.
Source:	[IEEE02a] Table 2 and Table 5
5.1.1.2-A.3 Electrical fast transient burst
Testing SHALL be conducted in accordance with the recommendations of IEEE Std C62.41.2™-2002 [IEEE02a] and IEEE Std C62.45™-2002 [IEEE02b].
Unlike the preceding two tests that are deemed to represent possibly destructive surges, the Electrical Fast Transient (EFT) Burst has been developed to demonstrate equipment immunity to non-destructive but highly disruptive events. Repetitive bursts of unidirectional 5/50 ns pulses lasting 15 ms and with 300 ms separation are coupled into terminals of the voting system by coupling capacitors for the power port and by the coupling clamp for the telephone connection cables.
Source:	[IEEE02a] Table 6, [ISO04b]
5.1.1.2-A.4 Sags and swells
Testing SHALL be conducted by applying gradual steps of overvoltage across the line and neutral terminals of the voting system unit.
Testing for sag immunity within the context of EMC is not necessary in view of Requirement Part 1: 6.3.4.3-A.4 that the voting system be provided with a two-hour back-up capability (to be verified by inspection). Testing for swells and permanent overvoltage conditions is necessary to ensure immunity to swells (no loss of data) and to permanent overvoltages (no overheating or operation of a protective fuse).
A) Short-duration Swells
As indicated by the ITI Curve [ITIC00], it is necessary to ensure that voting systems not be disturbed by a temporary overvoltage of 120 % normal line voltage lasting from 3 ms to 0.5 s. (Shorter durations fall within the definition of "surge.")
B) Permanent Overvoltage
As indicated by the ITI Curve [ITIC00], it is necessary to ensure that voting systems not be disturbed nor overheat for a permanent overvoltage of 110 % of the nominal 120 V rating of the voting system.
Comment by C R Williams (None)
"... permanent overvoltage of 110% of the nominal 120 V rating ..." A 110% overvoltage could mean that the total applied voltage is 120 v plus 110% of 120 v, or (120 + 132) v for 252 v total. I hope this is not what is meant. It would be good to clarify this, perhaps just stating what the total 'permanent withstand' voltage is for a nominal 120 v supply. (I'm guessing 132 v?)
5.1.1.2-B Communications (telephone) port disturbances
Testing SHALL be conducted in accordance with the telephone port stress testing specified in industry-recognized standards developed for telecommunications in general, particularly equipment connected to landline telephone service providers.
Voting systems, by being connected to the outside service provider via premises wiring, can be exposed to a variety of electromagnetic disturbances. These have been classified as emissions from adjacent equipment, lightning-induced, power-fault induced, power contact, Electrical Fast Transient (EFT), and steady-state induced voltage.
5.1.1.2-B.1 Emissions from other connected equipment
Testing SHALL be conducted in accordance with the emissions limits stipulated for other equipment of the voting system connected to the premises wiring of the polling place.
Emission limits for the power port of voting systems are discussed in Requirement Part 1: 6.3.4.2-B.1 with reference to numerical values stipulated in [Telcordia06]. EMC of a complete voting system installed in a polling facility thus implies that individual components of voting systems must demonstrate immunity against disturbances at a level equal to the limits stipulated for emissions of adjacent pieces of equipment.
Source:	[Telcordia06] subclause 3.2.3
5.1.1.2-B.2 Lightning-induced disturbances
Testing SHALL be conducted in accordance with the requirements of Telcordia GR-1089 [Telcordia06] for simulation of lightning.
Telcordia GR-6089 [Telcordia06] lists two types of tests, respectively (First-Level Lightning Surge Test and Second-Level Lightning Surge Test), as follows:
A) First-Level Lightning Surge Test
The particular voting system piece of equipment under test (generally referred to as "EUT") is placed in a complete operating system performing its intended functions, while monitoring proper operation, with checks performed before and after the surge sequence. Manual intervention or power cycling is not permitted before verifying proper operation of the voting system.
B) Second-Level Lightning Surge Test
Second-level lightning surge test is performed as a fire hazard indicator with cheesecloth applied to the particular EUT.
This second-level test, which can be destructive, may be performed with the EUT operating at a sub-assembly level equivalent to the standard system configuration, by providing dummy loads or associated equipment equivalent to what would be found in the complete voting system, as assembled in the polling place.
Source:	[Telcordia06] subclauses 4.6.7 and 4.6.8
5.1.1.2-B.3 Power faults-induced disturbances
Testing SHALL be conducted in accordance with the requirements of Telcordia GR-1089 [Telcordia06] for simulation power-faults-induced events.
Tests that can be used to assess the immunity of voting systems to power fault-induced disturbances are described in detail in [Telcordia06] for several scenarios and types of equipment, each involving a specific configuration of the test generator, test circuit, and connection of the equipment.
Source:	[Telcordia06] subclause 4.6
5.1.1.2-B.4 Power contact disturbances
Testing SHALL be conducted in accordance with the requirements of Telcordia GR-1089 [Telcordia06] for simulation of power-contact events.
Tests for power contact (sometimes called "power cross") immunity of voting systems immunity are described in detail in [Telcordia06] for several scenarios and types of equipment, each involving a specific configuration of the test generator, test circuit, and connection of the equipment.
5.1.1.2-B.5 Electrical Fast Transient (EFT)
Testing SHALL be conducted in accordance with the requirements of Telcordia GR-1089 [Telcordia06] for application of the EFT Burst.
Telcordia GR-1089 [Telcordia06] calls for performing EFT tests but refers to [ISO4b] for details of the procedure. While EFT generators, per the IEC standard [ISO4b], offer the possibility of injecting the EFT burst into a power port by means of coupling capacitors, the other method described by the IEC standard, the so-called "capacitive coupling clamp," would be the recommended method for coupling the burst into leads connected to the telephone port of the voting system under test. However, because the leads (subscriber wiring premises) vary from polling place to polling place, a more repeatable test is direct injection at the telephone port via the coupling capacitors.
Source:	[ISO04b] clause 6
5.1.1.2-B.6 Steady-state induced voltage
Testing SHALL be conducted in accordance with the requirements of Telcordia GR-1089 [Telcordia06] for simulation of steady-state induced voltages.
Telcordia GR-1089 [Telcordia06] describes two categories of tests, depending on the length of loops, the criterion being a loop length of 20 kft (sic). For metric system units, that criterion may be considered to be 6 km, a distance that can be exceeded for some low-density rural or suburban locations of a polling place. Therefore, the test circuit to be used should be the one applying the highest level of induced voltage.
Source:	[Telcordia06] sub-clause 5.2
5.1.1.2-C Interaction between power port and telephone port
Inherent immunity against data corruption and hardware damage caused by interaction between the power port and the telephone port SHALL be demonstrated by applying a 0.5 µs – 100 kHz Ring wave between the power port and the telephone port.
Although IEEE is in the process of developing a standard (IEEE PC62.50) to address the interaction between the power port and communications port, no standard has been promulgated at this date, but published papers in peer-reviewed literature [Key94] suggest that a representative surge can be the Ring Wave of [IEEE02a] applied between the equipment grounding conductor terminal of the voting system component under test and each of the tip and ring terminals of the voting system components intended to be connected to the telephone network.
Inherent immunity of the voting system might have been achieved by the manufacturer, as suggested in PC62.50, by providing a surge-protective device between these terminals that will act as a temporary bond during the surge, a function which can be verified by monitoring the voltage between the terminals when the surge is applied.
The IEEE project is IEEE PC62.50 "Draft Standard for Performance Criteria and Test Methods for Plug-in, Portable, Multiservice (Multiport) Surge Protective Devices for Equipment Connected to a 120/240 V Single Phase Power Service and Metallic Conductive Communication Line(s)." This is an unapproved standard, with estimated approval date 2008.
5.1.1.3	Radiated disturbances immunity
5.1.1.3-A Electromagnetic field immunity (80 MHz to 6.0 GHz)
Testing SHALL be conducted according to procedures in CISPR 24 [ANSI97], and either IEC 61000-4-3 [ISO06a] or IEC 61000-4-21:2003 [ISO06d].
IEC 61000-4-3 [ISO06a] specifies using an absorber lined shielded room (fully or semi anechoic chamber) to expose the device-under-test. An alternative procedure is the immunity testing procedures of IEC [ISO06d], performed in a reverberating shielded room (radio-frequency reverberation chamber).
Source:	[ANSI97], [ISO06a], [ISO06d]
5.1.1.3-B Electromagnetic field immunity (150 kHz to 80 MHz)
Testing for electromagnetic fields below 80 MHz SHALL be conducted according to procedures defined in IEC 61000-4-6 [ISO06b].
Source:	[FCC07], [ISO06b]
5.1.1.3-C Electrostatic discharge immunity
Testing SHALL be conducted in accordance with the recommendations of ANSI Std C63.16 [ANSI93], applying an air discharge or a contact discharge according to the nature of the enclosure of the voting system.
Applies to:	Electronic device DISCUSSION
Electrostatic discharges, simulated by a portable ESD simulator, involve an air discharge that can upset the logic operations of the circuits, depending on their status. In the case of a conducting enclosure, the resulting discharge current flowing in the enclosure can couple with the circuits and also upset the logic operations. Therefore, it is necessary to apply a sufficient number of discharges to significantly increase the probability that the circuits will be exposed to the interference at the time of the most critical transition of the logic. This condition can be satisfied by using a simulator with repetitive discharge capability while a test operator interacts with the voting terminal, mimicking the actions of a voter or initiating a data transfer from the terminal to the local tabulator.
Source:	[ANSI93], [ISO01]
5.1.2	Electromagnetic compatibility (EMC) emissions limits
Testing of voting systems for EMC emission limits will be conducted using the black box testing approach, which "ignores the internal mechanism of a system or component and focuses solely on the outputs generated in response to selected inputs and execution conditions" [IEEE00].
It will be necessary to subject voting systems to a regimen of tests to demonstrate compliance with emission limits. The tests should include most, if not all disturbances that might be expected to be emitted from the implementation under test, unless compliance with mandatory limits such as FCC regulations is explicitly stated for the implementation under test.
<a href="../comments/createComment?section=p3-5.1.2"> Add a Comment</a>
5.1.2.1 Conducted emissions limits
5.1.2.1.1	Power port – low/high frequency ranges
As discussed in Part 1: 6.3.5 "Electromagnetic Compatibility (EMC) emission limits", the relative importance of low-frequency harmonic emissions and the current drawn by other loads in the polling place will result in a negligible percentage of harmonics at the point of common connection, as discussed in [IEEE92]. Thus, no test is required to assess the harmonic emission of a voting station.
High-frequency emission limits have been established by Federal Regulations [FCC07] as a condition for offering equipment to the US market. In such cases, part of the requirements include affixing a label or notice stating that the equipment complies with the stipulated limits. Therefore, the VVSG does not suggest performing a redundant test.
<a href="../comments/createComment?section=p3-5.1.2.1.1"> Add a Comment</a>
5.1.2.1.2	Communications (Telephone) port
5.1.2.1-A Communications port emissions
Unintended conducted emissions from a voting system telephone port SHALL be tested for its analog voice band leads in the metallic as well as its longitudinal voltage limits.
Telcordia GR-1089 [Telcordia06] stipulates limits for both the common mode (longitudinal) and differential mode (metallic) over a frequency range defined by maximum voltage and terminating impedances.
5.1.2.2 Radiated emissions
5.1.2.2-A Radiated emission limits
Compliance with emission limits SHALL be documented on the hardware in accordance with the stipulations of FCC Part 15, Class B [FCC07].
Source:	[FCC07]
5.1.3 Other (non-EMC) industry-mandated requirements
5.1.3.1 Dielectric stresses
5.1.3.1-A Dielectric withstand
Testing SHALL be conducted in accordance with the stipulations of industry-consensus telephone requirements of Telcordia GR-1089 [Telcordia06].
Source:	[Telcordia06] Section 4.9.5
5.1.3.2 Leakage via grounding port
5.1.3.2-A Leakage current via grounding port
Simple verification of an acceptable low leakage current SHALL be performed by powering the voting system under test via a listed Ground-Fault Circuit Interrupter (GFCI) and noting that no tripping of the GFCI occurs when the voting system is turned on.
5.1.3.3 Safety
The presence of a listing label (required by authorities having jurisdiction) referring to a safety standard, such as [UL05], makes repeating the test regimen unnecessary. Details on the safety considerations are addressed in Part 1: 3.2.8.2 "Safety".
5.1.3.4 Label of compliance
Some industry mandated requirements require demonstration of compliance, while for others the manufacturer affixes of label of compliance, which then makes repeating the tests unnecessary and economically not justifiable.
5.1.4 Non-operating environmental testing
This type of testing is designed to assess the robustness of voting systems during storage between elections and during transporting between the storage facility and the polling place.
Such testing is intended to simulate exposure to physical shock and vibration associated with handling and transportation of voting systems between a jurisdiction's storage facility and polling places. The testing additionally simulates the temperature and humidity conditions that may be encountered during storage in an uncontrolled warehouse environment or precinct environment. The procedures and conditions of this testing correspond to those of MIL-STD-810D, "Environmental Test Methods and Engineering Guidelines."
5.1.4-A Tests of non-operating equipment
All voting systems SHALL be tested in accordance with the appropriate procedures of MIL-STD-810D, "Environmental Test Methods and Engineering Guidelines'' [MIL83].
Applies to: Voting system
Source:	[VVSG2005]
Note that the latest revision of MIL-STD-810 is revision F (dated 1 January 2000). The most recent change notice (Notice #3) for that standard is dated 5 May 2003. Recommend that this requirement be updated to indicate the most recent revision of this standard. (This latest revision may result in impacts to the requirements in the sub-sections below 5.1.4-A.)
5.1.4-A.1 Bench handling
All voting systems SHALL be tested in accordance with MIL-STD-810D, Method 516.3. Procedure VI.
This test simulates stresses faced during maintenance and repair.
5.1.4-A.2 Vibration
All voting systems SHALL be tested in accordance with MIL-STD-810D, Method 514.3, Category 1 – Basic Transportation, Common Carrier.
This test simulates stresses faced during transport between storage locations and polling places.
5.1.4-A.3 Storage temperature
All voting systems SHALL be tested in accordance with MIL-STD-810D: Method 502.2, Procedure I – Storage and Method 501.2, Procedure I – Storage. The minimum temperature SHALL be -4 degrees F, and the maximum temperature SHALL be 140 degrees F.
This test simulates stresses faced during storage.
5.1.4-A.4 Storage humidity
All voting systems SHALL be tested in accordance with humidity testing specified by MIL-STD-810D: Method 507.2, Procedure II – Natural (Hot-Humid), with test conditions that simulate a storage environment.
This test is intended to evaluate the ability of voting equipment to survive exposure to an uncontrolled temperature and humidity environment during storage.
5.1.5 Operating environmental testing
This type of testing is designed to assess the robustness of voting systems during operation.
5.1.5-A Tests of operating equipment
5.1.5-A.1 Operating temperature
All voting systems SHALL be tested according to the low temperature and high temperature testing specified by MIL-STD-810-D [MIL83]: Method 502.2, Procedure II -- Operation and Method 501.2, Procedure II -- Operation, with test conditions that simulate system operation.
5.1.5-A.2 Operating humidity
All voting systems SHALL be tested according to the humidity testing specified by MIL-STD-810-D: Method 507.2, Procedure II – Natural (Hot –Humid), with test conditions that simulate system operation.
5.2 Functional Testing
Functional testing is performed to confirm the functional capabilities of a voting system. The accredited test lab designs and performs procedures to test a voting system against the requirements outlined in Part 1. Additions or variations in testing may be appropriate depending on the system's use of specific technologies and configurations, the system capabilities, and the outcomes of previous testing.
Functional tests cover the full range of system operations. They include tests of fully integrated system components, internal and external system interfaces, usability and accessibility, and security. During this process, election management functions, ballot-counting logic, and system capacity are exercised.
The accredited test lab tests the interface of all system modules and subsystems with each other against the manufacturer's specifications. For systems that use telecommunications capabilities, components that are located at the poll site or separate vote counting site are tested for effective interface, accurate vote transmission, failure detection, and failure recovery. For voting systems that use telecommunications lines or networks that are not under the control of the manufacturer (e.g., public telephone networks), the accredited test lab tests the interface of manufacturer-supplied components with these external components for effective interface, vote transmission, failure detection, and failure recovery.
The security tests focus on the ability of the system to detect, prevent, log, and recover from a broad range of security risks. The range of risks tested is determined by the design of the system and potential exposure to risk. Regardless of system design and risk profile, all systems are tested for effective access control and physical data security. For systems that use public telecommunications networks to transmit election management data or election results (such as ballots or tabulated results), security tests are conducted to ensure that the system provides the necessary identity-proofing, confidentiality, and integrity of transmitted data. The tests determine if the system is capable of detecting, logging, preventing, and recovering from types of attacks known at the time the system is submitted for qualification. The accredited test lab may meet these testing requirements by confirming the proper implementation of proven commercial security software.
5.2.1.1 General test template
Most tests will follow this general template. Different tests will elaborate on the general template in different ways, depending on what is being tested.
Establish initial state (clean out data from previous tests, verify resident software/firmware);
Program election and prepare ballots and/or ballot styles;
Generate pre-election audit reports;
Configure voting devices;
Run system readiness tests;
Generate system readiness audit reports;
Precinct count only:
Open poll;
Run precinct count test ballots; and
Run central count test ballots (central count / absentee ballots only);
Generate in-process audit reports;
Generate data reports for the specified reporting contexts;
Inspect ballot counters; and
Inspect reports.
5.2.1.2 General pass criteria
5.2.1.2-A Applicable tests
The test lab need only consider tests that apply to the classes specified in the implementation statement, including those tests that are designated for all systems. The test verdict for all other tests SHALL be Not Applicable.
Under 5.2.1.2, even though the title of the chapter is "General Pass Criteria", none of the subsections to 5.2.1.2 defines criteria for "Pass". Recommend adding a section defining the criteria for "Pass" -- unless (a) applicability of tests, (b) test assumptions, (c) missing functionality, and (d) demonstratable violations comprises an absolute finite list of conditions considered "non-pass."
5.2.1.2-B Test assumptions
If the documented assumptions for a given test are not met, the test verdict SHALL be Waived and the test SHALL NOT be executed.
5.2.1.2-C Missing functionality
If the test lab is unable to execute a given test because the system does not support functionality that is required per the implementation statement or is required for all systems, the test verdict SHALL be Fail.
5.2.1.2-D Any demonstrable violation justifies an adverse opinion
A demonstrable violation of any applicable requirement of the VVSG during the execution of any test SHALL result in a test verdict of Fail.
The nonconformities observed during a particular test do not necessarily relate to the purpose of that test. This requirement clarifies that a nonconformity is a nonconformity, regardless of whether it relates to the test purpose.
See Part 3: 2.5.5 "Test practices" for directions on termination, suspension, and resumption of testing following a verdict of Fail.
5.2.2 Structural coverage (white-box testing)
This section specifies requirements for "white-box" (glass-box, clear-box) testing of voting system logic.
For voting systems that reuse components or subsystems from previously tested systems, the test lab may, per Requirement Part 2: 5.1-D, find it unnecessary to repeat instruction, branch, and interface testing on the previously tested, unmodified components. However, the test lab must fully test all new or modified components and perform what regression testing is necessary to ensure that the complete system remains compliant.
5.2.2-A Instruction and branch testing
The test lab SHALL execute tests that provide coverage of every accessible instruction and branch outcome in application logic and border logic.
This is not exhaustive path testing, but testing of paths sufficient to cover every instruction and every branch outcome.
Full coverage of third-party logic is not mandated because it might include a large amount of code that is never used by the voting application. Nevertheless, the relevant portions of third-party logic should be tested diligently.
There should be no inaccessible code in application logic and border logic other than defensive code (including exception handlers) that is provided to defend against the occurrence of failures and "can't happen" conditions that cannot be reproduced and should not be reproducible by a test lab.
Source:	Clarification of [VSS2002]/[VVSG2005] II.6.2.1 and II.A.4.3.3
The voting system application software should not contain "...a large amount of code that is never used by the voting application." In fact, it should not contain any code that is not used by the voting application. All code in the voting application should only exist because it satisfies a requirement. If exception handlers in the source code cannot be logically invoked, recommend that it must be determined if any of this code is "deactivated code" or whether it is "dead code." If it is "deactivated code," evidence should be made available by the manufacturer that the deactivated code is disabled for the environments where its use is not intended. Unintended activation of deactivated code due to abnormal system conditions is the same as unintended activation of activated code. A combination of analysis and testing should show that the means by which such code could be inadvertantly executed are prevented, isolated, or eliminated. "Dead code" is executable code which, as a result of a design error cannot be executed or used in an operational configuration of the target computer environment and is not traceable to a system or software requirement. The "dead code" should be removed and an analysis performed to assess the effect and the need for reverification.
While this cites the VSS and VVSG this is not a clarification but a new requirement. The requirement for 100% coverage of every accessible instruction and outcome processing during unit test is not achievable in a finite amount of time. Based upon personal experience of flight software the level of scope identified in this requirement exceeds industry best practices for high reliability commercial products. We suggest reconciling this requirement with this type of product.
What is meant by "this is not exhaustive path testing"? I believe that "coverage of every accessible instruction and branch outcome" requires instrumented code and/or unit testing procedures which can be exhaustive. Would recommend that this be written as to indicate that all commonly used instruction and branch outcomes be exercised or something that more effectively communicates the scope expected.
5.2.2-B Interface testing
The test lab SHALL execute tests that test the interfaces of all application logic and border logic modules and subsystems, and all third-party logic modules and subsystems that are in any way used by application logic or border logic.
Source:	Clarification of [VSS2002]/[VVSG2005] II.6.3
5.2.2-C Pass criteria for structural testing
The test lab SHALL define pass criteria using the VVSG (for standard functionality) and the manufacturer-supplied system documentation (for implementation-specific functionality) to determine acceptable ranges of performance.
Because white-box tests are designed based on the implementation details of the voting system, there can be no canonical test suite. Pass criteria must always be determined by the test lab based on the available specifications.
Since the nature of the requirements specified by the manufacturer-supplied system documentation is unknown, conformity for implementation-specific functionality may be subject to interpretation. Nevertheless, egregious disagreements between the behavior of the system and the behavior specified by the manufacturer should lead to a defensible adverse finding.
Source:	[VSS2002]/[VVSG2005] II.A.4.3.3
5.2.3 Functional coverage (black-box testing)
All voting system logic, including any embedded in COTS components, is subject to functional testing.
For voting systems that reuse components or subsystems from previously tested systems, the test lab may, per Requirement Part 2: 5.1-D, find it unnecessary to repeat functional testing on the previously tested, unmodified components. However, the test lab must fully test all new or modified components and perform what regression testing is necessary to ensure that the complete system remains compliant.
5.2.3-A Functional testing, VVSG requirements
The test lab SHALL execute test cases that provide coverage of every applicable, mandatory ("SHALL"), functional requirement of the VVSG.
Depending upon the design and intended use of the voting system, all or part of the functions listed below must be tested:
Ballot preparation subsystem;
Test operations performed prior to, during, and after processing of ballots, including:
Logic tests to verify interpretation of ballot styles, and recognition of precincts to be processed;
Accuracy tests to verify ballot reading accuracy;
Status tests to verify equipment statement and memory contents;
Report generation to produce test output data; and
Report generation to produce audit data records.
Procedures applicable to equipment used in the polling place for:
Opening the polls and enabling the acceptance of ballots;
Maintaining a count of processed ballots;
Monitoring equipment status;
Verifying equipment response to operator input commands;
Generating real-time audit messages;
Closing the polls and disabling the acceptance of ballots;
Generating election data reports;
Transfer of ballot counting equipment, or a detachable memory module, to a central counting location; and
Electronic transmission of election data to a central counting location.
Procedures applicable to equipment used in a central counting place:
Initiating the processing of a ballot deck, programmable memory device, or other applicable media for one or more precincts;
Verifying interaction with peripheral equipment, or other data processing systems;
Generating precinct-level election data reports;
Generating summary election data reports;
Transfer of a detachable memory module to other processing equipment;
Electronic transmission of data to other processing equipment; and
Producing output data for interrogation by external display devices.
Security controls have been implemented, are free of obvious errors, and operating as described in security documentation.
Setup inspection;
System integrity management;
Audit, electronic, and paper records; and
System event logging.
This requirement is derived from [VSS2002]/[VVSG2005] II.A.4.3.4, "Software Functional Test Case Design," in lieu of a canonical functional test suite. Once a complete, canonical test suite is available, the execution of that test suite will satisfy this requirement. For reproducibility, use of a canonical test suite is preferable to development of custom test suites
In those few cases where requirements specify "fail safe" behaviors in the event of freak occurrences and failures that cannot be reproduced and should not be reproducible by a test lab, the requirement is considered covered if the test campaign concludes with no occurrences of an event to which the requirement would apply. However, if a triggering event occurs, the test lab must assess conformity to the requirement based on the behaviors observed.
Source: [VSS2002]/[VVSG2005] II.A.4.3.4
<a href="../comments/createComment?section=p3-5.2.3-A"> Add a Comment</a>
Comment by alan (General Public)
gather previous cards used in previous election and use these as test samples to be input in test cycle. gather people off the street and ask them to fill in the test cards as a sample test set give no instructions on how to fill in the cards, all fill in instructions should be on the material presented. testers/developers/team members will fill in the cards in ways that the public will not, so get a sample from the public.
5.2.3-B Functional testing, capacity tests
The test lab SHALL execute tests to verify that the system and its constituent devices are able to operate correctly at the limits specified in the implementation statement; for example:
Maximum number of ballots;
Maximum number of ballot positions;
Maximum number of ballot styles;
Maximum number of contests;
Maximum vote total (counter capacity);
Maximum number of provisional, challenged, or review-required ballots;
Maximum number of contest choices per contest; and
Any similar limits that apply.
See Part 1: 2.4 "implementation statement". Every kind of limit is not applicable to every kind of device. For example, EBMs may not have a limit on the number of ballots they can handle.
Source:	Generalization from [VSS2002]/[VVSG2005] II.6.2.3
<a href="../comments/createComment?section=p3-5.2.3-B"> Add a Comment</a>
Comment by Carolyn Coggins (Voting System Test Laboratory)
This requirement is a problem in the earlier standards and 5.2.3-B and 5.2.3-B1 still does not provide sufficient guidance to ensure consistency of testing across all VSTLs. Specific benchmarks needs to be provided to the labs, manufacturers and election officials on acceptable stress test limits for a thru g. These guidelines can provide a benchmark for 5.2.3.B-1 on what is a practical test. Further identify: 1) A matrix of what limits are applicable to which voting systems 2) If the manufacturer may set a limit lower than the acceptable stress test limit 3) What/where must this information be documented (provide a reference if it is already identified)
Comment by Frank Padilla (Voting System Test Laboratory)
Who sets the limits and determines if they are accurate or cover enough?
5.2.3-B.1 Practical limit on capacity operational tests
If an implementation limit is sufficiently great that it cannot be verified through operational testing without severe expense and hardship, the test lab SHALL attest this in the test report and substitute a combination of design review, logic verification, and operational testing to a reduced limit.
For example, since counter capacity can easily be designed to 232 and beyond without straining current technology, some reasonable limit for required operational testing is needed. However, it is preferable to test the limit operationally if there is any way to accomplish it.
<a href="../comments/createComment?section=p3-5.2.3-B.1"> Add a Comment</a>
This is a perfect example of a situation in which a test fixture can drive the system to a limit that is impractical to reach with end-to-end testing. We should permit the use of risk-focused tests that are not end-to-end as a more desirable alternative to testing with reduced limits. .......... (Affiliation Note: IEEE representative to TGDC)
5.2.3-C Functional testing, stress tests
The test lab SHALL execute tests to verify that the system is able to respond gracefully to attempts to process more than the expected number of ballots per precinct, more than the expected number of precincts, higher than expected volume or ballot tabulation rate, or any similar conditions that tend to overload the system's capacity to process, store, and report data.
In particular, Requirement Part 1: 7.5.6-A should be verified through operational testing if the limit is practically testable.
Source: [VSS2002]/[VVSG2005] II.A.4.3.5
<a href="../comments/createComment?section=p3-5.2.3-C"> Add a Comment</a>
"Gracefully" is not a testable requirement. Please identify testable pass/fail criteria.
"Process more than the number of" is subjective and not testable or repeatable across labs.
5.2.3-D Functional testing, volume test
The test lab SHALL conduct a volume test in conditions approximating normal use in an election. The entire system SHALL be tested, from election definition through the reporting and auditing of final results.
Data collected during this test contribute substantially to the evaluations of reliability, accuracy, and misfeed rate (see Part 3: 5.3 "Benchmarks").
Source:	[CA06]
<a href="../comments/createComment?section=p3-5.2.3-D"> Add a Comment</a>
Subjective. What is normal use in an election? This needs to be defined.
Comment by ACCURATE (Aaron Burstein) (Academic)
Volume testing is a vital element of future certification with respect to voting system reliability. It simulates the load that a typical machine might encounter during its peak use period and does so on many devices at once. It has become important in California, at least; but state-level volume testing can never be as instrumentally effective as volume testing performed during national certification. Flaws found during national certification can be fixed immediately and the system re-certified during the ongoing certification process instead of having to re-submit a delta change under a new certification attempt. Thus, this requirement should be adopted.
5.2.3-D.1 Volume test, vote-capture devices
For systems that include VEBDs, a minimum of 100 VEBDs SHALL be tested and a minimum of 110 ballots SHALL be cast manually on each VEBD.
Applies to: VEBD
For vote-by-phone systems, this would mean having 100 concurrent callers, not necessarily 100 separate servers to answer the calls, if one server suffices to handle many incoming calls simultaneously. Other client-server systems would be analogous.
To ensure that the correct results are known, test voters should be furnished with predefined scripts that specify the votes that they should cast.
<a href="../comments/createComment?section=p3-5.2.3-D.1"> Add a Comment</a>
100 voting terminals is not a practical number of units for several reasons: ·	? Physical space requirements ·	? Power requirements ·	? Availability from manufacturer A possible solution is a discrete event simulation model developed by the manufacturer and verified for accuracy by the VSTL.
5.2.3-D.2 Volume test, precinct tabulator
For systems that include precinct tabulators, a minimum of 50 precinct tabulators SHALL be tested. No fewer than 10000 test ballots SHALL be used. No fewer than 400 test ballots SHALL be counted by each precinct tabulator.
Applies to: Precinct tabulator
[GPO90] 7.5 specified, "The total number of ballots to be processed by each precinct counting device during these tests SHALL be at least ten times the number of ballots expected to be counted on a single device in an election (500 to 750), but in no case less than 5,000."
It is permissible to reuse test ballots. However, all 10000 test ballots must be used at least once, and each precinct tabulator must count at least 400 (distinct) ballots. Cycling 100 ballots 4 times through a given tabulator would not suffice. See also, Requirement Part 3: 2.5.3-A (Complete system testing).
<a href="../comments/createComment?section=p3-5.2.3-D.2"> Add a Comment</a>
Is the number of test units supportable and representative of the requirements?
50 tabulators is not a practical number of units for several reasons: ·	? Physical space requirements ·	? Power requirements ·	? Availability from manufacturer A possible solution is a discrete event simulation model developed by the manufacturer and verified for accuracy by the VSTL.
5.2.3-D.3 Volume test, central tabulator
For systems that include central tabulators, a minimum of 2 central tabulators SHALL be tested. No fewer than 10000 test ballots SHALL be used. A minimum ballot volume of 75000 (total across all tabulators) SHALL be tested, and no fewer than 10000 test ballots SHALL be counted by each central tabulator.
Applies to: Central tabulator
[CA06] did not specify test parameters for central tabulators. The test parameters specified here are based on the smallest case provided for central count systems in Exhibit J-1 of Appendix J, Acceptance Test Guidelines for P&M Voting Systems, of [GPO90]. An alternative would be to derive test parameters from the test specified in [GPO90] 7.3.3.2 and (differently) in [VSS2002]/[VVSG2005] II.4.7.1. A test of duration 163 hours with a ballot tabulation rate of 300 / hour yields a total ballot volume of 48900—presumably, but not necessarily, on a single tabulator.
[GPO90] 7.5 specified, "The number of test ballots for each central counting device SHALL be at least thirty times the number that would be expected to be voted on a single precinct count device, but in no case less than 15,000."
The ballot volume of 75000 is the total across all tabulators; so, for example, one could test 25000 ballots on each of 3 tabulators. The test deck must contain at least 10000 ballots. A deck of 15000 ballots could be cycled 5 times to generate the required total volume. See also, Requirement Part 3: 2.5.3-A (Complete system testing).
Source:	[GPO90] Exhibit J-1 (Central Count)
<a href="../comments/createComment?section=p3-5.2.3-D.3"> Add a Comment</a>
5.2.3-D.4 Test imperfect marks and folds
The testing of MCOS SHALL include marks filled according to the recommended instructions to voters, imperfect marks as specified in Requirement Part 1: 7.7.5-D, and ballots with folds that do not intersect with voting targets.
Applies to: MCOS
Source:	Numerous public comments and issues
<a href="../comments/createComment?section=p3-5.2.3-D.4"> Add a Comment</a>
test with dull #2 pencil test with black ink pen test with blue ink pen test where X is marked instead of circle filled in test if two circles filled in test if single line drawn through circle (all directions - | \ /) test with one circle is partially filled and the second circle on same line is filled completely test if mark is erased test if mark is partially erased test if mark is filled in and X'd out and other circle is filled in test if circle is ripped, torn or punctured test if circle is missing (bad form is used) test if circle is partially printed (bad form is used)
Requirement is subjective. How are these imperfect marks to be input?
5.2.3-E Functional testing, languages
The test lab SHALL execute tests to verify that the system is able to produce and utilize ballots in all of the languages that are claimed to be supported in the implementation statement.
See Part 1: 2.4 "Implementation Statement".
<a href="../comments/createComment?section=p3-5.2.3-E"> Add a Comment</a>
5.2.3-F Functional testing, error cases
The test lab SHALL execute tests to verify that the system is able to detect, handle, and recover from abnormal input data, operator actions, and conditions.
See Requirement Part 1: 6.4.1.8-A and Part 1: 6.4.1.9.
<a href="../comments/createComment?section=p3-5.2.3-F"> Add a Comment</a>
Subjective: "abnormal input data" is not testable or repeatable.
5.2.3-F.1 Procedural errors
The test lab SHALL execute tests to verify that the system detects and handles operator errors such as inserting control cards out of sequence or attempting to install configuration data that are not properly coded for the device.
Source:	[GPO90] 8.8
<a href="../comments/createComment?section=p3-5.2.3-F.1"> Add a Comment</a>
Testing should not be constrained to "sunny day"/positive tests. Testing should include negative test cases. Test cases should validate both the positive and negative aspects of a requirement. There should be no minimum number of tests (both positive/negative).
5.2.3-F.2 Hardware failures
The test lab SHALL execute tests to check that the system is able to respond to hardware malfunctions in a manner compliant with the requirements of Part 1: 6.4.1.9 "Recovery".
This capability may be checked by any convenient means (e.g., power off, disconnect a cable, etc.) in any equipment associated with ballot processing.
This test pertains to "fail safe" behaviors as discussed in Requirement Part 3: 5.2.3-A. The test lab may be unable to produce a triggering event, in which case the test is passed by default.
Source:	[GPO90] 8.5
<a href="../comments/createComment?section=p3-5.2.3-F.2"> Add a Comment</a>
5.2.3-F.3 Communications errors
For systems that use networking and/or telecommunications capabilities, the test lab SHALL execute tests to check that the system is able to detect, handle, and recover from interference with or loss of the communications link.
Source: [VSS2002]/[VVSG2005] II.6.3
<a href="../comments/createComment?section=p3-5.2.3-F.3"> Add a Comment</a>
5.2.3-G Functional testing, manufacturer functionality
The test lab SHALL execute tests that provide coverage of the full range of system functionality specified in the manufacturer's documentation, including functionality that exceeds the specific requirements of the VVSG.
Source: [VSS2002]/[VVSG2005] II.3.2.3, II.6.7
<a href="../comments/createComment?section=p3-5.2.3-G"> Add a Comment</a>
5.2.3-H Functional test matrix
The test lab SHALL prepare a detailed matrix of VVSG requirements, system functions, and the tests that exercise them.
<a href="../comments/createComment?section=p3-5.2.3-H"> Add a Comment</a>
5.2.3-I Pass criteria for functional testing
Pass criteria for tests that are adopted from a canonical functional test suite are defined by that test suite. For all other tests, the test lab SHALL define pass criteria using the VVSG (for standard functionality) and the manufacturer-supplied system documentation (for implementation-specific functionality) to determine acceptable ranges of performance.
<a href="../comments/createComment?section=p3-5.2.3-I"> Add a Comment</a>
5.3.1 General method
Reliability, accuracy, and misfeed rate are measured using ratios, each of which is the number of some kind of event (failures, errors, or misfeeds, respectively) divided by some measure of voting volume. The test method discussed here is applicable generically to all three ratios; hence, this discussion will refer to events and volume without specifying a particular definition of either.
By keeping track of the number of events and the volume over the course of a test campaign, one can trivially calculate the observed cumulative event rate by dividing the number of events by the volume. However, the observed event rate is not necessarily a good indication of the true event rate. The true event rate describes the expected performance of the system in the field, but it cannot be observed in a test campaign of finite duration, using a finite-sized sample. Consequently, the true event rate can only be estimated using statistical methods.
In accordance with the current practice in voting system testing, the system submitted for testing is assumed to be a representative sample, so the variability of devices of the same type is out of scope.
The test method makes the simplifying assumption that events occur in a Poisson distribution, which means that the probability of an event occurring is assumed to be the same for each unit of volume processed. In reality, there are random events that satisfy this assumption but there are also nonrandom events that do not. For example, a logic error in tabulation software might be triggered every time a particular voting option is used. Consequently, a test campaign that exercised that voting option often would be more likely to indicate rejection based on reliability or accuracy than a test campaign that used different tests. However, since these VVSG require absolute correctness of tabulation logic, the only undesirable outcome is the one in which the system containing the logic error is accepted. Other evaluations specified in these VVSG, such as functional testing and logic verification, are better suited to detecting systems that produce nonrandom errors and failures. Thus, when all specified evaluations are used together, the different test method complement each other and the limitation of this particular test method with respect to nonrandom events is not bothersome.
For simplicity, all three cases (failures, errors, and misfeeds) are modeled using a continuous distribution (Poisson) rather than a discrete distribution (Binomial). In this application, where the probability of an event occurring within a unit of volume is small, the difference in results from the discrete and continuous models is negligible.
The problem is approached through classical hypothesis testing. The null hypothesis (H0) is that the true event rate, rt, is less than or equal to the benchmark event rate, rb (which means that the system is conforming).
The alternative hypothesis (H1) is that the true event rate, rt, is greater than the benchmark event rate, rb (which means that the system is non-conforming).
Assuming an event rate of r, the probability of observing n or less events for volume v is the value of the Poisson cumulative distribution function.
Let no be the number of events observed during testing and vo be the volume produced during testing. The probability α of rejecting the null hypothesis when it is in fact true is limited to be less than 0.1. Thus, H0 is rejected only if the probability of no or more events occurring given a (marginally) conforming system is less than 0.1. H0 is rejected if 1−P(no−1,rbvo)<0.1, which is equivalent to P(no−1,rbvo)>0.9. This corresponds to the 90th percentile of the distribution of the number of events that would be expected to occur in a marginally conforming system.
If at the conclusion of the test campaign the null hypothesis is not rejected, this does not necessarily mean that conformity has been demonstrated. It merely means that there is insufficient evidence to demonstrate non-conformity with 90 % confidence.
Calculating what has been demonstrated with 90 % confidence, after the fact, is completely separate from the test described above, but the logic is similar. Suppose there are no observed events after volume vo. Solving the equation P(no,rdvo)=0.1 for rd finds the "demonstrated rate" rd such that if the true rate rt were greater than rd, then the probability of having no or fewer events would be less than 0.1. The value of rd could be greater or less than the benchmark event rate rb mentioned above.
Please note that the length of testing is determined in advance by the approved test plan. To adjust the length of testing based on the observed performance of the system in the tests already executed would bias the results and is not permitted. A Probability Ratio Sequential Test (PRST) [Wald47][Epstein55][MIL96] as was specified in previous versions of these VVSG varies the length of testing without introducing bias, but practical difficulties result when the length of testing determined by the PRST disagrees with the length of testing that is otherwise required by the test plan.
<a href="../comments/createComment?section=p3-5.3.1"> Add a Comment</a>
The benchmarks are determined by the 'observed event"; however, this critical value is not defined. It is the basis for benchmarking and must be defendable. As it is currently stated this is not defendable.
Amendment 2 to: USACM Comment #25. Section 5.3.1 General method [incorrect] Reference [2] Musa, Software Reliability Engineering (http://members.aol.com/JohnDMusa/book.htm)
VVSG should not estimate reliability (or acceptability in any other way) of software by calculating the number of failures (test events?) divided by the number of tests (test volume?). It is too easy to influence this estimator by including large numbers of easy-to-pass tests. .......... (Affiliation Note: IEEE representative to TGDC)
Lab tests focused on conformance testing do not model usage patterns in the field and therefore test results based on them cannot estimate failure rates in the field. This is not a defensible method for estimating reliability. .......... It may be possible to develop operational profiles from which reliability tests could be developed but this will require extensive research that would not be part of the approval process of any particular voting system. .......... (Affiliation Note: IEEE representative to TGDC)
USACM Comment #25. Section 5.3.1 General method [incorrect] USACM recommends correcting the factual errors present in this subsection, as minimally enumerated below: 1. The system submitted for testing might be a representative sample of the pool of identical hardware/software systems, but the pool of tests should not be a representative sample of the events that happen during an election. 2. There is no reason to expect software reliability, software accuracy, and hardware misfeed rate to follow the same distribution. 3. The Poisson distribution is discrete, not continuous. 4. The Poisson process typically assumes a stationary underlying exponential distribution. The idea that software reliability, software accuracy, and hardware misfeed rates follow the same underlying distribution, or that the concatenation of these three (if there are only three) distributions would be anything like exponential is remarkable in its unlikelihood. 5. The observed event rate ("events" divided by "volume" over the course of a test campaign) is a highly biased measure. a. The first problem is that a regression test suite repeats the same tests from build to build. This gives rise to the classic problem of the "pesticide paradox" [ ]. The test suite is a tiny sample of the collection of possible tests. When the suite reveals bugs, they are fixed. Ultimately, the test suite becomes a collection of tests that have one thing in common: the software has passed all of them at least once. This differs from almost every other possible test (all of the ones that have not been run). Therefore, the reliability of the software is probably vastly overestimated. b. The second problem is that the pool of tests is structured to cover a specification. It does not necessarily target vulnerabilities of the software. Nor is it designed to reflect usage frequencies in the field [ ]. 6. Determining the length of testing in advance by an approved test plan sounds scientific, but many practitioners consider this software testing malpractice. There is substantial evidence that bugs cluster. Given a failure, there is reason to do follow-up testing to study this area of the product in more detail. Rigidly adhering to a plan created in the absence of failure data is to rigidly reject the idea of follow-up testing. This underestimates the number of problems in the code. Worse, this testing method reduces the chance that defects will be found and fixed because it reduces — essentially bans — the follow-up testing that would expose those additional defects.
VVSG should not model software failure rates with a Poisson distribution or a Poisson process, or with any other distribution or stochastic process unless that distribution or process is derived from a logically and empirically-defensible model. .......... There is no reason to think that a mixture distribution that combines hardware and software events would be a simple Poisson. It is important to recognize that we are estimating performance in the tails of the distribution. To the extent that the true underlying distribution differs from the Poisson, deviations are particularly likely in the tails, yielding overestimates or underestimates of the significance of a given number of failures in a given period of activity. The statistical tables may have validity for hardware-related failures or for misfeeds, but there is no reason to think they would be valid for software or for the mixture distribution. .......... Robert Austin [Measuring and Managing Performance in Organizations, Dorset, 1996] wrote a particularly compelling discussion of the risks associating with basing high-stakes decisions on metrics that are not tightly tied to the underlying attribute that the metric attempts to estimate. Equipment vendors have a strong interest in making their numbers look good, or at least good enough to pass testing, and therefore they have an incentive to optimize their behavior in ways that improve the numbers. They also have an incentive to challenge the ways in which the numbers (total failures, total volume) are calculated, to arrive at a result that is more favorable. .......... I assume in all that follows that the manufacturers are acting in good faith. When we tell someone that their performance will be passed or failed according to a criterion, there is nothing dishonest in optimizing efforts to meet that criterion. If anything, that is what the criterion is there to accomplish. In particular, you should read the comments that follow with the understanding that I am explicitly and intentionally assuming that the vendors will be factually honest in everything that they do and that they are primarily motivated to achieve a "pass" from the system and not particularly motivated to do so in such a way as to mislead anyone about the underlying quality of the product. .......... *	The VVSG (5.3.1) correctly notes that one of the characteristics of the Poisson model is that the probability of an event occurring stays constant over each "unit of volume processed." It then notes that this is not exactly correct for software because software errors might be nonrandom, that is, they might be triggered every time the same set of conditions is tested. It then dismisses this problem by saying "Thus, when all specific evaluations are used together, the different test methods complement each other and the limitation of the particular test method with respect to nonrandom events is not bothersome." I think this is a novel conclusion. I do not understand how mixing nonrandom events with random ones (to the extent that there are random failures in software) is a good foundation for a model that assumes that all events are random. .......... (a) For most software failures, the failure itself is not random at all. Given THESE conditions, THAT failure will occur. What might be thought of as random is whether and when the particular test that includes those conditions is presented to the software. The probability that a given test will yield a failure thus depends on at least two factors: how many problems remain in the software and how powerful the test is with respect to the types of problems that remain. As the software goes through testing, problems are fixed, and so the number of remaining problems diminishes. Therefore the assumption that the rate parameter of the Poisson distribution is stationary is implausible. .......... (b) The power of software tests run is not related to the underlying reliability of the software. Test power is (analogous to the power of a statistical test) the ability of the test to detect an error of a certain type if it is there. Note that no test has "absolute" power—a test that is optimized to expose an off-by-one error might be a weak detector of rounding errors. Thus, a lab can achieve a low failure rate (high apparent reliability) by running relatively low-power tests and a high failure rate (low apparent reliability) by running relatively high-power tests. Regression tests lose their power as they are used repeatedly, because the errors they are optimized to detect get found and fixed. (This problem was labeled the "pesticide paradox" by Boris Beizer in Software Testing Techniques, Van Nostrand, 2nd Edition, 1990; see also Kaner, Testing Computer Software, McGraw-Hill, 1987, p. 94). The improvement in apparent reliability with repeated use of regression tests should not be expected to predict improvement of reliability in the field, because users in the field do other things with the software beyond running these particular regression tests. Varying testing, for example by changing parameter values, combining tests, or running the tests in long random sequences, probably does a better job of mitigating operational risk but under the VVSG benchmarking definition, this testing will drive down estimated reliability at the same time as it contributes to the actual improvement of reliability. .......... (c)	If there are B bugs in the software and we find a bug with 100% certainty if we run a specific test (or ones sufficiently like it), the probability of detecting one of the errors boils down to the probability that an error-revealing test makes it into the test suite. That depends on the sampling strategy (any test design strategy can be seen as a sampling strategy), whose details are under the control of the test lab, with some influence by the vendor and the VVSG. It is not clear what this sampling strategy has to do with the underlying reliability of the software. The VVSG-specified rate parameter probably has more to do with this sampling strategy than with operational reality. .......... (d) A Poisson process model for failures makes several assumptions. The first that I noted is that the probability of discovery of a failure is constant over time. This is implausible because the program presumably gets more stable, and the tests (if they are the fully scripted regression tests required by VVSG) get less powerful over time. The second is that instants of time (or units of volume) (that is, tests) are independent. This is also implausible. A widely reported pattern in test data is that some modules are much more error prone than others. Presumably, this is due to the inherent difficulty of some problems, the tremendous variations in individual programmer competence, perhaps a difference in time pressure associated with completing some tasks compared to others, etc. A sensible testing strategy adds new tests to further investigate areas that have shown some failures. If this is done, the probability of these tests exposing problems is relatively high, but that is a conditional probability—test X2 has a high probability of exposing a problem because test X1 did expose a problem. This is precisely the opposite of the assumption of independence between X1 and X2. Of course, one can preserve apparent independence of tests by never adding new tests to more carefully study areas of the program that seem weak. However, if the objective is to check the quality of the software, this restriction (no follow-up with related tests) would be bad testing, in conflict with the objective. Another problem for the idea of independence is the problem of identicality. Do we really think that the same test, run a second or third or fourth time (regression testing) should be treated as an independent sample from the pool of possible tests? .......... (e) Another problem with the Poisson process model is that some bugs are inherently harder to detect than others. Thus, if we have B bugs, the probability of detecting each one is not the same. It is usually easier to find a bug if it depends only on one feature or one parameter of a feature. A relatively simple test will do the trick. The only risk of obscurity is the possibility that only one value of the parameter would lead to failure. Special cases do exist, but they are often (and not always) at boundaries that are either visible externally or on review of the code. VVSG requires testing at boundaries and therefore most (and not all) of the single-variable special-case bugs are probably covered. However, some bugs involve combinations of two, three, or more variables or functions. Some of those variables might be relative timing of events (race conditions) or amount of free memory when a given task is attempted or access to some other resource. These are harder to detect with simple tests. .......... (f)	Even the assumption that one test can only expose one defect is empirically challengeable. Unless a test is so successfully focused on the processing of one variable by one method that multiple problems are impossible (unit testing can achieve this, but not system testing), a given test might trip first over one feature and next over another. A test that combines 10 features might yield 10 (or more!) failures. This is not a merely theoretical possibility. It is a common heuristic in system testing that testing should start with single-feature tests and progress to relatively simple multi-feature combinations and then progress to user-meaningful rich scenarios. This is based on experience: companies that do mainly the multi-function scenario testing often find their tests blocked—a failure in the first steps of the test blocks continuation to the later steps. After the first bug is fixed, the test fails again, in a way that blocks further execution, and then when used again, it fails again. In the practitioner community, there are many anecdotes of bugs that should have been easy to find being found very late in testing because the planned test that finally exposed the bug was blocked by other bugs. Thus, we have commonplace examples of tests exposing many more than one defect. .......... In sum, there is no reason to think that any of the assumptions underlying a Poisson process model apply. .......... The VVSG provides a table for determining critical values associated with the ratio of the number of test events to the test volume. The idea is that even if the Poisson distribution is not a perfect estimator, perhaps it is a good first approximation. I am not a professional statistician, but I do have about 8-12 semesters of probability/statistics courses, a few more on modeling, and some practical research experience. My understanding is that if two distributions are similarly shaped, using one as an approximation of the other is possible—but the relative differences are likely to grow as you go out to the tails. That is, similar distributions often differ most in their assignment of probabilities to lower-probability events. A 90% criterion value is pretty far out in the tail of the distribution. If none of the assumptions of the Poisson model apply to software testing, it is hard to believe that numbers taken from the tail of that distribution accurately predict much about the system under test. .......... Here are some other problems associated with the VVSG's estimator of software reliability: .......... (1)	As far as I can tell, in its treatment of reliability estimation, the VVSG assumes that the test volume is a fixed value, not itself a random variable. This is only true if one set of tests is run once and no other tests or other events are considered. Given that there will be regression tests, this is not true. Even if we count each regression tests only once, no matter how many times it is run (but that is unfair if the same test later exposes a different bug), a competent test lab does additional testing around any bug reported fixed. That is, if a given set of test conditions exposes a failure, and the equipment vendor fixes the bug and returns a new version of the software, a competent tester will not only test the fix with the original test that exposed the bug but will create new tests to see whether the fix actually covered the underlying problem. These can expose new problems and so they must be new units of test volume. Of course, to preserve a fixed volume of testing, we could choose not to allow such testing. However, as in many cases considered above, it might be highly undesirable to allow an incorrect model to be used as an excuse to constrain the power of testing. .......... (2)	The VVSG assumes that the test results obtained in conformance testing can be used as a sound statistical estimator of the population reliability (see 5.3.1). This assumption is unreasonable. The reliability of the voting equipment software, in the field, will depend on how the software is used in the field. The tests designed for conformance testing are not designed with an objective of mapping to field usage. They are designed to achieve a level of simple coverage of the code, another level of simple coverage of documented requirements, another level of coverage of boundary values of individual variables, and so on. There are sound statistical methods available for estimating the reliability of the software in the field (see, for example, Musa, Software Reliability Engineering, McGraw-Hill, 1998), but they start from development of operational profiles—profiles of ways in which people will actually use the software. The next task is estimation of relative frequency of occurrence of each profile—a usage pattern twice as likely in the field should be involved in twice as many of the reliability tests. From here, one generates a large pool of tests, deriving each test from one of the profiles (varying specific parameters, or sequences of operation in ways consistent with the profile). Ideally, that generation should be itself driven by a random process that reflects usage patterns. From there, failure rate over the sample of tests might well be a valid estimator of field failure rate. If it is important to have a failure rate estimator, it is important to have a number that bears a defensible relationship to the underlying parameter. .......... (3)	Development of operational profiles is an expensive proposition. Some vendors (such as AT&T and Microsoft) have access to customer usage patterns and, at significant expense, can develop profiles on their own. It is not clear that voting equipment vendors have this level of access to customer usage of their own equipment. In addition, the better study might be of usage of voting equipment generally, across vendors. If the profiles are essentially the same across vendors and equipment models, the same profiles can be used with new models as they are introduced, rather than requiring a hugely burdensome (in time and money) research program for each new model. Rather than requiring voting equipment vendors to do this type of research, it might make more sense for NIST (or some other agency) to fund independent (e.g. university-based) research to develop such profiles and assess their commonality across devices. This will take some number of years. Until those profiles are developed and usable, I think it is inadvisable to predicate any decisions on estimators of software reliability. .......... (5)	If we assume that the TEST CAMPAIGN includes all tests done by the independent test lab, then the campaign includes all regression tests, no matter how many times these are repeated. Suppose that a given test is repeated ten times. When we compute the TEST VOLUME of the campaign, is this 10 tests or 1? .......... (6)	It is one thing to say that the lab cannot qualify a device based on the testing of a prototype. It is another thing to bar the vendor from submitting a prototype to the lab for evaluation. Evaluating prototypes gives the lab an opportunity to build expertise with the system under test, making its ultimate testing of the final version more effective. And it gives the vendor an opportunity to discover the weaknesses it is blind to, enabling it to fix problems earlier in the development cycle. It is widely believed in the software engineering community that earlier testing improves quality and reduces costs. While VVSG should not require vendors to submit early versions to the lab (there may be more cost effective ways to evaluate early versions), surely it should not ban it. If a vendor does submit an early version for testing, do those tests count as part of the test campaign? Do those failures count in the ultimate total of test events? .......... (7)	Suppose two equipment manufacturers have equivalent internal processes, in terms of the quality and functionality of their software, and (for simplicity) equivalent products. One submits its software to the independent test lab a little earlier in development than the other. The first submitter goes to the lab with a few more bugs and goes through one or more rounds of regression testing. Ultimately, the same bugs are found and fixed in both systems. Thus, at the end of testing, we have two equivalently reliable systems. What is the effect on the numbers? If the test campaign counts each time a regression test is run as a separate test, then the first submitter is increasing the measured test volume enormously by submitting early. If the product has only a few more bugs than the product submitted by the late submitter, then even though the first submitter’s test event total will be higher, its ratio of test events to test volume will be lower. In contrast, if regression tests are not counted twice but fixed bugs are counted as test events, then the incentive will go to the vendor who waits until the last possible minute to submit product to the lab. If we want VVSG to drive this strategy as a matter of policy, VVSG should explicitly consider and state the policy and the policy choice should be publicly reviewed. Instead, the method of calculation creates an implicit policy. .......... (8)	To the extent that test volume is left loosely defined, the estimated reliability will vary enormously depending on how the test lab (paid by the vendor) computes the test volume. A rational vendor would spend effort advocating for the largest possible interpretation of volume, so as to make the denominator as large as possible. .......... (9)	Consider applying a high-volume test strategy to the testing of the device. High-volume strategies have been used effectively for automotive software, telephone switching software, firmware in office automation products, and undoubtedly many other contexts. I will emphasize my own work below, and other work I am personally familiar with, not because I think it is the best in the field but because I can write with authority about the underlying observations. High-volume testing is a well-funded, fashionable area of work. Examples are state-model based testing that execute long sequences of sub-tests, each involving a controlled state transitions; testing using genetic algorithms; search-based testing, in which the test sequence involves test values chosen to be different from each other in a specified way (e.g. maximally dissimilar from the previous tests in the sequence); random-input tests or random-event tests in which a random source generates data or traffic for a long period or until the system crashes; and various types of extreme value attacks (heavy load, big input, extreme combinations, corrupted files) that string many individual tests into one long, grueling sequences of harsh tests. These are often done as security tests today, but they were seen as tests of robustness twenty years ago. These are not fundamentally new ideas. The concerns I raise below in the context of the testing types that I mention applies just as well to all of these other methods. Given that preamble, consider a specific example that I know well: suppose the lab applies long-sequence randomized regression testing (LSRRT), which McGee & Kaner ("Experiments with High Volume Test Automation," Workshop on Empirical Research in Software Testing, ACM SIGSOFT Software Engineering Notes, 29(5) 1-3 2004 discussed under the label "extended random regression"). In LSRRT, you take a set of tests that a particular build of the software has passed individually and string them together in an arbitrarily long random sequence. The key advantage of LSRRT over many other tests that push a device through very long sequences of tests is that the expected results of each test are known and therefore failures can be detected in terms of unexpected responses rather than waiting until the software crashes. A unexpected responses might be unexpected data, but it also might be unexpected behavioral timing. Oracle Corporation used a method like this in its early qualification of its database, for example. If a test that took T1 time to complete at one point in testing took T2 (much longer or much shorter) time a bit later, system engineers investigated the cause of the difference, often finding coding errors. (Unpublished oral personal communication from Bob Miner, 1987) As McGee & Kaner reported in their short case study summary, LSRRT exposed a large number of serious problems that were not being exposed by the individual tests themselves. Similarly, I have seen serious failures exposed in a different type of long-sequence testing by a PBX manufacturer whose code had gone through thorough unit testing. Stack corruption that built up over time, memory corruption triggered by particular subsequences of events or particular combinations of data, race conditions involving unexpected busy-ness of one of the processors in a multiprocessor system--these are examples of the kinds of problems exposed by long-sequence testing that are much harder to find by testing with one distinct functional test at a time. During an election, a voting system has to run without failure for many hours. Long-sequence testing addresses the question of operation over that long period. One-functional-test-at-a-time testing does not. Should a test lab employ this style of testing? If so, how should we count the test volume? At "Mentsville" (a fictitious name, requested by the well-known equipment manufacturer whose processes McGee and Kaner studied), LSRRT was often restricted to 25 distinct tests that were repeated in a random order. Fewer than 25 wasn’t seen as diverse enough. Many more than 25 made troubleshooting a failure a nightmare. (Why? Remember that the system can pass each test on its own, so the secret of the failure lies somewhere in the sequencing. If failure occurs after 48 hours of apparently-trouble-free operation, analysis of that sequence can be very complex. Limiting the number of distinct tests in the sequence was one way to limit that complexity.) Suppose that the test lab runs a 50-test LSRRT for 12 hours, i.e. a sequence that repeats 50 regression tests in random order until testing is terminated by failure or by successful completion of a 12-hour run. Suppose that on average, each of the 50 tests runs 100 times. Is this test volume 1, 50, or 5000? If the test volume is 1, equipment vendors will have a strong incentive to argue that very little of this testing should be done, because this is a very harsh style of testing. If test volume is 5000, equipment vendors will have a strong incentive to encourage the lab to do lots of this type of testing. I submit that the decision to apply this style of testing, the amount of testing to be done, and the characteristics of the tests combined in each suite should be based on other factors than the calculation of test-events/test-volume, but that calculation will drive potentially harsh debates. In practice, I have been told by testers of regulated products that they don’t do long sequence testing specifically because the metrics are impossible to agree on. Benchmark-estimation rules should NEVER drive decisions about what style of testing would be most effective for illuminating the risks associated with a product. .......... (Affiliation Note: IEEE representative to TGDC)
It is inappropriate to treat software regression tests as if they were a representative sample of the behavior of the system under test because the system is optimized to pass them as they are repeatedly run. The more times they are run, the less predictive power they have with respect to other tests that involve other data, other combinations of functions, or other sequences of events. .......... (Affiliation Note: IEEE representative to TGDC)
As it applies to software, this section's terminology is ambiguous or undefined. What is a test event? What is a test volume? What is a test campaign? .......... (Affiliation Note: IEEE representative to TGDC)
Amendment to: USACM Comment #25. Section 5.3.1 General method [incorrect] [1] Boris Beizer, Software Testing Techniques, Second Edition, 1990
5.3.2 Critical values
For a fixed probability p and a fixed value of n, the value of rv satisfying P(n,rv)=p is a constant. Part 3: Table 5-1 provides the values of rv for p=0.1 and p=0.9 for 0≤n≤750.
Given no observed events after volume vo, the demonstrated event rate rd is found by solving P(no,rdvo)=0.1 for rd. The pertinent factor is in the second column (p=0.1) in the row for n=no; dividing this factor by vo yields rd. For example, a volume of 600 with no events demonstrates an event rate of 2.302585/600, or 3.837642×10−3.
Since the condition for rejecting H0 is P(no−1,rbvo)>0.9, the critical value vc, which is the minimum volume at which H0 is not rejected for no observed events and event rate benchmark rb, is found by solving P(no−1,rbvc)=0.9 for vc. The pertinent factor is in the third column (p=0.9) in the row for n=no−1; dividing this factor by rb yields vc. For example, if a test with event rate benchmark rb=10−4 resulted in one observed event, then the system would be rejected unless the actual volume was at least 0.1053605/10−4, or 105.3605. Where the measurement of volume is discrete rather than continuous, one would round up to the next integer.
The values in Part 3: Table 5-1 were generated by the following script and Octave[2] version 2.1.73.
silent_functions=1
# Function for the root finder to zero.  fsolve won't pass extra
# parameters to the function being solved, so we must use globals.
# nGlobal is number of events; pGlobal is probability.
function rvRootFn = rvRoot (rv)
global nGlobal pGlobal
rvRootFn = poisson_cdf (nGlobal, rv) - pGlobal
# Find rv given n and p.  To initialize the root finder, provide
# startingGuess that is greater than zero and approximates the
function rvFn = rv (n, p, startingGuess)
nGlobal = n
pGlobal = p
startingGuess &gt; 0 || error (&quot;bad starting guess&quot;)
[rvFn, info] = fsolve (&quot;rvRoot&quot;, startingGuess)
if (info != 1)
perror (&quot;fsolve&quot;, info)
printf (&quot; n      P=0.1        P=0.9\n&quot;)
for n = 0:750
rv01 = rv (n, 0.1, -4.9529e-05*n*n + 1.0715*n + 2.302585093)
rv09 = rv (n, 0.9,  4.9522e-05*n*n + 0.9285*n + 0.105360516)
printf (&quot;%3u %.6e %.6e\n&quot;, n, rv01, rv09)
fsolve_options (&quot;tolerance&quot;, 5e-12)
Table 5-1 Factors for calculation of critical values
rv satisfying P(n,rv)=0.1
rv satisfying P(n,rv)=0.9
0.1053605
272.5461
231.8821
530.9192
3.88972
0.5318116
273.5864
232.8418
531.9478
474.4804
5.32232
1.102065
274.6267
233.8015
532.9764
475.4519
6.680783
1.74477
275.6669
234.7613
534.0049
476.4233
7.99359
2.432591
235.7212
535.0334
477.3948
9.274674
3.151898
236.6812
536.0619
478.3663
10.53207
3.894767
237.6412
537.0904
479.3379
11.77091
4.656118
279.8269
238.6013
538.1188
480.3094
12.99471
5.432468
280.8667
239.5615
539.1472
481.2811
6.221305
281.9064
240.5218
540.1755
482.2527
15.40664
7.020747
241.4822
541.2039
483.2243
16.59812
7.829342
283.9856
242.4426
542.2322
484.196
17.78159
8.645942
285.0251
243.4031
543.2605
485.1677
18.95796
9.469621
286.0645
244.3637
544.2887
486.1395
20.12801
10.29962
287.1039
245.3243
545.317
487.1113
21.29237
288.1432
246.2851
546.3452
488.0831
22.45158
11.97613
289.1824
247.2459
547.3734
489.0549
23.60609
12.82165
290.2215
248.2067
548.4015
490.0267
24.75629
13.67148
291.2605
249.1677
549.4296
490.9986
25.90253
14.52526
292.2995
250.1287
550.4577
491.9705
27.0451
15.38271
293.3384
251.0898
551.4858
492.9424
28.18427
16.24356
294.3773
252.0509
552.5138
493.9144
29.32027
17.10758
253.0122
553.5418
494.8864
17.97457
296.4547
253.9735
554.5698
495.8584
31.58356
18.84432
297.4934
254.9349
555.5978
496.8304
32.71121
19.71669
298.5319
255.8963
556.6257
497.8025
33.83639
20.59152
299.5704
256.8578
557.6536
498.7746
34.95926
21.46867
300.6088
257.8194
558.6815
499.7467
22.34801
301.6472
559.7094
500.7189
23.22944
302.6855
259.7428
560.7372
501.691
24.11285
303.7237
260.7046
561.765
502.6632
39.42982
24.99815
304.7618
261.6664
562.7928
503.6355
40.54274
25.88523
305.7999
262.6283
563.8205
504.6077
41.65395
26.77403
306.8379
263.5903
564.8482
42.76352
27.66447
307.8758
264.5524
565.8759
506.5523
28.55647
308.9137
265.5145
566.9036
507.5246
44.97802
309.9515
266.4767
567.9313
508.497
46.08308
30.34493
310.9893
568.9589
509.4694
47.18676
31.24126
312.0269
268.4013
569.9865
510.4418
48.2891
32.13892
313.0646
269.3637
511.4142
49.39016
33.03786
314.1021
270.3261
572.0416
512.3866
50.48999
33.93804
315.1396
271.2886
573.0691
513.3591
51.58863
34.83941
272.2512
574.0966
514.3316
52.68612
35.74192
317.2144
273.2138
575.1241
515.3042
53.7825
36.64555
318.2517
274.1765
576.1515
516.2767
54.87781
37.55024
319.2889
275.1393
577.1789
517.2493
55.97209
320.3261
276.1021
578.2063
518.2219
57.06535
39.36271
321.3632
277.065
579.2337
519.1945
58.15765
40.27042
322.4002
580.261
520.1672
41.17907
323.4372
581.2884
521.1399
60.33944
42.08863
324.4741
279.9541
582.3156
522.1126
61.42899
42.99909
325.511
280.9172
583.3429
523.0853
62.51768
43.9104
326.5478
281.8804
584.3702
524.0581
63.60553
44.82255
327.5845
282.8437
585.3974
525.0309
64.69257
45.73552
328.6212
586.4246
526.0037
65.77881
46.64928
329.6578
284.7704
587.4517
526.9765
66.86429
330.6944
285.7338
588.4789
527.9493
67.94901
48.47908
331.7309
286.6973
589.506
528.9222
49.39509
332.7673
287.6609
590.5331
70.11628
50.31182
333.8037
288.6245
591.5602
530.8681
71.19887
51.22923
289.5882
592.5872
531.841
72.28078
52.14733
335.8763
290.5519
593.6142
532.814
73.36203
53.06608
336.9125
291.5157
594.6412
74.44263
53.98548
337.9486
292.4796
595.6682
75.5226
338.9847
293.4435
596.6952
535.7331
76.60196
55.82616
340.0208
294.4074
597.7221
536.7061
77.68071
56.74741
295.3715
537.6792
78.75888
57.66924
342.0927
296.3355
599.7759
538.6523
79.83647
58.59165
343.1285
297.2997
600.8028
539.6255
80.9135
59.51463
344.1643
298.2639
601.8296
540.5986
81.98997
60.43815
299.2281
602.8564
541.5718
83.06591
61.36221
346.2358
300.1924
603.8832
542.545
84.14132
347.2714
301.1568
604.9099
543.5183
85.21622
63.21191
348.307
302.1212
605.9367
544.4915
86.29061
349.3426
303.0857
606.9634
545.4648
87.3645
65.06364
304.0502
607.9901
546.4381
88.4379
65.99023
351.4135
305.0148
609.0168
547.4115
89.51083
66.91731
352.4488
305.9794
610.0434
548.3848
90.58329
67.84485
353.4842
306.9441
549.3582
91.65529
68.77285
354.5194
307.9088
612.0966
550.3316
92.72684
69.7013
355.5546
308.8736
613.1232
93.79795
70.63019
356.5898
309.8384
614.1498
552.2785
94.86863
71.55951
357.6249
310.8033
615.1763
553.2519
95.93888
72.48927
358.6599
311.7683
616.2028
554.2254
97.00871
73.41944
359.6949
312.7333
617.2293
555.1989
98.07813
74.35002
360.7299
313.6983
618.2558
556.1725
99.14714
361.7648
314.6634
619.2822
557.146
100.2158
76.21239
362.7996
315.6286
620.3086
558.1196
77.14416
363.8344
316.5938
559.0932
102.3518
78.07631
364.8692
317.5591
622.3614
560.0668
103.4193
79.00885
365.9038
318.5244
623.3878
561.0405
104.4864
79.94175
366.9385
319.4897
624.4141
562.0141
105.5531
80.87502
367.9731
320.4552
625.4404
562.9878
106.6195
369.0076
321.4206
626.4667
563.9615
107.6855
82.74263
370.0421
322.3861
627.493
108.7512
83.67695
371.0765
323.3517
628.5192
109.8165
84.61162
372.1109
324.3173
629.5454
566.8828
110.8815
85.54663
373.1453
630.5716
567.8566
111.9462
86.48197
374.1796
326.2487
631.5978
568.8304
113.0105
87.41764
375.2138
327.2144
569.8043
114.0745
88.35362
328.1802
633.6501
570.7781
115.1382
89.28993
377.2821
329.1461
634.6762
571.752
90.22655
378.3162
635.7023
572.7259
117.2647
91.16347
379.3503
636.7284
573.6999
92.1007
380.3843
637.7544
574.6738
119.3899
93.03823
381.4182
638.7804
120.4521
93.97605
382.4521
333.9761
639.8064
576.6218
94.91416
334.9422
640.8324
577.5958
122.5756
95.85256
384.5198
335.9084
641.8584
578.5699
123.6369
96.79124
385.5536
336.8747
642.8843
579.5439
97.7302
386.5873
337.841
643.9102
125.7587
98.66944
387.6209
338.8073
644.9361
581.4921
126.8192
99.60895
388.6546
339.7737
645.962
582.4662
127.8794
100.5487
389.6881
340.7401
646.9879
583.4404
128.9394
101.4888
390.7217
341.7066
648.0137
584.4145
102.4291
391.7552
342.6731
649.0395
585.3887
131.0586
103.3696
392.7886
343.6396
650.0653
586.3629
132.1177
104.3104
393.822
344.6062
651.0911
587.3372
133.1767
105.2515
394.8553
345.5729
652.1168
588.3114
134.2354
106.1928
395.8886
346.5396
653.1426
589.2857
135.2938
107.1344
396.9219
347.5063
654.1683
108.0762
397.9551
348.4731
591.2343
109.0182
398.9883
349.4399
656.2196
592.2086
138.4677
109.9605
400.0214
350.4068
657.2453
139.5252
401.0545
351.3737
658.2709
594.1573
140.5825
111.8457
402.0875
352.3407
659.2965
595.1317
141.6395
112.7887
403.1205
353.3077
660.3221
596.1061
142.6963
113.7318
404.1535
354.2748
661.3477
597.0806
143.7529
114.6753
405.1864
355.2419
662.3732
144.8093
115.6189
406.2192
663.3987
599.0295
145.8655
116.5627
407.252
357.1762
664.4242
146.9214
117.5068
408.2848
358.1434
665.4497
600.9785
147.9771
118.4511
409.3176
359.1107
666.4752
149.0326
119.3955
410.3503
667.5006
602.9276
411.3829
361.0453
668.5261
603.9022
151.1431
121.2851
412.4155
362.0127
669.5515
604.8768
122.2302
413.4481
362.9802
670.5768
605.8514
153.2527
414.4806
363.9476
671.6022
154.3072
415.5131
364.9152
672.6276
607.8007
155.3615
125.0667
416.5455
365.8827
673.6529
608.7754
156.4156
417.5779
366.8503
674.6782
609.7501
157.4695
126.9586
418.6103
675.7035
610.7248
158.5233
127.9049
419.6426
368.7856
676.7287
611.6995
159.5768
128.8514
420.6749
369.7534
677.754
612.6743
160.6302
421.7071
370.7211
678.7792
161.6834
130.7448
422.7393
371.689
679.8044
614.6238
162.7364
131.6918
423.7714
372.6568
680.8296
615.5986
163.7892
424.8035
373.6247
681.8548
616.5735
164.8418
425.8356
374.5926
682.8799
617.5483
165.8943
134.5339
426.8676
375.5606
683.905
618.5232
166.9465
135.4816
427.8996
376.5286
684.9302
619.4981
167.9987
136.4295
428.9316
377.4966
685.9552
620.473
169.0506
137.3776
429.9635
378.4647
686.9803
621.4479
170.1024
138.3258
430.9954
379.4329
688.0054
622.4229
139.2742
432.0272
689.0304
623.3978
172.2054
140.2228
381.3692
690.0554
624.3728
173.2567
141.1715
434.0907
382.3375
691.0804
625.3478
174.3078
142.1204
435.1225
383.3058
692.1054
626.3228
175.3587
143.0695
436.1541
384.2741
693.1304
627.2979
176.4095
437.1858
385.2425
694.1553
628.2729
177.4601
144.9681
438.2174
386.2109
695.1802
178.5106
145.9176
439.2489
387.1793
696.2051
630.2231
179.5609
146.8673
440.2805
388.1478
631.1982
180.6111
147.8171
441.3119
389.1163
698.2549
632.1734
181.6611
148.7671
442.3434
390.0848
699.2797
633.1485
182.7109
149.7173
443.3748
391.0534
700.3045
634.1237
183.7606
150.6676
444.4062
392.0221
701.3293
635.0989
184.8102
445.4375
392.9907
702.3541
636.0741
185.8596
152.5686
446.4688
393.9594
703.3789
637.0493
186.9089
153.5193
447.5001
394.9282
704.4036
638.0246
154.4702
448.5313
395.8969
705.4284
638.9999
189.0069
155.4213
449.5625
396.8658
706.4531
639.9751
190.0558
450.5936
397.8346
707.4778
640.9505
191.1045
157.3237
451.6247
398.8035
708.5025
641.9258
158.2752
452.6558
399.7724
709.5271
642.9011
159.2268
453.6868
400.7414
710.5518
643.8765
194.2497
160.1785
454.7178
401.7104
711.5764
644.8518
195.2978
161.1304
455.7488
402.6794
645.8272
196.3458
162.0824
456.7797
403.6485
713.6256
646.8027
197.3937
163.0345
457.8106
404.6176
714.6501
647.7781
198.4414
163.9868
458.8415
405.5867
715.6747
648.7535
164.9392
459.8723
406.5559
716.6992
200.5365
165.8917
460.9031
407.5251
717.7237
650.7045
201.5839
166.8443
461.9338
408.4944
718.7482
202.6311
167.7971
462.9646
409.4637
719.7727
652.6555
203.6781
168.7501
463.9952
410.433
720.7972
653.6311
204.7251
169.7031
465.0259
411.4023
721.8216
654.6066
205.7719
170.6563
466.0565
412.3717
722.8461
655.5822
206.8186
171.6096
467.0871
413.3412
723.8705
656.5578
207.8652
468.1176
414.3106
724.8949
657.5334
208.9117
173.5165
469.1481
415.2801
725.9192
174.4702
470.1786
416.2496
726.9436
659.4847
211.0043
175.4239
417.2192
727.9679
660.4603
212.0504
176.3778
472.2394
418.1888
728.9922
661.436
213.0963
177.3319
473.2698
419.1584
730.0165
662.4117
214.1422
474.3001
420.1281
731.0408
663.3874
215.1879
179.2403
475.3304
421.0978
732.0651
664.3631
180.1946
476.3607
422.0675
733.0893
665.3389
217.2791
181.1491
477.3909
423.0373
734.1136
666.3147
218.3245
182.1037
478.4211
424.0071
735.1378
667.2904
219.3698
183.0584
479.4513
424.9769
736.162
668.2662
184.0133
480.4814
425.9468
737.1862
669.2421
184.9682
481.5115
426.9167
738.2103
670.2179
185.9232
482.5416
427.8866
739.2345
671.1938
223.5498
186.8784
483.5716
428.8566
740.2586
672.1696
224.5945
187.8337
484.6016
429.8266
741.2827
673.1455
225.6392
485.6316
430.7966
742.3068
674.1214
226.6837
189.7445
486.6615
431.7667
743.3309
675.0973
227.7281
190.7001
487.6914
432.7368
744.355
676.0733
228.7724
488.7213
433.7069
745.379
677.0492
229.8166
192.6116
489.7511
434.6771
678.0252
230.8607
193.5675
435.6473
747.427
679.0012
231.9047
194.5235
491.8107
436.6175
748.451
679.9772
232.9485
195.4797
492.8405
437.5878
749.475
680.9532
233.9923
196.4359
493.8702
438.5581
750.499
681.9293
197.3922
494.8999
439.5284
751.5229
236.0796
198.3486
495.9295
440.4987
752.5468
683.8814
237.1231
199.3051
496.9591
441.4691
753.5708
684.8575
238.1664
200.2618
497.9887
442.4395
754.5946
685.8336
239.2097
201.2185
499.0182
755.6185
686.8097
240.2529
202.1753
500.0478
444.3805
756.6424
687.7859
203.1322
501.0773
757.6662
204.0892
502.1067
446.3215
758.6901
689.7382
243.3819
205.0463
503.1361
447.2921
759.7139
690.7144
244.4247
206.0035
504.1655
448.2627
760.7377
691.6906
245.4674
206.9608
505.1949
449.2333
761.7614
692.6668
207.9182
506.2242
762.7852
693.643
247.5525
208.8757
507.2535
451.1747
763.8089
694.6193
248.5949
209.8333
508.2828
452.1454
764.8327
695.5956
249.6372
453.1162
765.8564
696.5718
250.6795
211.7488
510.3413
454.087
766.8801
697.5482
251.7216
212.7066
511.3704
455.0578
767.9038
698.5245
252.7636
213.6646
512.3996
456.0287
768.9274
699.5008
253.8056
214.6226
513.4287
456.9995
769.9511
700.4772
254.8475
215.5807
514.4578
457.9704
701.4535
255.8893
216.539
515.4869
458.9414
771.9983
702.4299
217.4973
516.5159
459.9123
773.0219
703.4063
257.9726
218.4557
517.5449
460.8833
774.0455
704.3827
259.0141
219.4141
518.5739
461.8544
775.0691
705.3592
260.0555
220.3727
519.6028
462.8254
776.0926
706.3356
261.0969
221.3314
520.6317
463.7965
777.1162
707.3121
262.1381
222.2901
521.6606
464.7676
778.1397
708.2885
263.1793
223.2489
522.6894
465.7388
779.1632
709.265
264.2204
224.2078
523.7183
780.1867
710.2416
265.2614
225.1668
524.7471
467.6812
781.2102
711.2181
266.3023
226.1259
525.7758
468.6524
782.2336
712.1946
267.3431
227.0851
526.8046
469.6237
783.2571
713.1712
268.3839
228.0443
527.8333
784.2805
714.1478
269.4246
229.0037
528.862
471.5663
785.3039
715.1243
270.4652
229.9631
529.8906
472.5376
786.3273
Table 5-2 Plot of values from Table 5-1
Amendment to: USACM Comment #26. Section 5.3.1 General method [incorrect] USACM recommends correcting the factual errors present in this subsection, as minimally enumerated below: 1. The system submitted for testing might be a representative sample of the pool of identical hardware/software systems, but the pool of tests should not be a representative sample of the events that happen during an election. 2. There is no reason to expect software reliability, software accuracy, and hardware misfeed rate to follow the same distribution. 3. The Poisson distribution is discrete, not continuous. 4. The Poisson process typically assumes a stationary underlying exponential distribution. The idea that software reliability, software accuracy, and hardware misfeed rates follow the same underlying distribution, or that the concatenation of these three (if there are only three) distributions would be anything like exponential is remarkable in its unlikelihood. 5. The observed event rate ("events" divided by "volume" over the course of a test campaign) is a highly biased measure. a. The first problem is that a regression test suite repeats the same tests from build to build. This gives rise to the classic problem of the "pesticide paradox" [1]. The test suite is a tiny sample of the collection of possible tests. When the suite reveals bugs, they are fixed. Ultimately, the test suite becomes a collection of tests that have one thing in common: the software has passed all of them at least once. This differs from almost every other possible test (all of the ones that have not been run). Therefore, the reliability of the software is probably vastly overestimated. b. The second problem is that the pool of tests is structured to cover a specification. It does not necessarily target vulnerabilities of the software. Nor is it designed to reflect usage frequencies in the field [2]. 6. Determining the length of testing in advance by an approved test plan sounds scientific, but many practitioners consider this software testing malpractice. There is substantial evidence that bugs cluster. Given a failure, there is reason to do follow-up testing to study this area of the product in more detail. Rigidly adhering to a plan created in the absence of failure data is to rigidly reject the idea of follow-up testing. This underestimates the number of problems in the code. Worse, this testing method reduces the chance that defects will be found and fixed because it reduces — essentially bans — the follow-up testing that would expose those additional defects. References [1] Boris Beizer, Software Testing Techniques, Second Edition, 1990 [2] Musa, Software Reliability Engineering (http://members.aol.com/JohnDMusa/book.htm
USACM Comment #26. Section 5.3.2. Critical Values [incorrect] This section should be adjusted according to factual corrections made in the previous comment.
5.3.3 Reliability
5.3.3-A Reliability, pertinent tests
All tests executed during conformity assessment SHALL be considered "pertinent" for assessment of reliability, with the following exceptions:
Tests in which failures are forced;
Tests in which portions of the system that would be exercised during an actual election are bypassed (see Part 3: 2.5.3 "Test fixtures").
USACM Comment #27. 5.3.3 Reliability [incorrect] USACM notes the apparent self-contradictions in this sub-section, enumerated in the discussion below. DISCUSSION: 1. Failure rate data are not relevant to prediction of reliability in the field unless we assume that the failure rate in the lab is representative of the failure rate that will be found in the field. This might be rational for hardware, but unless we structure the software tests to map to usage in the field, there is no rational basis for this assumption vis-à-vis the software. 2. Pass/fail criteria are based on the concatenation of hardware and software failures. A paper jam rates the same as miscount of votes. 3. Counting all "failures" for statistical purposes creates an adversarial dynamic around the classification of anomalous behaviors. To the extent that an apparently-incorrect behavior is arguably not inconsistent with the specification, there is an incentive to class it as a non-bug and therefore not fix it. The incentives should favor improving the software, not classifying problems as non-problems
5.3.3-B Failure rate data collection
The test lab SHALL record the number of failures and the applicable measure of volume for each pertinent test execution, for each type of device, and for each applicable failure type in Part 1: Table 6-3 (Part 1: 6.3.1.5 "Requirements").
Applies to: Voting device
"Type of device" refers to the different models produced by the manufacturer. These are not the same as device classes. The system may include several different models of the same class, and a given model may belong to more than one class.
How are failures defined? Is the number of failures (either recoverable or non-recoverable) counted without respect to the severity?
5.3.3-C Failure rate pass criteria
When operational testing is complete, the test lab SHALL calculate the failure total and total volume accumulated across all pertinent tests for each type of device and failure type. If, using the test method in Part 3: 5.3.1 "General method", these values indicate rejection of the null hypothesis for any type of device and type of failure, the verdict on conformity to Requirement Part 1: 6.3.1.5-A SHALL be Fail. Otherwise, the verdict SHALL be Pass.
Benchmarks are typically based on industry-wide data. What comparable industry was used for the factors for calculation of critical values? We see how the lab is expected to generate the true event rate (although we don't agree with the collection method). How are the labs supposed to evaluate the benchmark event rate in order to evaluate the null hypothesis (conforming or non-conforming)?.
5.3.4 Accuracy
The informal concept of voting system accuracy is formalized using the ratio of the number of errors that occur to the volume of data processed, also known as error rate.
5.3.4-A Accuracy, pertinent tests
All tests executed during conformity assessment SHALL be considered "pertinent" for assessment of accuracy, with the following exceptions:
Tests in which errors are forced;
USACM Comment #28. Section 5.3.4. Accuracy [incorrect] USACM notes the apparent self-contradictions in this sub-section as described in the discussion below. DISCUSSION: Accuracy is operationalized (not formalized) as a ratio of errors found to volume of data processed. One may assume that the word "error" is tied tightly to events that yield a miscount of the votes, allow someone to cast extra votes, or cause someone to be unable to cast a vote. If "error" includes anything in the behavior of the program that would not create an error in election result, it is difficult to understand what this operationalization has to do with the naturalistic concept of "accuracy" in a system that collects and counts votes. The operationalization is defective as an estimator unless the pool of tests is designed so as to be representative of the pool of behaviors in the field. If some aspect of the system causes a small mistake (e.g. 1-vote miscount), but is only tested once, that might be a major source of inaccuracy if everyone encounters it while voting, and it might be a trivial source if almost no one encounters it. For example, imagine a system that allowed ballots that could accept write-in votes for up to 100 candidates. Imagine an error in which 1 vote in 10 is lost in the 100th race that includes a write-in candidate. As a boundary case, this error might show up in several tests. However, it might never show up in an election. What is the accuracy using the described metric? Without a mapping from the estimator to the construct being estimated, the metric is worthless. This is a fundamental issue in measurement. We normally call it construct validity. The argument that this measure of accuracy estimates underlying system accuracy lacks even face validity.
5.3.4-B Calculation of report total error rate
Given a set of vote data reports resulting from the execution of tests, the observed cumulative report total error rate SHALL be calculated as follows:
Define a "report item" as any one of the numeric values (totals or counts) that must appear in any of the vote data reports. Each ballot count, each vote, overvote, and undervote total for each contest, and each vote total for each contest choice in each contest is a separate report item. The required report items are detailed in Part 1: 7.8.3 "Vote data reports";
For each report item, compute the "report item error" as the absolute value of the difference between the correct value and the reported value. Special cases: If a value is reported that should not have appeared at all (spurious item), or if an item that should have appeared in the report does not (missing item), assess a report item error of one. Additional values that are reported as a manufacturer extension to the standard are not considered spurious items;
Compute the "report total error" as the sum of all of the report item errors from all of the reports;
Compute the "report total volume" as the sum of all of the correct values for all of the report items that are supposed to appear in the reports. Special cases: When the same logical contest appears multiple times (e.g., when results are reported for each ballot configuration and then combined or when reports are generated for multiple reporting contexts), each manifestation of the logical contest is considered a separate contest with its own correct vote totals in this computation;
Compute the observed cumulative report total error rate as the ratio of the report total error to the report total volume. Special cases: If both values are zero, the report total error rate is zero. If the report total volume is zero but the report total error is not, the report total error rate is infinite;
Source:	Revision of [GPO90] F.6
5.3.4-C Error rate data collection
The test lab SHALL record the report total error and report total volume for each pertinent test execution.
Accuracy is calculated as a system-level metric, not separated by device type.
5.3.4-D Error rate pass criteria
When operational testing is complete, the test lab SHALL calculate the report total error and report total volume accumulated across all pertinent tests. If, using the test method in Part 3: 5.3.1 "General method", these values indicate rejection of the null hypothesis, the verdict on conformity to Requirement Part 1: 6.3.2-B SHALL be Fail. Otherwise, the verdict SHALL be Pass.
5.3.5 Misfeed rate
This benchmark applies only to paper-based tabulators and EBMs. Multiple feeds, misfeeds (jams), and rejections of ballots that meet all manufacturer specifications are all treated collectively as "misfeeds" for benchmarking purposes (i.e., only a single count is maintained).
5.3.5-A misfeed rate, pertinent tests
All tests executed during conformity assessment SHALL be considered "pertinent" for assessment of misfeed rate, with the following exceptions:
Tests in which misfeeds are forced.
5.3.5-B Calculation of misfeed rate
For paper-based tabulators and EBMs, the observed cumulative misfeed rate SHALL be calculated as follows:
Compute the "misfeed total" as the number of times that unforced multiple feed, misfeed (jam), or rejection of a ballot that meets all manufacturer specifications has occurred during the execution of tests. It is possible for a given ballot to misfeed more than once – in such a case, each misfeed would be counted:
Compute the "total ballot volume" as the number of successful feeds of ballot pages or cards during the execution of tests. (If the pages of a multi-page ballot are fed separately, each page counts; but if both sides of a two-sided ballot are read in one pass through the tabulator, it only counts once);
Compute the observed cumulative misfeed rate as the ratio of the misfeed total to the total ballot volume. Special cases: If both values are zero, the misfeed rate is zero. If the total ballot volume is zero but the misfeed total is not, the misfeed rate is infinite.
Applies to: Paper-based device Λ Tabulator, EBM
"During the execution of tests" deliberately excludes jams that occur during pre-testing setup and calibration of the equipment. Uncalibrated equipment can be expected to jam frequently. Source:	New requirement
5.3.5-C Misfeed rate data collection
The test lab SHALL record the misfeed total and total ballot volume for each pertinent test execution, for each type of device.
"Type of device" refers to the different models of paper-based tabulators and EBMs produced by the manufacturer.
<a href="../comments/createComment?section=p3-5.3.5-C"> Add a Comment</a>
5.3.5-D Misfeed rate pass criteria
When operational testing is complete, the test lab SHALL calculate the misfeed total and total ballot volume accumulated across all pertinent tests. If, using the test method in Part 3: 5.3.1 "General method", these values indicate rejection of the null hypothesis for any type of device, the verdict on conformity to Requirement Part 1: 6.3.3-A SHALL be Fail. Otherwise, the verdict SHALL be Pass.
5.4 Open-Ended Vulnerability Testing
Vulnerability testing is an attempt to bypass or break the security of a system or a device. Like functional testing, vulnerability testing can falsify a general assertion (namely, demonstrate that the system or device is secure) but it cannot verify the security (show that the system or device is secure in all cases). Open-ended vulnerability testing (OEVT) is conducted without the confines of a pre-determined test suite. It instead relies heavily on the experience and expertise of the OEVT Team Members, their knowledge of the system, its component devices and associated vulnerabilities, and their ability to exploit those vulnerabilities.
The goal of OEVT is to discover architecture, design and implementation flaws in the system that may not be detected using systematic functional, reliability, and security testing and which may be exploited to change the outcome of an election, interfere with voters’ ability to cast ballots or have their votes counted during an election or compromise the secrecy of the vote. The goal of OEVT also includes attempts to discover logic bombs, time bombs or other Trojan Horses that may have been introduced into the system hardware, firmware, or software for said purposes.
If the OEVT relies heavily on the experience and expertise of the OEVT Team Members, the testing is not repeatable and does not comply with the NIST 150-22 for repeatability (4.13.3).
Comment by Brit Williams (Academic)
This entire appears to be hastily written and poorly thought out. The purpose of certification testing is to verify that the voting system complies with the voting system guidelines. OEVT, as written, is a scatter shot approach to testing that does not address compliance with amy specific guideline. There is no management/oversight structure of the OEVT test team presented. There is no process for selecting team members presented. This is not to say that these types of tests have no value, but they should be part of the EAC Certification Procedures and not part of the VVSG. I will submit specific comments below.
OEVT is laudable, but difficult to incorporate into voting system design features. It is, by definition, subjective and undefined resulting in a security threshold that is difficult, if not impossible, to design for. Given the fact that the current dynamic for the industry is the financing of the voting system certification, no provider wishes to submit a product through an expensive process to have it fail based on a subjective standard that is not repeatable. This security process should be renamed and should incorporate clear security benchmarks that are broad in scope, but clear in their performance requirement.
Comment by E Smith/L Korb (Manufacturer)
The requirements, scope, mandate and documentation requirements of this specification are poorly defined and are dependent upon the makeup of the OEVT team. Additional work should be done to specify a fair and reasonable test. Many of the requirements of the VVSG would seem to open the possibility for new types of denial of service attacks. Much work needs to be accomplished before this section is ready for implementation.
USACM Comment #29. OEVT Goal [imprecise] USACM recommends that the present stated goal for OEVT (Sect 5.4, par 2, first sentence) be modified to read as follows: "The goal of OEVT is to test and analyze the target system to discover flaws that could adversely affect the election process and that may reflect systemic development process weaknesses." DISCUSSION: The current text is focused on discovering flaws that could invalidate election results. The proposed language would allow OEVT to be used in checking for flaws in other aspects of election operations, including accessibility and usability. OEVT is not meant as a replacement for quality design. The use of OEVT must be carefully described. It is a process that is difficult to replicate (if it were easy to replicate, it would not really be open-ended), so any requirements on OEVT in this VVSG should focus on the process — how the team is selected, the scope of work — of OEVT rather than specific steps. As with other parts of the testing process, this must be open to review by qualified scholars.
Comment by Matt Bishop, Mark Gondree, Sean Peisert, Elliot Proebstel (Academic)
We are excited by the addition of an "open-ended vulnerability testing" (OEVT) phase of the certification process, as described in the Aug 31, 2007 draft of the EAC's VVSG standards. Adding this type of testing, which is widely used in other venues for systems that must provide high security and reliability, will be invaluable in finding problems that could disrupt elections. As computer security practitioners, researchers, and graduate students, we anticipate that such a phase will be invaluable in future versions of these standards. The value of the OEVT lies in its ability to detect flaws that arise during use. In computer security, many flaws arise because mechanisms are integrated into a single unit, and even if the mechanisms are secure, the integration may inject unanticipated errors and problems. Further, humans are imperfect; so, as they interact with systems, problems--including security flaws--become evident. Lastly, developers of security mechanisms often make assumptions about the environment and use of their systems that differ from the environment and use of their systems in practice. Red teams can exploit this discrepancy to find flaws that security analysts who examine the systems and source code cannot. Therefore, from the point of view of computer security, open ended vulnerability testing is a valuable and necessary addition to the voting system federal certification process. The federal certification process will be better able to catch unanticipated design and implementation vulnerabilities before these flaws become "headliners". This will improve the effectiveness of testing requirements that require source code, design, and document review. It can help to identify areas where procedural defenses are critical to system security. As a result, this may lead to an increase in voter confidence and a decrease in the discrepancies between the results of federal certification and state testing, ultimately reducing the need for testing at the state level. This can also provide a means for more thoroughly testing systems that are not software independent, should such systems be "grandfathered" for a period. Requiring software independence helps the red teams frame their analysis, because one part of their work can be to verify whether the system meets this requirement--for if it does, undetected software flaws become much less harmful. In fact, software independence provides assurance that the system will function correctly even if there are software flaws. Red teaming cannot establish the absence of flaws, merely their presence, and so software independence adds assurance that the voting system will meet its goals. Based on our experience with similar Red Teaming exercises (members of our group have participated in numerous Red Teaming exercises and security evaluations relevant to election security, including the 2007 FSU report and 2007 CA Top-to-Bottom review), we have enumerated several comments below. We often refer to the part of the test lab performing the OEVT review as the "Red Team," while referring to the rest of the staff as "the lab." We refer to those threats achieving goals which would fall under the focus of the Red Team (as defined in 5.4.1-B) as "5.4.1-B threats."
This section lists requirements for vulnerability testing including team composition, the scope of testing, resources made available, level of effort to be expended and the rules of engagement for evaluation of the system. The team make-up and qualifications requirements are designed such that the testers possess a high level of expertise. Contrary to some criticisms of the VVSG draft as having no requirements to define vulnerability testing, these concrete requirements help to define this type of testing. They should be included in the VVSG.
5.4.1 OEVT scope and priorities
5.4.1-A Scope of open-ended vulnerability testing
The scope of open ended vulnerability testing SHALL include the voting system security during all phases of the voting process and SHALL include all manufacturer supplied voting system use procedures.
The scope of OEVT includes but is not limited to the following:
Voting system security;
Voting system physical security while voting devices are:
In storage;
Being configured;
Being transported; and
Voting system use procedures.
This section should specify that the voting system under test should be installed in an election environment, including all of the procedural security features used during an actual election.
5.4.1-B Focus of open-ended vulnerability testing
OEVT Team members SHALL seek out vulnerabilities in the voting system that might be used to change the outcome of an election, to interfere with voters’ ability to cast ballots or have their votes counted during an election or to compromise the secrecy of vote.
<a href="../comments/createComment?section=p3-5.4.1-B"> Add a Comment</a>
The OEVT team SHALL have at least one member (and not be the same person) with 6 or more years of experience in the area of software engineering, at least one member with 6 or more years of experience in the area of information security, at least one member with 6 or more years of experience in the area of penetration testing and at least one member with 6 or more years of experience in the area of voting system security. This requirement should not individualize the experience described above. These experience requirements should be contained by multiple team members.
Is the review of resumes to verify compliance of the test lab established OEVT members part of the test plan submitted to the EAC for approval (extension of Part 2 section 5.2-F)?
What is the rational for requiring that the security team members have six years experience and the electiion management team member have eight years experience?
Red Team member requirements 5.4.2-C requires the Red Team be composed of "at last one member with 6 or more years of experience in the area of voting system security." Under this requirement, no members of the groups involved in the CA Top-to-Bottom review, the Florida State report, the Johns Hopkins report, and the RABA report would qualify. Further, expertise in voting systems, while helpful, has not proven essential to previous work. This clause should probably be a recommendation and not a requirement, and change "6 years" to "3 years".
5.4.1-C OEVT General Priorities
The OEVT team SHALL prioritize testing efforts based on:
threat scenarios for the voting system under investigation;
the availability of time and resources;
the OEVT team’s determination of easily exploitable vulnerabilities; and
the OEVT team’s determination of which exploitation scenarios are more likely to impact the outcome of an election, interfere with voters’ ability to cast ballots or have their votes counted during an election or compromise the secrecy of the vote.
Following are suggestions for OEVT prioritization in the areas of threat scenarios, COTS products and Internet based threats. The intent here is to provide guidance on how to prioritize testing efforts given specific voting device implementations.
All threat scenarios must be plausible in that they should not be in conflict with the anticipated implementation, associated use procedures, the workmanship requirements in section 6.4 (assuming those requirements were all met) or the development environment specification as supplied by the manufacturer in the TDP;
Open-ended vulnerability testing should not exclude those threat scenarios involving collusion between multiple parties including manufacturer insiders. It is acknowledged that threat scenarios become less plausible as the number of conspirators increases;
It is assumed that attackers may be well resourced and may have access to the system while under development;
Threats that can be exploited to change the outcome of an election and flaws that can provide erroneous results for an election should have the highest priority;
Threats that can cause a denial of service during the election should be considered of very high priority;
Threats that can compromise the secrecy of the vote should be considered of high priority;
A threat to disclosure or modification of metadata (e.g., security audit log) that does not change the outcome of the election, does not cause denial of service during the election, or does not compromise the secrecy of ballot should be considered of lower priority;
If the voting device uses COTS products, then the OEVT team should also investigate publicly known vulnerabilities; and
The OEVT team should not consider the voting device vulnerabilities that require Internet connectivity for exploitation if the voting device is not connected to the Internet during the election and otherwise. However, if the voting device is connected to another device which in turn may have been connected to the Internet (as may be the case of epollbooks), Internet based attacks may be plausible and should be investigated.
Comment by Harry VanSickle (State Election Official)
Please explain how "experts" will be identified for purposes of OEVT team composition. More specifically, please identify who makes that decision. The standards outline the requisite skills that are necessary to be an OEVT team member, but the standards are not clear regarding who determines whether a candidate meets the criteria.
USACM Comment #31. 5.4.2-E OEVT team knowledge [incorrect] USACM recommends that the word "Complete" in items numbered a and b in this subsection be replaced by the word "Expert". DISCUSSION: No one, and no small team, can have "complete knowledge." A patently impossible requirement offers no guidance as to the expected level of knowledge and competence. On the other hand, "expert knowledge", while subjective, is a recognized standard.
5.4.2 OEVT resources and level of effort
5.4.2-A OEVT team resources
The OEVT team SHALL use the manufacturer supplied Technical Data Package (TDP) and User documentation, have access to voting devices configured similar to how they are to be used in an election, and have access to all other material and tools necessary to conduct a thorough investigation.
Materials supplied to the OEVT team should include but not be limited to the following:
Threat analysis describing threats mitigated by the voting system;
Security architecture describing how threats to the voting system are mitigated;
High level design of the system;
Any other documentation provided to the testing laboratory;
Operational voting system configured for election, but with the ability for the OEVT team to reconfigure it;
Testing reports from the developer and from the testing laboratory including previous OEVT results;
Tools sufficient to conduct a test lab build; and
Procedures specified by the manufacturer as necessary for implementation and secure use.
The task of the OEVT should not be to verify anything; it should operate from the assumption that what has been handed to it is defective. The OEVT is tasked with discovering new types of problems which were missed in conventional testing. Focusing the OEVT effort on the materials already relied on by the other testers is a good way to encourage OEVT failure. If you aren't going to unchain the OEVT team, don't waste the manufacturer's money on them. .......... (Affiliation Note: IEEE representative to TGDC)
5.4.2-B Open-ended vulnerability team establishment
The test lab SHALL establish an OEVT team of at least 3 security experts and at least one election management expert to conduct the open-ended vulnerability testing.
The OEVT team should not have the authority to fail a voting system. The OEVT team should conduct their tests and submit to the VSTL a report of their findings. Similarly, the VSTL prepares a report of their tests, including the results of the OEVT, and submits their report to the EAC. The EAC is the only entity that has the authority to fail a voting system. Also, the OEVT should not be allowed to release any of their findings to any organization other than the VSTL. Ultimately, the EAC has the responsibility to determine which reports should be released to the public and which should remain proprietary in order to protect the intellectual property of the vendor or to protect the national security.
A system's security is based on it's threat model. If the OEVT Team modifies a system's threat model with subjective and unsubstantiated threats, then any system would fail. If the guidelines require OEVT testing, then the guidelines should provide a threat model to guide manufacturer's in the design of their security. Proposed Change: Provide a threat model in the guidelines and remove the ability for the OEVT team to modify that threat model except through formal requests for changes to revisions of the guidelines.
5.4.2-C OEVT team composition – security experts
The OEVT team SHALL have at least one member with 6 or more years of experience in the area of software engineering, at least one member with 6 or more years of experience in the area of information security, at least one member with 6 or more years of experience in the area of penetration testing and at least one member with 6 or more years of experience in the area of voting system security.
5.4.2-D OEVT Team Composition- Election Management Expert
The OEVT team SHALL have at least one member with at least 8 years of experience in the area of election management.
The OEVT team will require consultation from an elections expert who is familiar with election procedures, how the voting systems are installed and used, and how votes are counted.
5.4.2-E OEVT team knowledge
The OEVT team knowledge SHALL include but not be limited to the following:
Complete knowledge of work done to date on voting system design, research and analysis conducted on voting system security, and known and suspected flaws in voting systems;
Complete knowledge of threats to voting systems;
Knowledge equivalent to a Bachelor’s degree in computer science or related field;
Experience in design, implementation, security analysis, or testing of technologies or products involved in voting system; and
Experience in the conduct and management of elections.
5.4.2-F OEVT level of effort – test plan
In determining the level of effort to apply to open-ended vulnerability testing, the test lab SHALL take into consideration the size and complexity of the voting system; any available results from the "close ended" functional, security, and usability testing activities and laboratory analysis and testing activities; the number of vulnerabilities found in previous security analyses; and testing of the voting system and its prior versions.
5.4.2-G OEVT level of effort – commitment of resources
The OEVT team SHALL examine the system for a minimum of 12 staff weeks.
5.4.3 Rules of engagement
5.4.3-A Rules of engagement – context of testing
Open ended vulnerability testing SHALL be conducted within the context of a process model describing a specific implementation of the voting system and a corresponding model of plausible threats.
The specification of these models is supported by information provided by the manufacturer as part of the TDP. See Requirement Part 2: 3.5.1.
5.4.3-B Rules of engagement – adequate system model
The OEVT team SHALL verify that the manufacturer provided system model sufficiently describes the intended implementation of the voting system.
Manufacturer’s system model and associated documentation should reliably describe the voting system and all associated use procedures given the environment in which the system will be used.
5.4.3-C Rules of engagement – adequate threat model
The OEVT team SHALL verify that the threat model sufficiently addresses significant threats to the voting system.
Significant threats are those that could:
Change the outcome of an election;
Interfere with voters’ ability to cast ballots or have their votes counted during an election; or
Compromise the secrecy of vote.
OEVT team may modify the manufacturer’s threat model to include additional, plausible threats.
5.4.4 Fail criteria
5.4.4-A OEVT fail criteria – violation of requirements
The voting device SHALL fail open ended vulnerability testing if the OEVT team finds vulnerabilities or errors in the voting device that violate requirements in the VVSG.
While the OEVT is directed at issues of device and system security, a violation of any requirement in the VVSG can lead to failure. Following are examples of issues for which the test lab must give a recommendation of "fail":
Evidence that any single person can cause a violation of a voting system security goal (e.g., integrity of election results, privacy of the voter, availability of the voting system), assuming that all other parties follow procedures appropriate for their roles as specified in the manufacturer’s documentation;
Manufacturer's documentation fails to adequately document all aspects of system design, development, and proper usage that are relevant to system security. This includes but is not limited to the following:
System security objectives;
Initialization, usage, and maintenance procedures necessary to secure operation;
All attacks the system is designed to resist or detect; and
Any security vulnerabilities known to the manufacturer.
Use of a cryptographic module that has not been validated against FIPS 140-2;
Ability to modify electronic event logs without detection;
A VVPR that has an inaccurate or incomplete summary of the cast electronic vote;
Unidentified software on the voting system;
Identified software which lacks documentation of the functionality it provides to the voting device;
Access to configuration file without authentication;
Ability to cast more than one ballot within a voting session;
Ability to perform restore operations in Activated State;
Enabled remote access in Activated State; and/or
Ballot boxes without appropriate tamper evidence countermeasures.
Comment by Kevin Baas (Academic)
I that think instead of saying 3. All attacks the system is designed to resist or detect; and 4. Any security vulnerabilities known to the manufacturer. it would be better to know: 3. All attacks the system is NOT designed to resist or detect; and 4. Any security vulnerabilities NOT known to the manufacturer. to accomplish this, instead of - or rather, in addition to - listing these things it would be better to say "list all the points in the process and if each point has or does not have x. For instance, they might say "oh, we use an access database here, and that's pretty secure.. but fail to mention that it's not password protected, though it obviously could be with out much effort. So there should be a list of all points and a checklist for each point, perhaps made by security experts. The goal, of course, is to have as complete a checklist as possible, so community input through something like a wiki would be very helpful in making the list more complete (by reducing the probability that an item is missed). I propose this section be amended with the following additions (or something similar which retains the principle of them): 13. All ballot/vote storage/transfer/processing points/channels in the voting, tabulation, and tabulation reporting system, in the order of flow. (i.e. following the vote from the point of entry to final certification.) And their physical and logical security environment. 14. "Point of entry" includes the software (if a computer-like system is used, such as a touchscreen voting machine) which first records the vote, and by implication, therefore, includes the source code to that software. The source code to the software that runs on the machine should be provided for review, and its vulnerabilities and so forth should be considered as with. Regarding point 13: security environment includes: a. hardware used b. logical access to hardware 1. local access security (password protected account login?, is a screen saver password used?, etc.) 2. network access (through network, modem, etc. (including shared filesystems)) security as local access security, plus network-specific items such as firewalls etc. 3. surveillance (logging, transparency) c. physical access to hardware 1. who is authorized to use the equipment 2. what is required to get into the room where the hardware is stored? (security cards, keys, etc.) 3. who has these tools (to get into the room) and how are they secured? 4. surveillance - cameras, people, etc. b. software and protocols used c. encryption used or lack thereof d. password protection used or lack thereof Regarding point 14: By "computer-like system" I mean a system that includes a subsystem which is "turing complete" or near turing complete, such that it's programmability allows if functionality to be substantially altered. For instance, this applies to systems where computer software is used in the vote entry process, such as "e-voting" machines, due to vulnerabilities inherent in programmable devices. First thing that needs to be cleared up once and for all, is that all claims that the computer program that records the vote is "proprietary software" are bogus, as: 1. Releasing the source code, privately or publicly, does not put the company at a competitive disadvantage. Writing a computer program to record a vote is trivial. (And thus fails to meet the requirements for a software patent.) (And by trivial I mean TRIVIAL.) The manufacturer is not liable to suffer any monetary damage from the private or public disclosure of the source code, as any competent programmer could produce software for recording a vote just as easily without it. Therefore, when it comes to recording a vote, having access to and/or using a competitor's source code would not provide a company with a competitive advantage (as that software is trivial to produce), and therefore would not put the source company at a competitive disadvantage. Any claims to the contrary are bogus. Any honest, competent, computer programmer will tell you this. 2. Releasing the source code, privately or publicly, does subject the company to any potential monetary losses/damages, save those incurred from the consumer being informed of a potential hazard of the product (in this case, the hazard being a flawed election), which information a consumer is legally entitled to for their protection (in this case, the protection of their right to vote). 3. Releasing the source code, privately or publicly, does not create a security vulnerability, as: a. to write source code for a machine that interfaces with input/output devices and runs on the platform, one needs knowledge of the machines _hardware_. b. to install software on a machine one needs physical access to the machine c. if such physical access necessary to install software on the machine were available to a non-specialist, that in itself would constitute a severe security vulnerability. (and therefore a failure of this test.) 4. Withholding the source code from third-party review constitutes a security vulnerability, as: a. without third-party review, the software could be written to do almost anything. It could change votes arbitrarily, or even just ignore votes all together and report an arbitrary total. The total could be preprogrammed, or an outcome could be preprogrammed and a total calculated that reasonably matches the votes but guarantees a pre-programmed outcome. etc. etc. Without explicit software review there is absolutely NO protection against these threats. b. There remains, in fact, even after code review, a number of software-related threats, including but not limited to: 1. Installation testing: The threat that said software is not the software actually installed on the machines. To secure against this, reviewers must be able to install or observe the installation process. 2. Turing completeness: The threat that said software, after being installed on the machine, is not executed by the machine. (That instead, another program is run from previously installed software or from a hardware device such as Read Only Memory.) To make sure that the system is actually running software when it is installed on it, reviewers must be able to provide different programs on the machine and see if the machine executes those instructions and produces the predicted result. (I.e. the machine should be able to demonstrate turing completeness.) 3. Timed code: a system may be set up such that failure scenarios 1 and 2 above may occur only during pre-selected times, such as only on election day. A tester/reviewer must be able to roll forward/back any and all system/internal/connected clocks to an arbitrary date/time, and test the system while it is in a state such that all relevant hardware and software thinks that it is the date/time that the system has been rolled forward (or back) to. The system should be tested in this manner for the date/time that it is supposed to function properly during (such as election day and the day after). c. and as well, the threat of tampering with electronic data, which is much greater than that of tampering with data stored physically, such as on a piece of paper, because data stored electronically is easier to access and alter. To protect against this increased threat, electronic voting systems must provide: 1. Methods, procedures, equipment, storage, and retrieval mechanisms for backing up and securing voting records for later review & verification until a time to be determined by law should be in place and tested. 2. Methods, procedures, equipment, etc. for preventing the digital tampering of voting records should be in place, reviewed, and tested. Such methods include: a. distributed backup/parity check: multiple copies of the data are distributed to different physical locations, such that at a later date they can be compared against each other for discrepancies. (which would constitute evidence of tampering) b. public key encryption: this protects against tampering by making the data essentially read-only. The data is encrypted with an asymmetric encryption scheme, and the decryption key is made public, while the encryption key is kept private, perhaps even randomly generated by the machine at election time, and then wiped from computer memory when all the votes have been recorded, encrypted and transfered. The corresponding decryption key is stored on the machine so that people and widely distributed as soon as possible to prevent tampering (via distributed backup/parity check). In asymmetric encryption, the encryption key cannot decrypt, and the decryption key cannot encrypt. This provides read-only functionality because only those with the encryption key can create data that can be decrypted by the decryption key. Data that cannot be decrypted by the decryption key was not encrypted by the encryption key, and was therefore clearly either "tampered with" after being encrypted, or was not from a person or machine that has the encryption key. There are two weaknesses to this (besides the strength of the encryption): 1. you have to make sure you have the right decryption key (the one corresponding to the one that the desired data was encrypted in, and not some look-alike data that someone else made up and encrypted) Perhaps the machine could produce a key pair right before election, and distribute the public key, and a trial run could be done to ensure that that is in fact, the key that the machine is using. 2. you have to make sure that the encryption key is kept secure, or even "thrown away" immediately after use, so that people can't generate fake voting records that pass the decryption test. c. digital signing. similar to asymmetric encryption, this helps insure that the data is from a trusted source.
All requirements within the VVSG are being tested and the voting system is verified to be in compliance (Parts 1 and 2): however, this requirement is to again fail the voting system if any of those requirements are not met. This is already a basis for voting system certification and not enhanced by the OEVT.
Despite assurances that the OEVT would not result in a "failing" mark, this section speaks to a troublesome feature of the new VVSG. The fail criteria reveals the redundant nature of the OEVT as it does not consider the role of the VSTL itself during this process.
The low priority assigned to the goals in 5.4.1-C Discussion point 7 seems to contradict 5.4.4 part 4 & 5, which state that "the lab must give a recommendation of 'fail'" when it is possible to modify logs and cause incomplete audits to be generated.
USACM Comment #32. Section 5.4.4-A OEVT Fail Criteria — Failure Interpretation [incomplete] USACM recommends that the following new subsection, with the title above, be added to Section 5.4 as follows: Software testing, including open-ended testing, cannot demonstrate the absence of flaws. Thus, its contribution to the certification process is twofold: a. A final filter to prevent faulty voting system software from achieving certification b. Detect vendors whose development processes are not sufficiently mature to consistently produce high assurance products. The OEVT team should consider a final finding of "failure" to indicate a need to redesign the system or the system testing strategy. DISCUSSION: There is no software testing regimen that can claim comprehensive fault detection. Thus, the best that an OEVT team can hope to do is (1) Detect well-known faults left as a result of immature development processes and (2) Detect subtle faults that the team’s specific skill sets enable them to find and that routine or even mature development processes may not prevent or detect. During deliberations, the OEVT team must assess the vulnerabilities as they apply relative to vendor prescribed procedures. Fail criteria must reflect that an attack based on whether the identified vulnerability would be likely to occur, succeed, and escape detection.
This section further defines concrete requirements for vulnerability testing by specifying the fail criteria for vulnerability tests: a system can fail if 1) the vendors system in conjunction with use procedures and security controls do not adequately mitigate significant threats (Part 3:5.4.4-B); or 2) if found vulnerabilities could be used to: "change the outcome of an election, interfere with voters' ability to cast ballots or have their votes counted during an election, or compromise the secrecy of vote [...]" (Part 3:5.4.4-C). Thus, this section should be adopted by the EAC.
5.4.4-B Threat model - failure
Voting systems SHALL fail open ended vulnerability testing if the manufacturer’s model of the system along with associated use procedures and security controls does not adequately mitigate all significant threats as described in the threat model.
Team may use a threat model that has been amended based on their findings in accordance with 5.4.3-C.
This section should be deleted. The VTSL will determine whether or not a voting system fails certification testing based on the totality of their findings, including the report from the OEVT.
ACCURATE's comments to Part 3:5.4.4-B (including the recommendation to adopt) apply equally to this requirement.
5.4.4-C OEVT fail criteria – critical flaws
The voting device SHALL fail open ended vulnerability testing if the OEVT team provides a plausible description of how vulnerabilities or errors found in a voting device or the implementation of its security features could be used to:
Compromise the secrecy of vote without having to demonstrate a successful exploitation of said vulnerabilities or errors.
The OEVT team does not have to develop an attack and demonstrate the exploitation of the vulnerabilities or errors they find. They do however have to offer a plausible analysis to support their claims.
This sectiion should be deleted. The VTSL will recommend that the system pass or fail based on the totality of the certification testing, including the report from the OEVY.
"The voting device SHALL fail open ended vulnerability testing if the OEVT team provides a plausible description of how vulnerabilities or errors found in a voting device or the implementation of its security features" This section should be stricken as it is too permissive. The OEVT is essentially saying that the security of a voting system doesn't actually have to be penetrated, only a "plausible description" of how the penetration would occur. The bar for failing a voting system has been set low that the OEVT will remain as the final arbiter for the certification of a voting system based on a subjective review and one that only requires a description of events that may cast doubt on the system's security.
USACM Comment #33. Section 5.4.4-C OEVT Fail Criteria - Critical Flaws [incorrect] USACM recommends that subsection 5.4.4-C be modified as follows: The voting device shall fail open-ended vulnerability testing if the OEVT team demonstrates one or more critical flaws that allow an attacker to violate VVSG requirements as specified in paragraph 5.4.4-A above, under a plausible description of how vulnerabilities or errors found in a voting device or the implementation of its security features are used to: a. Change the outcome of an election; b. Interfere with voters’ ability to cast ballots or have their votes counted during an election; or c. Compromise the secrecy of vote without having to demonstrate a successful exploitation of said vulnerabilities or errors. Potential vulnerabilities for which no exploit is demonstrated may be noted as observations, but may not rise to the level of findings. DISCUSSION: OEVT failure is a serious event that may have severe financial ramifications. Thus, it cannot be justified by hypothetical attacks. OEVT testers must be held to high scientific standards that can only be reflected by the three level process of: a. Detecting vulnerability b. Envisioning an exploit for each identified instance and by c. Demonstrating each envisioned attack under plausible conditions.
5.4.5 OEVT reporting requirements
5.4.5-A OEVT reporting requirements
The OEVT team SHALL record all information discovered during the open-ended vulnerability test, including but not limited to:
Names, organizational affiliations, summary qualifications, and resumes of the members of the OEVT;
Time spent by each individual on the OEVT activities;
List of hypotheses considered;
List of hypotheses rejected and rationale;
List of hypotheses tested, testing approach, and testing outcomes; and
List and description of remaining vulnerabilities in the voting system:
A description of each vulnerability including how the vulnerability can be exploited and the nature of the impact;
For each vulnerability, the OEVT team should identify any VVSG requirements violated; and
The OEVT team should flag those vulnerabilities as serious if the vulnerability can result in the violation of one or more VVSG requirements; a change of the outcome of an election; or a denial of service (lack of availability) during the election.
Examples of the impact of an exploited vulnerability are over-count of ballots for a candidate; undercount for a candidate; very slow response time during election; erasure of votes; and lack of availability of the voting device during election.
This section needs to be expanded to state that these results will be contained in a report presented to the VSTL and, furthermore, the OEVT will not release any portion of this report to any organization other thant the VTSL.
Please explain how the results of testing and threat vulnerability be disseminated to state and county election officials. Access to this information is a critical element for the state’s own certification process. There should be a section that clearly outlines how and when the results of OEVT testing will be made available. Moreover, state and county election officials should have access to the actual OEVT report, not just a condensed report by EAC. Please see our suggestion for language below. Within six (6) weeks after testing is complete at the federal level, EAC shall provide to state election officials a copy of any and all OEVT reports for the voting system.
The more reporting, the less testing. .......... If you want time-constrained testing to yield worthwhile test results, most of the time has to be spent on testing (imagining risks,designing tests, implementing tests, executing tests, and evaluating the results). Documentation time is on top of this, and time spent on it subtracts from the time available for the testing. .......... (Affiliation Note: IEEE representative to TGDC)
Focus on defense in depth We approve of the view that the the Red Teaming exercises must consider the entire system---including physical security and security procedures---as a whole. But the current version of the OEVT section strongly implies that, when procedures ameliorate potential threats to the technological parts of the voting system, the weaknesses in the technology would not be considered flaws, and would go unreported. Thus, the reports from the OEVT would not provide enough information for election officials to know the consequences of failing to follow procedural defenses that mitigate technical flaws, or to determine whether the procedural defenses are appropriate for their locality. In our experience, the fundamental principle of defense-in-depth is best captured by layers of procedural and technical defenses. We would like to see the reporting requirements re-written as follows: Reporting requirements: Include in reporting requirements a list and description of any flaws in the voting system that are remediated by procedures, full descriptions of the associated procedures, and the consequences of not following those procedures. We feel such a discussion would be useful to election officials as they integrate the system's use procedures into their local procedures. In this process, the intention behind a specific procedure may be misunderstood (especially those procedures which serve multiple purposes) and integrated into local procedures in a way that does not address all the vulnerabilities. With a discussion of those system threats for which there are no, or very few, technological defenses in place would better assist officials during system integration.
Structured Note-taking [incomplete] USACM recommends adding a paragraph 5.4.5-B as follows: 5.4.5-B. OEVT team process documentation requirement. Each OEVT team will conduct structured note-taking during the analysis. Where possible, all notes will be shared among team members during the entire review period, but must be shared by all members during deliberations, before the final report is prepared. These structured notes become part of the team product and must be delivered along with the OEVT final report. DISCUSSION: It is difficult to overstate the value of structured note-taking during the review process and making the notes database a work-product of each review. The level of continuity it provides between reviews justifies including it as a VVSG requirement. There are also two other benefits that may be equally as important: 1. Process Improvement. Understanding the details of the process that each team goes through can be a gold mine of best practices. 2. Accountability. OEVT is critically dependent on the skill and knowledge of the investigators. Structured note taking provides an avenue to analyze the team’s effort.
5.4.6 VSTL response to OEVT
5.4.6-A VSTL Response to OEVT
The VSTL SHALL examine the OEVT results in the context of all other security, usability, and core function test results and update their compliance assessment of the voting system based on the OEVT.
The testing laboratory should examine each vulnerability that could result in the violation of one or more VVSG 2007 requirements; a change of the outcome of an election; or a denial of service (lack of availability) during the election and use the information to form the basis for non-compliance. If significant vulnerabilities are discovered as a result of open-ended vulnerability testing, this may be an indication of problems with test lab procedures in other areas as well as voting system design or implementation.
Comment by Kevin Wilson (Voting System Test Laboratory)
This section implies the OEVT team is not necessarily a part of the VSTL. Can the OEVT be members of the VSTL?
USACM Comment #35. Section 5.4.6. VSTL Response to OEVT [incomplete] USACM recommends changing the first full sentence in Section 5.4.6-A to read: "The VSTL SHALL: 1. Forward the OEVT results to the VSTL licensing authority for their use in assessing vendor development process maturity and to assess potential corrective action; and 2. Examine the OEVT results in the context of all other security, usability, and core function test results and update their compliance assessment of the voting system based on the OEVT." DISCUSSION: The addition of requirement one will encourage feedback to testing lab authorities and the Election Assistance Commission about issues, errors and anomalies uncovered during the testing process that are not connected to specific requirements of the VVSG. Without a feedback process for problems outside the terms of the VVSG, the testing process would be subject to the voting systems equivalent of teaching to the test — covering only those items outlined in the test, and ignoring anything else — regardless of how it could influence voting, voting administration and elections.