EAC Decision on Request for Interpretation 2009-06 

(Temperature and Power Variation Tests) 

2002 VSS Volume I, Section 3.4.3 

2002 VSS Volume II, Section 4.7.1, 4.7.2, Appendix Sec. C.4 

2005 VVSG Volume I, Section 4.3.3 

2005 VVSG Volume II, Section 4.7.1, 4.7.3, Appendix Sec. C.4 

EAC Decision on Request for Interpretation 2008-1 

Date: 

April 16, 2010 

Questions: 

1. The EAC decision in RFI 2008-1 states that the test length per unit could be as 

short as 64 hours (of which 48 must be in an environmental chamber). Are the 

systems to remain voting in the test ballot counting cycle of Section 4.7.1 for the 

entire period? 

2. If these 2 tests are combined as stated in EAC RFI 2008-1 and Appendix C, how 

can the test method incorporate the statement: 

For example, a typical system operations scenario for environmental operating 

hardware tests will consist of approximately 45 hours of equipment operation. 

Broken down, this time allotment involves 30 hours of equipment setup and 

readiness testing and 15 hours of elections operations. 

3. EAC excerpt from Interpretation: The Reliability test (Section 4.7.2) and the 

Accuracy (Section 4.7.1.1) criteria are constant hazard or time independent test 

criteria (The Accuracy criteria for failures are event based but the same logic 

applies). These are tests where the test criteria are valid for any number of devices 

provided the test represents a full operational cycle of all operating modes over 

the specified total hours. 

4. What exact operations can be completed in a counting cycle as stated in the 

manual or an operational cycle as stated in the EAC interpretation? Are cycles 

different between Temperature Power and Reliability?

1 
Section of Standards or Guidelines: 

2002 VSS Vol. I, Section 3.4.3 & Vol. 2, 4.7.1, Appendix C, Sec. C.4 

2005 VVSG Vol. I, Section 4.3.3 & Vol. 2, 4.7.1 

VSS/VVSG Volume I 

VSS 3.4.3 / VVSG 4.3.3 Reliability 

The reliability of voting system devices shall be measured as Mean Time Between 

Failure (MTBF) for the system submitted for testing. MBTF is defined as the value of the 

ratio of operating time to the number of failures which have occurred in the specified 

time interval. A typical system operations scenario consists of approximately 45 hours of 

equipment operation, consisting of 30 hours of equipment set-up and readiness testing 

and 15 hours of elections operations. For the purpose of demonstrating compliance with 

this requirement, a failure is defined as any event which results in either the: 

 • Loss of one or more functions 

 • Degradation of performance such that the device is unable to perform its intended 

 function for longer than 10 seconds 

The MTBF demonstrated during certification testing shall be at least 163 hours. 

VSS/VVSG Volume II 

VSS / VVSG 4.7.1 Temperature and Power Variation Tests 

This test is similar to the low temperature and high temperature tests of MIL-STD-810-D, 

Method 502.2 and Method 501.2, with test conditions that correspond to the requirements 

of the performance standards. This procedure tests system operation under various 

environmental conditions for at least 163 hours. During 48 hours of this operating time, 

the device shall be in a test chamber. For the remaining hours, the equipment shall be 

operated at room temperature. The system shall be powered for the entire period of this 

test; the power may be disconnected only if necessary for removal of the system from the 

test chamber. 

Operation shall consist of ballot-counting cycles, which vary with system type. An output 

report need not be generated after each counting cycle. The interval between reports, 

however, should be no more than 4 hours to keep to a practical minimum the time 

between the occurrence of a failure or data error and its detection. Test Ballots per 

Counting Cycle 

Precinct count systems 100 ballots/hour 

Central count systems 300 ballots/hour 

2 
The recommended pattern of votes is one chosen to facilitate visual recognition of the 

reported totals; this pattern shall exercise all possible voting locations. System features 

such as data quality tests, error logging, and audit reports shall be enabled during the test. 

Each operating cycle shall consist of processing the number of ballots indicated above. 

Step 1: Arrange the equipment in the test chamber. Connect as required and provide 

for power, control, and data service through enclosure wall. 

Step 2: Set the supply voltage at 117 voltage alternating current. 

Step 3: Power the equipment, and perform an operational status check as in Section 

4.6.1.5. 

Step 4: Set the chamber temperature to 50 degrees F, observing precautions against 

thermal shock and condensation. 

Step 5: Begin 24 hour cycle. 

Step 6: At T=4 hrs, lower the supply voltage to 105 vac. 

Step 7: At T=8 hrs, raise the supply voltage to 129 vac. 

Step 8: At T=11:30 hrs, return the supply voltage to 117 vac and return the chamber 

temperature to lab ambient, observing precautions against thermal shock and 

condensation. 

Step 9: At T=12:00 hrs, raise the chamber temperature to 95 degrees Fahrenheit. 

Step 10: Repeat Steps 5 through 8, with temperature at 95 degrees Fahrenheit, 

complete at T=24 hrs. 

Step 11: Set the chamber temperature at 50 degrees Fahrenheit as in Step 4. 

Step 12: Repeat the 24 hour cycle as in Steps 5-10, complete at T=48 hrs. 

Step 13: After completing the second 24 hour cycle, disconnect power from the 

system and remove it from the chamber if needed. 

Step 14: Reconnect the system as in Step 2, and continue testing for the remaining 

period of operating time required until the ACCEPT/REJECT criteria of 

Subsection 4.7.1.1 have been met. 

VSS/VVSG Volume II 

VSS 4.7.3 Reliability Test 

The ITA shall test for reliability based on the provisions of Volume I, Section 3 for the 

acceptable mean time between failure (MBTF). The MBTF shall be measured during the 

conduct of other system performance tests specified in this section, and shall be at least 

163 hours. Volume II, Appendix C, Section C.4 provides further details of the calculation 

for this testing period. 

3 
VVSG 4.7.3 Reliability Test 

The accredited test lab shall test for reliability based on the provisions of Volume I, 

Section 4 for the acceptable Mean Time Between Failure (MBTF). The MBTF shall be 

measured during the conduct of other system performance tests specified in this section, 

and shall be at least 163 hours. Appendix C provides further details of the calculation for 

this testing period. 

VSS/VVSG Volume II 

VSS / VVSG Appendix C.4 Time-based Failure Testing Criteria 

The equivalence between a number of events and a time period can be established when 

the operating scenarios of a system can be determined with precision. Some of the 

performance test criteria of Volume II, Section 4, use this equivalence. 

System acceptance or rejection can be determined by observing the number of relevant 

failures that occur during equipment operation. The probability ratio for this test is 

derived from the exponential probability distribution. This distribution implies a constant 

hazard rate for equipment failure that is not dependent on the time of testing or the 

previous failures. In that case, two or more systems may be tested simultaneously to 

accumulate the required number of test hours, and the validity of the data is not affected 

by the number of operating hours on a particular unit of equipment. However, for 

environmental operating hardware tests, no unit shall be subjected to less than two 

complete 24-hour test cycles in a test chamber as required by Volume II, Subsection 

4.7.1. 

In this case, the null hypothesis is that the Mean Time Between Failure (MTBF), as 

defined in Volume I, Subsection 4.3.3 is at least as great as some value, here the Nominal 

Specification Value. The alternate hypothesis is that the MTBF is no better than some 

value, here the Minimum Acceptable Value. 

For example, a typical system operations scenario for environmental operating hardware 

tests will consist of approximately 45 hours of equipment operation. Broken down, this 

time allotment involves 30 hours of equipment setup and readiness testing and 15 hours 

of elections operations. If the Minimum Acceptable Value is defined as 45 hours, and a 

test discrimination ratio of 3 is used (in order to produce an acceptably short expected 

time of decision), then the Nominal Specification Value equals 135 hours. 

With a value of decision risk equal to 10 percent, there is no more than a 10 percent 

chance that a system would be rejected when, in fact, with a true MTBF of at least 135 

hours, the system would be acceptable. It also means that there is no more than a 10 

percent chance that a system would be accepted with a true MTBF lower than 45 hours 

when it should have been rejected. 

Therefore, 

H0: MTBF = 135 hours 

H1: MTBF = 45 hours 

a = 0.10 

4 
b = 0.10. 

.... 

Discussion: 

The intention of this RFI is to facilitate testing efficiencies while assuring these 

specifications are fully meet. To clarify the questions addressed in this RFI the 

following discussion and conclusions are offered: 

1. Question: 

The EAC decision in RFI 2008-1 states that the test length per unit could be as 

short as 64 hours (of which 48 must be in an environmental chamber). Are the 

systems to remain voting in the test ballot counting cycle of Section 4.7.1 for the 

entire period? 

Discussion:

The intention of the requirement is stated to be: 

“This procedure tests system operation under various environmental conditions 

for at least 163 hours.” 

Therefore the system must be in the test ballot counting cycle for the entire period 

of the test. The following discussion is offered to provide further detail but does 

not modify this conclusion. 

VSS/VVSG V1:4.7.1, Temperature and Power Variation Tests, states: 

“When the voting unit under test is capable of meeting the minimum counting rate 

the minimum ballot counting rate shall be: 

Test Ballots per Counting Cycle 

Precinct count systems 100 ballots/hour 

Central count systems 300 ballots/hour “ 

For system that are not capable of meeting these rates the system shall be tested 

casting ballots on a continuous basis at a rate determined by the ability of the 

system being tested. The rate to be used shall be determined and included in the 

test plan for the system. 

This rate shall be maintained throughout the duration of test 163 hours or subset 

thereof. If multiple modes of operation are to be tested during this phase 

additional systems can be utilized for this purpose. (The 163 hour test must be 

completed on all equipment, including COTS equipment, but the chamber portion 

is waived for the additional systems. However, the mode most vulnerable to 

failure shall be subjected to the chamber portion. The test plan shall provide an 

analysis explaining the rationale used in selecting the mode to be used during the 

chamber portion of the test). 

5 
2. Question:

If these 2 tests are combined as stated in EAC RFI 2008-1 and Appendix C, how 

can the test method incorporate the statement: 

For example, a typical system operations scenario for environmental operating 

hardware tests will consist of approximately 45 hours of equipment operation. 

Broken down, this time allotment involves 30 hours of equipment setup and 

readiness testing and 15 hours of elections operations. 

Discussion:

The section quotes is part of VSS 3.4.3 / VVSG 4.3.3, Reliability, given as an 

example of how time is distributed in a typical operation, however the 

requirement, given subsequently is to demonstrate an MTBF of at least 163 hours. 

In Annex C additional discussion is given of how to calculate the time required in 

order to demonstrate a specific MTBF. It should be noted that test efficiencies are 

only possible when a solid understanding is available to verify that the failure 

mechanisms are “...a constant hazard rate for equipment failure that is not 

dependent on the time of testing or the previous failures”. Neither of these 

discussions modifies the requirement of VSS / VVSG 4.7.1: 

“This procedure tests system operation under various environmental conditions 

for at least 163 hours. During 48 hours of this operating time, the device shall be 

in a test chamber.”

3. Question:

EAC excerpt from Interpretation: The Reliability test (Section 4.7.2) and the 

Accuracy (Section 4.7.1.1) criteria are constant hazard or time independent test 

criteria (The Accuracy criteria for failures are event based but the same logic 

applies). These are tests where the test criteria are valid for any number of devices 

provided the test represents a full operational cycle of all operating modes over 

the specified total hours. 

Discussion: 

In the Test Plan the VSTL is to provide an analysis that supports its test strategy 

for the specific voting system under test. This analysis must demonstrate that is 

has an understanding of the potential failure modes and their dependencies, e.g. 

time dependent, event dependent. Where the failure modes or their dependencies 

are not fully known the test must be designed to reveal both time and event 

dependent failures. 

4. Question:

What exact operations can be completed in a counting cycle as stated in the 

manual or an operational cycle as stated in the EAC interpretation? Are cycles 

different between Temperature Power and Reliability? 

6 
Discussion:

A ballot counting cycle is the period between the input of the voter’s selections, 

through the processing of the ballot, to the final storage of the cast vote record or vote 

tabulation. The exact beginning and end point will vary by system type but is to 

reproduce system use during an election. The Temperature and Power Variation 

Tests specifically calls for ballot counting cycles to be used. The Reliability test 

allows for VSTL’s to design focused tests for particular failure modes. When the 

tests are combined the ballot counting cycle must be used. If the tests are run 

separately and the failure modes are well known, more focused operations can be 

utilized, so long as they can be defended as fully testing for the specified MTBF of 

the device.

Conclusion: 

1. Question:

The EAC decision in RFI 2008-1 states that the test length per unit could be as 

short as 64 hours (of which 48 must be in an environmental chamber). Are the 

systems to remain voting in the test ballot counting cycle of Section 4.7.1 for the 

entire period? 

Response: 

Yes, the system must be in the test ballot counting cycle for the entire period of 

the test. 

2. Question:

If these 2 tests are combined as stated in EAC RFI 2008-1 and Appendix C, how 

can the test method incorporate the statement: 

For example, a typical system operations scenario for environmental operating 

hardware tests will consist of approximately 45 hours of equipment operation. 

Broken down, this time allotment involves 30 hours of equipment setup and 

readiness testing and 15 hours of elections operations. 

Response: 

The section quotes is an example given in a discussion of the reliability test and 

does not modify the requirement of VSS / VVSG 4.7.1: 

“This procedure tests system operation under various environmental conditions 

for at least 163 hours. During 48 hours of this operating time, the device shall be 

in a test chamber.”

3. Question:

EAC excerpt from Interpretation: The Reliability test (Section 4.7.2) and the 

Accuracy (Section 4.7.1.1) criteria are constant hazard or time independent test 

criteria (The Accuracy criteria for failures are event based but the same logic 

applies). These are tests where the test criteria are valid for any number of devices 

provided the test represents a full operational cycle of all operating modes over 

7 
8 

 

Response:

In the Test Plan the VSTL must demonstrate that is has an understanding of the 

potential failure modes and their dependencies, e.g. time dependent, event 

dependent. This analysis is used to support the design of test for a specific 

system. Where the nature of the failure modes is not fully understood, worst case 

assumptions must be used. The test must be designed to reveal both time and 

event dependent failures. 

4. Question:

What exact operations can be completed in a counting cycle as stated in the 

manual or an operational cycle as stated in the EAC interpretation? Are cycles 

different between Temperature Power and Reliability? 

Response:

The Temperature and Power Variation Tests specifically calls for ballot counting 

cycles to be used. The Reliability test allows for VSTL’s to design focused tests 

for particular failure modes. When the tests are combined the ballot counting 

cycle must be used. If the tests are run separately and the failure modes are well 

known, more focused operations can be utilized, so long as they can be defended 

and fully testing for the specified MTBF of the device.

Effective Date: 

For all systems without an approved test plan.