Source: http://www.eac.gov/vvsg/part_3_testing_requirements/chapter_2_conformity_assessment_process.aspx
Timestamp: 2015-07-05 04:25:06
Document Index: 72123536

Matched Legal Cases: ['art 2', 'art 2', 'art 2', 'art 3', 'art 2', 'art 2', 'art 1', 'art 3', 'art 3', 'art 3', 'art 2', 'art 3', 'art 3', 'art 3', 'art 3', 'art 3', 'art 2', 'art 3', 'art 3', 'art 3', 'art 3', 'art 3', 'art 3', 'art 3', 'art 3', 'art 2']

HomeChapter 2: Conformity Assessment Process
When multiple accredited test labs are being used, the development of the test plan (see Part 2: Chapter 5: "Test Plan (test lab)") and the test report (see Part 2: Chapter 6: "Test Report (test lab)") must be coordinated by a lead accredited test lab. The lead test lab is responsible for ensuring that all testing has been performed and documented in accordance with the VVSG and is ultimately responsible for the summary finding of conformance (see Requirement Part 2: 6.1-F).
<a href="../comments/createComment?section=p3-2.1"> Add a Comment</a>
Comment by Christopher (None)
Accredited testing facilities are fine as long as the source code being compiled into end-point machines is released to the PUBLIC for review at some point. Take the difference between Windows and Linux operating systems. Why is Linux so secure and stable? Because it's constantly being reviewed by thousands of users at the source code level. Open source is the ONLY WAY to effectively review ANY code base where security and reliability are an issue.
"The accredited test lab may determine that a modified system is subjected only to limited testing…" The criteria provided is for functional changes only. Please provide guidance for form and fit changes?
Independent test labs play an important role in the qualification of critical products. VVSG should continue to require conformity assessment by independent test labs. However, it should also enable testing by the general public. .......... Test lab testing is stunningly expensive. Wise manufacturers do the vast majority of their testing in-house, long before submitting software to a regulatory-function independent lab. This is one of the key reasons that the labs normally focus on confirmatory tests, rather than open-ended, harsh investigations. (This is not a universally-accepted characterization of the relationship, but I speak from knowledge of several test lab executives who have been friends, coworkers, legal clients or technical clients and this is what I have heard repeatedly from the sharpest of them, and from several technical staff who are experienced, as vendors of non-voting software products, with taking products through independent lab assessments.) .......... In the case of voting equipment, .......... (1) There is reason to believe that voting equipment vendors' internal quality control processes to date have not been sufficiently thorough, and .......... (2) Potential conflicts of interest could motivate vendor staff to embed code that could be used to change the results of an election. This is a difference in kind from other regulated industries, in which the vendor and its staff have much to lose and nothing to gain by embedding defects into their code. .......... VVSG attempts to compensate for the (deserved or undeserved) lack of trust in vendor quality control and integrity by expanding the role of the independent test lab to include thorough source code review and open-ended vulnerability testing (i.e. exploratory testing). .......... However, the time cost to thoroughly review this much code is enormous and is almost certainly beyond the expectations of the authors of VVSG, the vendors or the labs. Similarly, the time cost to build skills needed to do a thorough job of exploratory testing is well beyond the 12 person-week scope of VVSG. Independent test labs offer strong skills in creating and executing thoroughly documented tests that trace well to unambiguous documents, but these are very different skill sets from those needed for exploratory testing. It is not clear that labs whose core competencies support traditional regulated-industry testing would even know how to assess the competence of the exploratory testing services they would subcontract. .......... Rather than stretching the role, capability, and cost of lab testing beyond sensible limits, VVSG should require voting equipment vendors to enable public testing with the following requirements: .......... (a) non-COTS components of voting equipment software should be publicly reviewable; .......... (b) all test-related artifacts (including test plans and test results) for voting equipment software be public records; .......... (c) all voting equipment specifications other than proprietary specifications for COTS components should be public records; .......... (d) that voting equipment tendered for sale to any government in the United States should be made available to the public at a similar cost, so that researchers can obtain and test the voting system; .......... (e) the license agreement for the voting equipment software must not prohibit publication of any test results, benchmarks, or other results or opinions stemming from evaluation of the equipment; and .......... (f) the license agreement for the voting equipment software must not prohibit reverse engineering any of the equipment's software (including COTS embedded software) to the extent necessary to discover implementation or security flaws. .......... Public testing offers several benefits, mitigating several inherent weaknesses in traditional conformance testing. .......... The underlying problem is that software defects are not like manufacturing defects. There is relatively little value in running the same test over and over. There is great value in testing the software in different ways. Two tests are distinct if the program can pass one and fail the other. For any nontrivial program, the number of distinct tests is virtually infinite. The test design challenge boils down to a selection problem: which tiny subset of the pool of distinct tests should we select? .......... To a degree, the testing problem can be reduced by thorough code review. However, many potential errors cannot be caught in code review. .......... Let me start with an example from one of the most successful computer peripherals manufacturers in the United States. Suppose that we have a pool of N automated regression tests, and in a given build, the software passes M of them. Restrict further work in this build to these M tests that the software passes individually. Do not change the software in any way. Now run random sequences of tests sampled from the M until either the software fails or the software has not failed for a criterion period of time. Under all code coverage models and under most or all failure models used to estimate software reliability, the expectation would be that the system should not fail random sequence testing because it has passed each and every test in the sequence. However, this technique exposes timing problems, conflicts involving multiple processors, memory leaks, memory corruption that builds over time, stack corruption that builds over time, and several other serious problems. Early in development, this manufacturer's code does well to survive a test like this for more than a minute of execution. Release to the public, in this company, requires survival (no failures in a long sequence) times of at least 72 hours. Variants of long-sequence automated testing have been used to qualify telephone systems and other embedded software for at least 25 years. .......... Creating well-documented scripts for this kind of testing would be appallingly expensive. I have never seen a test plan for regulatory independent testing that includes testing of this type. .......... The long-sequence example illustrates the fact that tests can be distinct in subtle ways. Here are more examples: .......... *	Features tested separately can show no problem, but tested together can yield a memory leak or a corrupted stack. .......... *	Features tested with most data values can pass, but can fail on special cases that are hard to predict a priori. In a famous example from the testing of the MASPAR computer's integer square root function, of the 2^32 possible tests (32-bit integer inputs to the function), all tests were run and only two cases failed. Neither case had an input value that was near any obvious boundary. The underlying cause was a rounding error that had an impact on the final result only twice (the error was rounded out before having an impact in several other cases). Boundary testing provides a heuristic, but one that is far from infallible. A more famous but less easily summarized problem of this class was the Pentium FDIV bug. .......... *	As another illustration, on the software-testing mailing list (2/18/2008), Ross Collard (a well known practitioner / teacher) said, "I have been sifting through archived data on bugs found by extensive date testing (Y2K testing), and correlating bugs with the conditions tested. No matter how I choose to define the boundaries, so far I have not found statistically valid evidence for that assertion that "errors lurk at boundaries." .......... *	Features that pass on one configuration can fail under a subtly different one. As a classic example of this, Intuit released a version of Quicken that, to their surprise, crashed during a database search if and only if the computer involved was running Microsoft's new IntelliSense keyboard driver. Other early Windows programs failed on configurations that included both resolution (1600x1200) video and high-resolution (600 dpi) printing. As one example (this problem affected several software developers), in a product that I worked on, no failures occurred with high resolution video but lower resolution printing or high resolution printing but lower resolution video, but in combination, some tasks (such as a print preview) corrupted system memory and caused a system crash. .......... None of these problems stand out in source code reviews. Long-sequence bugs that I personally worked with in telephone systems showed up in code that had been thoroughly reviewed and had been subject to glass-box testing that involved (for all of the code that was eventually implicated in the failure) 100% statement coverage, 100% branch coverage, and testing of every obvious boundary of every obvious (to a person reading the source code OR operating the program black box) variable. .......... Testing is a complex problem. Our current state of the art does not allow us to fully solve it and so we are well served by creating assessment processes that view the code and test the code from widely differing perspectives, in richly different ways. .......... Independent test lab testing is not well suited to this constant variation. Decisions about what to vary in what ways are often intuitive and hard to justify. Auditing the skill and thoroughness of this style of testing is difficult for a test manager working with a skilled staff who are willing to freely share their private thoughts about their strategies and choices. Auditing under more adversarial circumstances would be much more difficult--obviously bad work could be exposed, but the range from not-awful through excellent would be very hard to assess in the face of testing staff who were responding cautiously (defensively) as is not uncommon in adversarial audits. The labs could spend millions of dollars on testing and vendors and regulators could spend hundreds or thousands of hours arguing about whether a pool of tests (and failure rates) was sufficient, representative, fair, etc.--and at the end, we would still have significant uncertainty at high cost. .......... Rather than push labs beyond their zone of excellence, the public testing approach relies on the public to generate a wide variety of approaches that supplement the conformance testing done by the labs. .......... One objection that has been raised to this proposal is that it has been very difficult, in open source projects, to attract sustained testing at this level of skill. Voting systems, though, are special: .......... *	National Science Foundation merit criteria require every proposal to include discussion of the impact of the research for the public benefit. Some researchers will choose to improve the perceived merit associated with their NSF proposals by using voting equipment software as target platforms for the test technique, code review technique, reliability model, or whatever new technology they want to study. As the principal investigator for three NSF projects totaling over a million dollars in funding, and having served on several NSF panels, I would certainly do this whenever possible in my grant proposals and I would expect to see a lot of it in new proposals. These proposals would not be primarily targeting voting systems; they would be using the voting systems as test beds for the ideas that they wanted to explore. However, as they found problems in the voting systems, they would report them. .......... *	Many doctoral students would find it convenient to use voting equipment software as a test bed for their work because the software is thoroughly documented and fully available. Additionally, work on this type of product can neither be dismissed as work on "toy" applications nor as work on low-grade software chosen to misleadingly highlight the strengths of one particular technical approach or unfairly denigrate another. The intimidating oral exam question, "But why did you choose THIS for your test bed?" would be easy to handle--this would be very motivating for several doctoral students, at least many of the ones I know. .......... * It is likely that other nongovernmental organizations would raise money to fund testing efforts for this software. Given the enormous public mistrust, there is an enormous opportunity for fundraising for activities that could be perceived as mitigating the risks that lead to that mistrust. .......... *	With the publication of test plans and results, some test labs will be motivated to prove their capability by demonstrating that their approach to testing exposes bugs missed by the NIST-certified, prestigious labs that tested a given voting system. Such demonstrations would make for useful advertising copy. Live demonstrations of flaws in other well-tested software (e.g. Microsoft Office) have been the core of some keynote addresses at software testing conferences--very powerful marketing for the test group (and test techniques) involved. .......... Another objection raised to this proposal is that the results go nowhere. That is, if a research group does find a defect in a voting system, there is nothing in VVSG that closes the loop, requiring immediate repair by the voting equipment vendor. This might be true today, and it might stay true in all subsequent versions of VVSG, but if members of the public find and publish defects in a voting system, this can affect buying decisions by subsequent potential purchasers and it can also affect the reputation of the test lab that signed off on the system. Over time, systematic weaknesses in the assessment of voting systems will be understood and mitigated. .......... (Affiliation Note: IEEE representative to TGDC)
VVSG specifies the testing that an independent test lab will perform. .......... The first problem with this process is that the equipment vendor picks the test lab. This creates a strong incentive for the test lab to please the vendor, in order to obtain the vendor's repeat business and the business of this vendor's competitors. The labs therefore have a disincentive against creating tests that are more harsh or more extensive (more time consuming and more expensive) than the bare minimum specified in VVSG. This is not a genuinely independent set of tests and it is a poor way to engender public trust in the test results. .......... VVSG must address and eliminate the problem of test lab conflict of interest. .......... (Affiliation Note: IEEE representative to TGDC)
The accredited test lab determines which tests are necessary to reassess a modified system based on a review of the nature and scope of changes and other submitted information including the system documentation, manufacturer test documentation, configuration management records, and quality assurance information. The accredited test lab may determine that a modified system is subject only to limited retesting if the manufacturer demonstrates that the change does not affect demonstrated compliance with these VVSG for:
Performance of voting system functions;
The accredited test lab may determine that a modified system is subject only to limited retesting if the manufacturer demonstrates that the change does not affect demonstrated compliance with these VVSG for: Performance of voting system functions; Voting system security and privacy; Overall flow of system control; and The manner in which ballots are defined and interpreted, or voting data are processed. The EAC needs to ensure that not only functional changes are looked at but also system changes. Ex. A new micro processor could meet all the above requirements, but have flaws elsewhere if not thoroughly tested.
Since there are likely to be tests dependent upon the results of prior tests (e.g., progressive or cumulative tests), recommmend that in those instances, the test lab shall document in advance the planned sequence or progression of tests.
<a href="../comments/createComment?section=p3-2.4"> Add a Comment</a>
Conformity assessment is conducted for the initial version of a voting system as well as for all subsequent revisions to the system that are to be used in elections. As described in Part 3: 2.2 "Scope of Assessment", the nature and scope of testing for system changes or new versions is determined by the accredited test lab based on the nature and scope of the modifications to the system and on the quality of system documentation and configuration management records submitted by the manufacturer.
The EAC process should be referenced in this documentation. When is a lab the lead VSTL under the EAC program?
The manufacturer prepares and submits a TDP to the accredited test lab. The TDP consists of the materials described in Part 2: Chapter 3: "Technical Data Package (manufacturer)";
The test lab witnesses the production of the implementation for testing; What is meant by this statement? It is very vague.
The manufacturerSHALL submit to the test lab a Technical Data Package conforming to the requirements of Part 2: Chapter 3: "Technical Data Package (manufacturer)".
Manufacturers may seek to market a complete voting system or an interoperable component of a voting system. In all instances, manufacturers must submit for testing the specific system configuration that will be offered to jurisdictions or that comprises the component to be marketed plus the other components with which the component is to be used. Under no circumstances will a component be assessed except as part of a complete voting system, and that assessment is valid only when that component is used with that same system (see Part 1: 2.3 "Conformance Designations").
Comment by Gail Audette (State Election Official)
Is this VVSG allowing for certification of individual components? If so, where is the guidance for component testing within end-to-end testing? What is the lab reporting for certification testing? How will this be reconciled with the NIST 150-22 definition of a voting system?
Certification of individual components is not consistent with the current EAC guidelines or NIST guidelines on what a voting system is. Careful attention should be given to this subject as how this would affect cross market utilization and testing requirements.
If needed for compliance with Part 3: 2.4.3.4 "Unmodified COTS verification", the manufacturer SHALL supply the system with the COTS components omitted, for subsequent integration performed by or witnessed by the test lab.
See Part 3: 2.4.3.4 "Unmodified COTS verification".
The implications of the VSTLs procuring all COTS equipment for testing could adversely affect the cost of testing, as we will not be procuring the quantities or have the deals that manufacturers have with vendors for supplying these items. VSTLs should verify that COTS are available and are the same as the manufacturers provide.
Recommend enhancing the stated criteria to be "quality, form, fit, function, durability, performance, and reliability." The intent is to have the hardware submitted for conformity assessment to be as identical as possible to the actual production verson and not to just be "basically the same."
Hardware should be THE same as the production version. VSTLs are supposed to test the actual version that is being sold to the customers not prototypes. See 2.4.2.2-D
Recommend that this requirement be revised to never allow developmental prototypes to be submitted for conformity assessment. Best practices would never allow for this. However, if (for some reason) this must be allowed to occur, then recommend enhancing the stated criteria to be "quality, form, fit, function, durability, performance, reliability, and construction" (instead of just performance and construction). The intent is to have the equipment submitted for conformity assessment to be as identical as possible to the actual production verson and not to just be "basically the same." (Preference would be to never allow this practice at all.)
Contradicts what is said in 2.4.2.2-B
Please provide the rationale for allowing developmental prototypes to be accepted for testing as opposed to only the standard production units that will be placed in counties. Does allowing the developmental prototypes in some way contravene the stepped-up requirements and security measures in this iteration of the guidelines?
Previously built voting system software being updated may be able to use the requirements found Part 3: 2.4.3.3 "Updating previously built voting system software executable code" to create the updated executable code including application logic, border logic, and third party logic.
The document does not mention that a copy of the public key of the "digital signature" must be maintained with the software and trusted build or else the "digital signature" is effectively useless outside of the originating agency (and possibly even there if the public key is lost). Perhaps a better and more clear way to do so is to specify that an X.509 certificate be included on the CD and thus more clearly defining the intent of this section.
Source:	[EAC06] Section 5.6 and [VVSG2005] II.1.8.2.4 1 Comments
Recommend the requirement be established that this manufacturer representative must have demonstrated technical expertise and extensive, first-hand experience and knowledge of the build environment assembly.
This requirement needs to define where and how the test lab will obtain the software and hardware required to establish the build environement. (Requirement Part 2: 3.5.4-C only indicates that "...the manufacturer SHALL provide a list of all software and hardware required to assemble the build environment...")
Should state that the manufacturer will provide the test lab with the official version of the software and hardware required to establish the build environment.
Recommend that any suppliers used for procurement of COTS software and hardware for the build environment meet the same quality requirements as was used to procure the voting machine hardware and software. This just makes sense, since the build environment must be (at least) as good as the product being tested (hopefully, better in order to detect defects). Also, recommend that the test lab maintain a register of these approved suppliers that includes the scope of the approval (i.e., the quality requirements used to determine the suppliers' approval status). This is the best method for developing a supply base that supports material requirements for quality and other testing objectives for the test lab.
It is common for the current voting systems to rely upon COTS products that are no longer available from the open market. Is it the intention of this this requirement to effectively remove those products from the market? Is there an alternative allowable chain-of-evidence for such situations?
The implications of the VSTLs procuring all COTS equipment for testing could adversely affect the cost of testing as we will not be procuring the quantities or have discounts that the manufacturers offer to vendors for supplying these items. VSTLs should verify that COTS is available and the same as what the manufacturers provide.
Due to the fast changing COTS market and the slow pace of Voting System Certification and Development there is a very real possibility that the COTS software required to build the product will no longer be available on the open market.
EAC06, Section 5.6.1.1 states that "The device that will hold the build environment shall be completely erased by the VSTL to ensure a total and complete cleaning of it. The VSTL shall use commercial off-the-shelf software, purchased by the laboratory, for cleaning the device." It would not appear that "simply deleting files" meets the criteria defined in EAC06 5.6.1.1 to "...ensure a total and complete cleaning...." (In fact, EAC06 5.6.1.1 would not require the use of "COTS, purchased by the laboratory, for cleaning the device" if a simple deleting of files would suffice!) Data remanence plays a major role when storage media is erased for the purposes of reuse or release. (Data remanence is the residual physical representation of data that has been in some way erased.) After storage media is erased there may be some physical characteristics that allow data to be reconstructed. The integrity of the conformity assessment process will be compromised if this possibility is not accounted for.
In the discussion it is stated that simply deleting files...satsfies the requirement. The source cited is the EAC Testing and Certification Program Manual. The section cited states that the device holding the build environment shall use commercial off-the-shelf software to clean the device. This seems to contradict the VVSG requirement. Also, any other upper-level procedures for deleting files should be referenced.
Note that if deviation from the assembly procedures results in a different build environment than what was used by the voting machine manufacturer, then this should not be allowed. Need to ensure that the manufacturer can demonstrate that validation activities were performed successfully. (Validation: to demonstrate that a product or product component fulfills its intended use when placed in its intended environement.) Need to ensure that identical assembly procedures occur for an identical build environment.
The Test Lab should determine if changes should be incorporated into the TDP and state why not, if not included.
Source:	[EAC06] Section 5.9 2.4.3.1-A.10 Build environment pre-build binary image copy
The build environment is established using the requirements in Part 3: 2.4.3.1 "Build environment establishment".
<a href="../comments/createComment?section=p3-2.4.3.2-A.2"> Add a Comment</a>
Recommend that no deviation from the software executable code build procedures should be permitted. This goes to the heart of the integrity of the voting machine software. If the software build process is not repeatable then, nothing can be assured. If the code build procedures are not stable then, this puts into question all verification and validation activities performed by the manufacturer. (Verification: to ensure that the selected work products meet their specified requirements.) (Validation: to demonstrate that a product or product component fulfills its intended use when placed in its intended environement.) Need to ensure that build procedures from the manufacturer are accurately described and repeatable.
The following voting system software build requirements apply when updates to previously built voting system software has occurred. These requirements assume the original build environment can be used to create the updated software and a significant portion of original software is not being updated. If the original build environment cannot be used or a significant portion of the original software is being updated, then the requirements of Part 3: 2.4.3.1 "Build environment establishment" and Part 3: 2.4.3.2 "Build of voting system software executable code".
<a href="../comments/createComment?section=p3-2.4.3.3"> Add a Comment</a>
The last sentence in this paragraph needs to be completed.
Comment by k (Voting System Test Laboratory)
This section has inverted the 2005 requirements by requiring a trusted build prior to a witness build. It is not clear how these two terms are being used and what is the difference between a trusted and witnessed build. This needs to be clarified.
Recommend avoiding the use of the term "significant portion" that can be subject to different interpretations. At the very least, this term should be more clearly defined or quantified.
The final statement "If the test lab does not possess the required hardware and software to create the build" conflicts with requirement of 2.4.3.1-A.4.
As commented in Part 3, Chapter 2.4.3.1-A.5: In the discussion it is stated that simply deleting files...satsfies the requirement. The source cited is the EAC Testing and Certification Program Manual. The section cited states that the device holding the build environment shall use commercial off-the-shelf software to clean the device. This seems to contradict the VVSG requirement. Also, any other upper-level procedures for deleting files should be referenced.
<a href="../comments/createComment?section=p3-2.4.3.3-B.2"> Add a Comment</a>
Source:	[EAC06] Section 5.9 2.4.3.3-B.4 Original post build environment re-establishment record
Source:	[EAC06] Section 5.9	2.4.3.3-C Build of updated voting system software executable code
This requirement does not modify the requirement found in Section 5.6.4.2 of the EAC Testing and Certification Program Manual [EAC06] that states the executable files are created; and extends the requirement found at Section 1.8.2.4 of [VVSG2005] Volume II in [VVSG2005] by requiring the use of the build environment established in Part 3: 2.4.3.1 "Build environment establishment".
This requirement modifies the requirement found in Section 5.6.4.2 of the EAC Testing and Certification Program Manual [EAC06] by constraining the verification to digital signature from a "file signature" (which could be a hash value or digital signature); extends 5.6.2.1 by specifying the verification to happen before software is installed on the build environment; and does not call for the digital signature of the build environment to be verified before installing the source code.
Here the concept of a "digital signature" is declared to be a constraint to a "file signature". It would be better to make this declaration in section 2.4.3.2-A.8 addressing the Trusted Build and not here where the Witness build is addressed.
This section describes greater definition over a witness build than the trusted build from which it owes its trust. Why isn't this information in the preceding trusted build section? (see comment submitted to 2.4.3.3)
This requirement differs from the requirement found in Section 5.6.2.2 of the [EAC06] by creating a digital signature on the binary image after the software executable code has been built as opposed to a "file signature" which could be a hash value or digital signature before the software executable code is built; although requirement 5.6.3.1 of the EAC Testing and Certification Program Manual requires "file signatures" for updated executable code.
Yet again the requirements fail to mention a copy of the public key of the digital signature without which the digital signature is so many useless bits. Worse even than a "file signature" since without the public key it is undecipherable to the hash (file signature) it contains. (see comment 2.4.3)
Test labs will procure the COTS components "off-the-shelf" from suppliers of the COTS components.
<a href="../comments/createComment?section=p3-2.4.3.4-B"> Add a Comment</a>
Recommend that since the test labs are ultimately responsible for the quality of all products purchased from suppliers, the test labs should evaluate and select suppliers based on their ability to supply products in accordance with the test lab’s requirements. A register of these "approved suppliers" should be maintained. (The criteria for selection, evaluation and re-evaluation of these suppliers should be established.)
Does the NIST envision that voting systems will provide such a large market for COTS as to keep the COTS software available on the open market for as long as the voting system itself is marketable? The standard should address acceptable contingent methods.
he implications of the VSTLs procuring all COTS equipment for testing could adversely affect the cost of testing, as we will not be procuring the quantities or have the deals that manufacturers have with vendors for supplying these items. VSTLs should verify that COTS are available and the same as the manufacturers provide.
<a href="../comments/createComment?section=p3-2.4.3.4-C"> Add a Comment</a>
<a href="../comments/createComment?section=p3-2.4.3.4-C.1"> Add a Comment</a>
<a href="../comments/createComment?section=p3-2.4.3.4-C.2"> Add a Comment</a>
<a href="../comments/createComment?section=p3-2.4.3.4-C.3"> Add a Comment</a>
If the assembly and configuration of the COTS components is different from that submitted by the manufacturer, then this should not be allowed. Need to ensure that the manufacturer can demonstrate (and document) that validation activities were performed successfully. (Validation: to demonstrate that a product or product component fulfills its intended use when placed in its intended environement.) Need to ensure that identical assembly and configuration procedures occur for identical COTS components.
Test Lab should determine if changes should be incorporated into the TDP and stay why not, if not included.
<a href="../comments/createComment?section=p3-2.5"> Add a Comment</a>
Requirements on the content of the test plan are contained in Part 2: Chapter 5: "Test Plan (test lab)".
<a href="../comments/createComment?section=p3-2.5.1-A"> Add a Comment</a>
<a href="../comments/createComment?section=p3-2.5.2"> Add a Comment</a>
The accredited test lab may perform the tests in any facility capable of supporting the test environment. Directly contradicts EAC Lab accreditation manual currently under review, Section 2.11.6.
<a href="../comments/createComment?section=p3-2.5.2-A"> Add a Comment</a>
For this requirement, need to define "independent" and "qualified observer." Also, need to describe how this individual is selected, how the observer will "attest", how this will be documented, etc.
What is meant by tests "shall be witnessed by at least one independent qualified observer"? Does this mean the labs must now hire another person or agency to oversee testing?
Recommend more clearly defining "independent, qualified observer."
Comment by E Smith/J Homewood (Manufacturer)
There is a requirement for "at least one independent, qualified observer" too observe the setup. Who determines the person is independent and qualified? How does this observer get paid and stay independent? Will this be a representative of the EAC permanently available on site?
When a test is to be performed at conditions other than "standard" or "ambient," the test SHALL be performed at the required temperature and electrical supply voltage, regulated within the following tolerances:
Temperature ± 2.2 °C (± 4 °F)
<a href="../comments/createComment?section=p3-2.5.2-C"> Add a Comment</a>
Anomalies should also be reported to EAC per guidelines.
<a href="../comments/createComment?section=p3-2.5.5-C"> Add a Comment</a>
The term "a logic defect" is so broad that this requirement is unworkable. If a recording, reporting or tabulation defect is found during functional testing, how are the labs to interpret whether the defect is a "critical software defect"? How can it be ensured that all the labs can apply such nebulous criteria consistently? The VVSG definition of a logic defect is a "fault in software, firmware or hard wired logic". If the test campaign is terminated and the system rejected for recording, reporting or tabulation "logic defect", is the new application a software application or an EAC application?
I agree with that the VVSG comment that "the system cannot be considered trustworthy even after the known fault is corrected, because the cases that the test lab does not have the opportunity to test can be expected to conceal similar faults." .......... However, mitigation of this risk does not merely require "a new application and starting [the testing] over" (the VVSG requirement). Genuine mitigation of this risk requires new tests that look for those "similar faults" in other parts of the code that have not been tested this way for this particular type of error. .......... (Affiliation Note: IEEE representative to TGDC)
Rejection may be a foregone conclusion if sufficient evidence has been collected to show that the reliability benchmark is not satisfied (see Part 3: 5.3.3 "Reliability"). Notwithstanding that, the manufacturer will be given the opportunity to correct noncritical software defects. Revisions to the software must be performed within the manufacturer's quality assurance and configuration management processes and must undergo manufacturer regression testing before the conformity assessment process is resumed. When it is resumed, the test plan should be revised to include regression testing for the change that was made.
Rejection may be a foregone conclusion if sufficient evidence has been collected to show that the reliability benchmark is not satisfied (see Part 3: 5.3.3 "Reliability"). Notwithstanding that, the manufacturer may replace a component that has suffered a random failure, or the manufacturer may opt to suspend the test campaign in order to correct a hardware design defect that caused a nonrandom failure.
In the discussion it indicates manufacturer's may replace a component that has suffered a random failure. Please provide explicit instruction on when a component may be replaced. Providing a discussion that says it may be replaced and referencing a section that says all testing is pertinent to reliability except when you force the system to fail or bypass functionality is confusing. The issue of replacement of a random failure is important and the VVSG needs to provide unambiguous direction.
Please clarify if the allowable correction time for an anomaly other than a logic defect is clock time or work hour time? At what point is the eight hour timer started, meaning does this include or exclude the time to troubleshoot the anomaly?
Please clarify the timeline: does this include travel, troubleshooting or actual work time? What is the determination on what can and can not be replaced? This is not very clear.
Need to define the phrase "extended period of time" (is it 8 hours, 16 hours, 4 hours?). How much later can a test be resumed? Is "a later date" defined as 3 days, 1 week, etc.? With the complexity of hardware and software components nowadays and with no good way of knowing how one code module might impact another code module (strange things happen with software), recommend that if any design or manufacturing change has been made that all procedures must be repeated (whether or not it is assumed that earlier test results would not be invalidated). Again, we're trying to assure everyone of the quality of the product. As much as is possible, no room should be left for doubt.
We take issue with the phrase "extended period of time." States that require a voting system to undergo federal testing prior to certification for use in the state should not have to wait for some indefinite period of time for the federal testing to be completed. Please see our suggestion for language below. If the test campaign is suspended for an extended period of time, the accredited test lab SHALL maintain a record of the procedures that have been satisfactorily completed. When testing is resumed at a later date, repetition of the successfully completed procedures SHALL be waived provided that no design or manufacturing change has been made that would invalidate the earlier test results and the test campaign was suspended for a period no greater than six (6) months. No waiver SHALL be available where a suspension is greater than six (6) months. Any subsequent testing of a system after the six-month period requires a new application and starting over.
How will it be determined that no design or manufacturing change has been made once testing is resumed? Please further explain.
The test campaign may resume after a deficiency is found if:
The manufacturer submits a design, manufacturing, or packaging change notice to correct the deficiency, together with test data to verify the adequacy of the change;
Regarding 2.5.5-G(d), this will need to be verified. Verification, testing, and validation activities are not about taking someone's word. The incorporation of the change into all existing and further production units must be independently (of the manufacturer) verified.
What constitutes a "deficiency?" Please more fully explain.
<a href="../comments/createComment?section=p3-2.6.1"> Add a Comment</a>
The hyperlink for "third party logic" is not activated anywhere in this section.
The term "test lab build" refers to the voting system software executable code (including application logic, border logic, and third party logic) resulting from the test lab creating the executable code using (a) test lab procured equipment and build tools (such as compilers, linkers, etc.) and (b) source code and build procedures provided by the manufacturers. Note the test lab build is the result of using the requirements found in Part 3: 2.4.3 "Initial system build by test lab".
When required by Requirement Part 3: 2.6.1.1-A.2, the test lab SHALL use the requirements found in Part 3: 2.4.3 "Initial system build by test lab" to create a final test lab build of voting system software executable code including application logic, border logic, and third party logic
Software is traceable back to a software distribution package master copy containing the software. Copies of software distribution packages can be distributed on via modifiable media (physically on CD-RWs, memory cards, and hard drives; or electronically via email, FTP, and Websites) since digital signatures are created as part of software distribution packages. (See Requirement Part 3: 2.6.2.1-F)
Although the document specifies that the master copy contains the digital signature algorithm, it does not specify that the master document contains the public key or public key certificate of the digital signature nor does it specify a repository for such a public key. (See comment 2.4.3) A digital signature without a public key has less value than a file signature alone in terms of the possibility of making any type of verification of that signature.
Digital signatures are generated for the un-archived forms of each of the software files as well as archive files.
Some non-proprietary standard data formats for digital signatures include IETF RFC 3852: Cryptographic Message Syntax (CMS), RSA Public Key Cryptographic Standard #7: Cryptographic Message Syntax Standard, W3C XML-Signature Syntax and Processing.
<a href="../comments/createComment?section=p3-2.6.2.1-H"> Add a Comment</a>
"..the archive files SHALL be generated using algorithms and file formats in common usage." Some commonly used archive file are listed in the discussion; however, how does a lab determine if the vendor used archive files algorithm and/or file format is "common usage"? Please provide a list of acceptable formats and/or clarification of the steps to assess a format is "common usage".
This requirement is a bit confusing since a "binary image" of something that contains the generated signature will necessarily be changed when the generated signature is added to the "binary image"
This requirement allows notary repositories to provide the software integrity information they create for voting system software to parties that follow the request process documented by Requirement Part 3: 2.6.2.2-A.
This requirement establishes the software distribution package master copy that supports traceability of voting system software source and executable code back to the test lab.
Source:	[EAC06] Section 5.6.3.1
What organization is contracting with the repository?
Typographical errors in the discussion section – remove "in" in the first sentence, and use either "on" or "via" in the second sentence, but not both.
Sections 2.6.2.3 & 2.6.2.4 The requirements for both the test lab and manufacturer are duplicated. What is the reconciliation process, validation process, or verification step that justifies this duplication of effort?
This requirement establishes the software distribution package master copy that supports traceability of configuration files, installation programs, and third party software to be installed on programmed devices of the voting system back to the test lab.
Source:	[EAC06] Section 5.6.3.1 and 5.6.3.3
<a href="../comments/createComment?section=p3-2.6.2.3-B"> Add a Comment</a>
This requirement establishes the software distribution package master copy that supports traceability of configuration files, installation programs, and third party software to be installed on programmed devices of the voting system back to the test lab. Manufacturers will include a copy of this software distribution package as part of their TDP as required by Requirement Part 3: 2.6.2.4-D.
The accredited test lab SHALL prepare a test report conforming to the requirements of Part 2: Chapter 5: "Test Plan (test lab)".