Source: http://www.eac.gov/vvsg/part_2_documentation_requirements/chapter_2_quality_assurance.aspx
Timestamp: 2013-05-22 07:54:34
Document Index: 124927908

Matched Legal Cases: ['art 1', 'art 1', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2', 'art 2']

Chapter 2: Quality Assurance and Configuration
HomeChapter 2: Quality Assurance and Configuration
This section contains requirements on the content of the quality assurance and configuration management documentation that manufacturers must supply to the certification authority.
Comment by Patricia Berg (Advocacy Group)
For some reason I cannot get onto the next page containing Chapter Two subheading 4, regarding Software Independence. Because I have heard from a friend that the section after that one has been changed to make a VVPAT (Voluntary Voting Paper Audit Trail) legitimate to be used with the fully computerized DRE machines to continue to use this so-called paper trail. If this is so, then the Election Assistance Committee still does not understand what the scientists at NIST and the TGDC have been pointing out to them. In Cuyahoga County, after the 2006 primaries, the Election Science Institute performed a systematic audit on the Diebold DREs, which proved that the four recorded totals, the individually counted votes recorded on the paper trail, plus the paper trail summary of the votes, plus the memory card record of the total, and, finally, the ballot archive totals (presumably on the hard drive) were all different numbers from each other. When compared on six vectors to one another the differences between any two totals was as much as 20%, if I remember correctly. Counting paper ballots by hand would be far more accurate, and accuracy is above all what we need for election totals. The idea that a DRE could be developed that would not have this potential for error is beyond current technical expertise, from everything that I have read on the subject. The problem is that the software that does the counting also does the recording, and, whether by malicious intention or human error, the outcome cannot be trusted. This is particularly true when the software is proprietary, i.e., the secret intellectual property of a corporation which cannot be inspected routinely by the local election authorities. Even if the so-called "VVPAT" were somewhat reliable, a study done by MIT scientists showed that only about 38% of the voters check to see if the paper recorded their vote correctly. The paper trail on the iVotronic machines with which I am familiar in St. Louis County is difficult to read because of small font size and sometimes a light color, as though the ink is running out. These "paper trails" also have a high rate of getting stuck, crumpled, torn, etc., so they can be difficult to recount. I can understand the the members of the EAC being administrators who are faced with a problem that was created by legislators who wrote HAVA. I can understand them being under pressure from the White House to whitewash the mistakes made in previous elections. But I cannot understand their claim to being bona fide American citizens who also vote themselves while they still continue to support technology that has failed miserably in multiple states. Please wake up to the realities of the fact that history will prove that the EAC was not financially supported for the role it was intended to play. Read the writings of the first Director, Forest De Soaries. And do your bound duty to properly support a true vote count, by eliminating the DRE computerized vote counting machines from the November 2008 elections. Other ways to ascertain that disabled citizens can vote can be obtained, such as the ballot marking machines. I have heard that most disabled voters prefer to vote absentee, although I certainly do not support a massive move to mail-in ballots as a solution. Please attend to the voluminous research that has been done by citizen powered concerns, and help the United States remain a representative democracy in this hugely changed world. Thank you for the patience it requires for you to continue to read the comments from many citizens, including scientists, activists and, undoubtedly voting machine corporate members, among others. Those of us who want to get rid of the faulty technology that has been funded into existence by an ignorant Congress are not on the opposite side of the disabled. We stand with them, wishing their votes to be as accurately counted as possible. Patricia M. Berg Missourians for Honest Elections pat.rivercritter@gmail.com
Comment by Diane Gray (Voting System Test Laboratory)
This paragraph states manufacturers must submit documentation to the "certification authority". Introduction Chapter 1.3 Audience lists "National Certification Authority". Part 1 Chapter 5.1.3 refers to "a Certification Authority or "CA". This can become confusing.
2.0. How is a "certification authority" different from a "test lab"? 2.1-A. "complete" is untestable. 2.1-A.1. "detail" is not defined. 2.1-A.2. "declare" is not defined. "binding commitment" is not defined. "Entire VVSG" is unclear, as the VVSG contains requirements that are not mandatory (voting variations, for instance). What is an "entire manufacturer organization"? Does it include only manufacturing, not development, support, accounting, etc.? What if a large corporation owns a voting system company as an independent division? 2.1-A.2. Why is this "binding commitment" buried in the QA/CM manual? Should it not be part of the overall certification application to the EAC? 2.1-A.3. "clearly and unambigously" is neither. 2.1-A.5. "difficulties" is not defined; the discussion suggests a definition that is irrelevant and silly. 2.1-A.10. "all" implies that a sampling/statistical approach is not allowed. Please confirm. 2.1-A.10. The last sentence should be changed to require the records be retained as long as the voting system model is in use. 2.1-A.11. "all" implies that a sampling/statistical approach is not allowed. Please confirm. 2.1-A.12. This is pure insanity when applied to software modules. "might have" is speculative and untestable. 2.1-A.12. The last sentence does no indicate who is allowed to inspect the records. The VSTL? EAC? Jurisdiction? Public? 2.1-A.14. "Any defects" is vague. "difficulties" is vague. 2.1-A.15. "identify and maintain the financial capability" has no place in a VVSG
2.1 Quality and Configuration Management Manual
2.1-A Develop and present
All voting system manufacturers SHALL develop and present to the certification authority a complete Quality and Configuration Management Manual.
Applies To:	Voting system
Source:	New requirement
Comment by Gail Audette (Voting System Test Laboratory)
These requirements are based on a new design/development process. The majority of the systems being submitted for VSTL testing are legacy systems. How are these new requirements to be implemented by legacy systems?
2.1-A.1 Processes and procedures
The Manual SHALL detail the manufacturer's Quality Assurance and Configuration Management processes and procedures required by the VVSG. These processes and procedures SHALL conform to all requirements of the VVSG and the standards listed in Requirement Part 1: 6.4.2.1-A.
Recommend that all of the manufacturer's processes and procedures be certified to ISO 9001:2000 (and not simply "conform to"). This will go a long way to ensuring customer satisfaction and acceptance by the public by having an independent third party verify the status of the manufacturing organization's policies, processes, and procedures.
Comment by C. Coggins (Voting System Test Laboratory)
The scope of "all requirements of the VVSG and standards listed …" appears to be excessively broad. The requirement should specify the particular sections.
The standards referred to include ISO Standards. Please give guidance on how to ensure manufacturers conform to the referenced standards.
2.1-A.2 A binding commitment
The Manual SHALL declare that meeting the requirements of the entire VVSG is a binding commitment for the entire manufacturer organization.
2.1-A.3 Project plan
The Manual SHALL provide for the formulation of a project plan for the design and development of a voting system. It SHALL require the project plan to be clearly and unambiguously documented.
The project plan should be consistent with the Design and Development Planning requirements, as specified in ISO 9001:2000, Quality management systems – Requirements [ISO00] Section 8.3.1.
There is a problem with the reference in this paragraph. ISO 9001:2000, Quality management systems – Requirements, does not have a Section 8.3.1. (Section 8.3 in that standard is Control of Non-Conforming Product.) I think that what you meant to reference was Section 7.3.1, Design and Development Planning (not 8.3.1).
The project plan shall be a public record. .......... (Affiliation Note: IEEE representative to TGDC)
2.1-A.4 Quality check
The Manual SHALL require the project plan to include, at a minimum, one quality check at the end of the design phase, and one quality check at the end of the development phase. The project plan SHALL define the progress that is required before each quality check can be passed.
A "quality check" is the sum of the activities Design and Development Review, Design and Development Verification, and Design and Development Validation, as defined in [ISO00] Sections 7.3.4. through 7.3.6.
"Quality Checks" (also known in the industry as peer reviews) are invariably tied to the size of the product being developed. With the requirement being stated as "at a minimum, one quality check...," the tendency will be for exactly one (1) quality check to be planned. Having a single quality check at the end of the design phase and another at the end of the development phase for a product anticipated to be (for example) 30,000 lines of code certainly is not sufficient. Recommend that this requirement be changed to indicate that the number of "quality checks" be calculated based on industry standards for the number of lines of code being developed (or function points or some other standard) as well as the software language being used.
Since the required quality checks include specific ISO standards, could there be a standard template which includes all of these requirements?
The results of each quality check shall be a public record. .......... (Affiliation Note: IEEE representative to TGDC)
2.1-A.5 Problem log
The Manual SHALL require the manufacturer to maintain a log in which all difficulties encountered during the design and development phase for a voting system are required to be recorded. Any remedial action taken to correct a difficulty SHALL also be recorded. The log SHALL be available for inspection by the test lab.
"Difficulties" are any occasions when it is recognized that changes in past design decisions or in the project plan (see Requirement Part 2: 2.1-A.3) are necessary to complete the project.
This is an area that will benefit if the manufacturer is required to be certified to ISO 9001. Section 7.3.7 (Control of Design and development changes) of ISO 9001 requires that "Design and development changes shall be identified and records maintained. The changes shall be reviewed, verified and validated, as appropriate, and approved before implementation. The review of design and development changes shall include evaluation of the effect of the changes on constituent parts and product already delivered. Records of the results of the review of changes and any necessary actions shall be maintained." Requiring the manufacturer to be certified to ISO 9001 (instead of simply conform to ISO 9001 which gives the manufacturer too much wiggle-room) wil ensure that all changes during the development stage will be identified and documented, then approved by someone with the proper authorization.
Comment by Premier Election Solutions (Manufacturer)
Although this requirement may be a worthwhile endeavor and is good development practice, it is questionable as to whether it should be included in this standard. There are many well recognized Qualtity Management standards (such as ISO 9001) where it would be more practical to simply refer to those standards.
The problem log shall be a public record. .......... (Affiliation Note: IEEE representative to TGDC)
2.1-A.6 Critical parts, components, and assemblies
The Manual SHALL specify rules that define what parts, components, and assemblies of the voting system are to be considered as critical. A part, component, or assembly SHALL be defined as critical if its failure may:
Cause a faulty display of options;
Cause an uncertainty if voter's choice has been recorded;
Cause a false recording of vote cast;
Cause the change of stored votes;
Cause the false transmission for polling station totals;
Cause injury to voters or staff;
Provide an opening for tampering;
Violate a voter's privacy;
Cause a false accumulation of polling station totals;
Cause a false transmission for regional totals;
Give the appearance of irregularity;
Violate a voter's ability to vote independently; and
Impede the usability of the polling station for all voters.
As used here, "components" include software modules.
Comment by C.S. Van Nostrand (General Public)
E-voting machines in Ohio in 2004 were full of critical failures: 1200% turnout in certain districts, Bush winning by thousands of votes in precincts where only hundreds were registered, vote tallying servers mysteriously going offline for hours, the list goes on and on. How will the official record on all of the failures of 2004 become a part of this package and set the scope for what needs to be overcome? Why is it not important to investigate, understand, process, and litigate those failures so that a process such as this can be meaningful?
Clarify "includes software modules". Wording should be specific so that it cannot be interpreted that this only applies to software.
Item k. Give the appearance of irregularity: please provide clarification on what would be considered an irregularity.
This requirement needs to be better defined. Almost all components in a system could cause any of these failures. The requirement should be a non-detectable failure that could result in incorrect votes being recorded. If the requirement is too broad then the cost of designing, manufacturiing, and testing will make the products not cost effective for jurisdictions.
2.1-A.7 Testing statements for every part, component, and assembly
The Manual SHALL require that the design and development process of a voting system produce statements for every part, component, and assembly, whether to be manufactured by the manufacturer or obtained elsewhere, that impacts conformity to the VVSG. These statements SHALL define verifiable requirements against which the part, component, or assembly can be tested at the end of its manufacturing process, or upon delivery, as appropriate. The requirements SHALL be defined in such a way that any part, component, or assembly that meets the requirements will provide the functionality and reliability required of it for the voting system to meet the overall functionality and reliability requirements specified in the VVSG.
Assuming here that you're mandating that the manufacturer develops a requirements specification (i.e., hardware requirements specification, software requirements specification, and/or system requirements specification). Requirements specifications are where verifiable requirements are maintained. These verifiable requirements should be forward and backward traceable between the requirements and the design, the design and the implementation, and the implementation and the tests. Note that testing can be performed at the unit, component, configuration item, and/or system levels (as appropriate).
Comment by Alan A. Jorgensen, Ph.D., for the Association for Software Testing Special Interest Group on eVoting (Advocacy Group)
We recommend that this requirement specify that these testing statements be delivered to the test lab prior to the delivery of the system. We also recommend that the requirement define an iterative process wherein the test lab develops lists of tests and testability issues (such as ambiguity, redundancy, tautology) against the testing statements, and delivers these to the manufacturer. The manufacturer must then address those issues and redeliver improved testing statements for more feedback. The manufacturer should also distribute the test lab's nascent test plan to the product development team, in order to inform their own development and testing processes. Ideally the manufacturer would also iteratively deliver feedback on the test plan to the test lab.
If by "every part" this is meant to refer to such items as capacitors, resistors, wires, etc then this requrement is not realistic and not acheivable. Therefore, this requirement needs more clarification as to what its intent is and what scope of parts, components, and assemblies this is to refer to. Please clarify the level of parts, components, and assemblies are referred to in this requirement.
The testing statements shall be a public record. .......... (Affiliation Note: IEEE representative to TGDC)
2.1-A.8 Inspection processes for every part, component, and assembly
The Manual SHALL require that the design and development process define or identify processes by which all parts, components, and assemblies of a voting system can be tested for compliance with requirements developed under Requirement Part 2: 2.1-A.7.
This section should only apply to parts, components, and assemblies defined as critical. Proposed Change: Change the requirement to read as follows: 2.1-A.8 Inspection processes for every part, component, and assembly defined as critical The Manual SHALL require that the design and development process define or identify processes by which all parts, components, and assemblies, defined as critical, of a voting system can be tested for compliance with requirements developed under Requirement Part 2:2.1-A.7.
The descriptions or statements of inspection processes shall be a public record. .......... (Affiliation Note: IEEE representative to TGDC)
2.1-A.9 Testing statements for the entire voting system
The Manual SHALL require that the design and development process of a voting system produce a statement that defines verifiable requirements against which any voting system can be tested at the end of its manufacturing and assembly process in such a way that passing the test provides assurance that the voting system meets all requirements defined in the VVSG.
The testing statements for the entire voting system shall be a public record. .......... (Affiliation Note: IEEE representative to TGDC)
2.1-A.10 Inspection of all purchased parts, components, and assemblies
The Manual SHALL require that all purchased parts, components and assemblies are tested according to the testing requirements developed under Requirement Part 2: 2.1-A.7 and the processes developed under Requirement Part 2: 2.1-A.8 before they are incorporated into a voting system. The records SHALL be maintained until such time as the certification of the voting system model expires or is revoked.
Not only should the manufacturer be required to ensure that any purchased product conforms to the specified requirements but, they should be required (in advance) to evaluate and select their suppliers based on the supplier's ability to supply product(s) in accordance with the manufacturer's requirements. This puts additional responsibility and obligation on the manufacturer to ensure that any supplier they subcontract out to is capable of performing the work.
This section should only apply to parts, components, and assemblies defined as critical. Proposed Change: Change the requirement to read as follows: 2.1-A.10 Inspection of all purchased parts, components, and assemblies defined as critical The Manual SHALL require that all purchased parts, components and assemblies defined as critical are tested according to the testing requirements developed under Requirement Part 2:2.1-A.7 and the processes developed under Requirement Part 2:2.1-A.8 before they are incorporated into a voting system. The records SHALL be maintained until such time as the certification of the voting system model expires or is revoked.
2.1-A.11 Inspection of all manufactured parts, components, and assemblies
The Manual SHALL require that all manufactured parts, components, and assemblies are tested according to the testing requirements developed under Requirement Part 2: 2.1-A.7 and the processes developed under Requirement Part 2: 2.1-A.8 before they are incorporated into a voting system. The records shall be maintained until such time as the certification of the voting system model expires or is revoked.
This section should only apply to parts, components, and assemblies defined as critical. Proposed Change: Change the requirement to read as follows: 2.1-A.11 Inspection of all manufactured parts, components, and assemblies defined as critical The Manual SHALL require that all manufactured parts, components, and assemblies defined as critical are tested according to the testing requirements developed under Requirement Part 2:2.1-A.7 and the processes developed under Requirement Part 2:2.1-A.8 before they are incorporated into a voting system. The records SHALL be maintained until such time as the certification of the voting system model expires or is revoked.
2.1-A.12 Records of all critical parts, components, and assemblies
The Manual SHALL require that for each part, component, or assembly, whether purchased or manufactured by the manufacturer, that has been defined as critical (Requirement Part 2: 2.1-A.6), records SHALL be kept that document the complete history of the part, component, or assembly. The records SHALL include:
The processes used in the manufacture;
The time when critical manufacturing steps were taken;
The organization or person that performed each critical manufacturing step, and
The persons who performed the required inspections.
The records SHALL also include documentation of:
Any failures, discrepancies or anomalies that might have occurred during manufacture;
Any actions taken to correct the failure, discrepancy or anomaly; and
The final determination that the problem has been corrected.
These records shall be available for inspection.
Item c. The time when critical manufacturing steps were taken: need more definition of these critical steps.
This requirement is a very onerous and has almost no benefit. As with other items within this section there needs to be a better understanding of what the intent is so a more cost effective means of achieving the goal can be found. Please clarify the intent of this requirement and the goal it is supporting.
The records of all critical parts, components, and assemblies shall be a public record. .......... (Affiliation Note: IEEE representative to TGDC)
2.1-A.13 Technical capability for monitoring
The Manual SHALL require the manufacturer to identify and maintain the technical capability to monitor the in-service performance of each voting system sold throughout the life cycle of the voting system's model.
For the purpose of this and subsequent requirements in this section, the term life cycle of a voting system model is defined as the time period from the delivery of the first voting system of that model to the time when the certification of the model expires or is revoked.
In all cases, "shall" requirements are verifiable. How will 2.1-A.13 (Technical Capability for Monitoring) be verified? How often will it be verified?
"The Manual Shall require the manufacturer to identify and maintain the technical capability to monitor the in-service performance of each voting system sold throughout the life cycle of the voting system's model." (1) Does the VSTL have any responsibility for checking this requirement other than ensuring the manufacturer has documented it? (2) If so, need guidelines on what is required from the VSTL.
2.1-A.14 Technical capability for developing and implementing remedies
The Manual SHALL require the manufacturer to identify and maintain the technical capability to develop and implement remedies that are suitable to correct any defects that lead to in-service difficulties in all voting systems sold, throughout the life cycle of the voting system model.
"The Manual Shall require the manufacturer to identify and maintain the technical capability to develop and implement remedies...sold throughout the life cycle of the voting system's model." (1) Does the VSTL have any responsibility for checking this requirement? (2) If so, need guidelines on what is required from the VSTL.
2.1-A.15 Financial capability to provide the product support
The Manual SHALL require the manufacturer to identify and maintain the financial capability to provide product support, as defined in Requirements Part 2: 2.1-A.13 and Part 2: 2.1-A.14, throughout the life cycle of the voting system model.
How can an accrediting agency enforce a private business to remain in business throughout the life of its products? What is the pass/fail criteria for this requirement?