Elections: Federal Programs for Accrediting Laboratories That Test Voting Systems Need to Be Better Defined and Implemented
Released: 2008-09-09T00:00:00

Letter

Background

All levels of government share responsibility in the overall U.S. election system. At the federal level, Congress has authority under the Constitution to regulate presidential and congressional elections and to enforce prohibitions against specific discriminatory practices in all federal, state, and local elections. Congress has passed legislation that addresses voter registration, absentee voting, accessibility provisions for the elderly and persons with disabilities, and prohibitions against discriminatory practices. At the state level, individual states are responsible for the administration of both federal elections and their own elections. States regulate the election process, including, for example, the adoption of voluntary voting system guidelines, the state certification and acceptance testing of voting systems, ballot access, registration procedures, absentee voting requirements, the establishment of voting places, the provision of election day workers, and the counting and certification of the vote.

In total, the overall U.S. election system can be seen as an assemblage of 55 distinct election systems—those of the 50 states, 4 U.S. territories, and the District of Columbia. Further, although election policy and procedures are legislated primarily at the state level, states typically have decentralized election systems, so that the details of administering elections are carried out at the city or county levels, and voting is done at the local level. As we reported in 2001, local election jurisdictions number more than 10,000, and their sizes vary enormously—from a rural county with about 200 voters to a large urban county, such as Los Angeles County, where the total number of registered voters for the 2000 elections exceeded the registered voter totals in 41 states. Further, these thousands of jurisdictions rely on many different types of voting methods that employ a wide range of voting system makes, models, and versions. Because of the prominent role played by electronic voting systems, testing these systems against national standards is critical to ensuring their security and reliability. Equally critical is ensuring that the laboratories that perform these tests are competent to carry out testing activities.

The Overall U.S. Election System Depends on Effective Interactions among People, Processes, and Technology

In the United States today, most votes are cast and counted by electronic voting systems, and many states require use of systems that have been certified nationally or by state authorities. However, voting systems are but one facet of a multifaceted, continuous overall election system that involves the interplay of people, processes, and technology during the entire life of a system. All levels of government, as well as commercial voting system manufacturers and system testing laboratories, play key roles in ensuring that voting systems perform as intended.

Electronic voting systems are typically developed by manufacturers, then purchased as commercial, off-the-shelf products and operated by state and local election administrators. Viewed at a high level, these activities make up three phases of a system life cycle: product development, acquisition, and operations. (See fig. 1.) Key processes that span these life cycle phases include managing the people, processes, and technologies within each phase and across phases, and testing the systems and components during and at the end of each phase. Additionally, voting system standards are important through all of the phases because they provide criteria for developing, testing, and acquiring voting systems, and they specify the necessary documentation for operating the systems.

The product development phase includes activities such as establishing requirements for the system, designing a system architecture, developing software, and integrating components. Activities in this phase are performed by the system vendor.

The acquisition phase includes activities such as publishing a solicitation, evaluating offers, choosing a voting technology and a vendor, and awarding and administering contracts. For voting systems, activities in this phase are primarily the responsibility of state and local governments but entail some responsibilities that are shared with the system vendor (e.g., entering into the contract).

The operations phase consists of activities such as ballot design and programming, setup of systems before voting, pre-election testing, vote capture and counting during elections, recounts and system audits after elections, and storage of systems between elections. Responsibility for activities in this phase typically resides with local jurisdictions, whose officials may, in turn, rely on or obtain assistance from system vendors for aspects of these activities.

Standards for voting systems, as will be discussed in a later section, were developed at the national level by the Federal Election Commission in 1990 and 2002 and were updated by EAC in 2005. In the product development phase, voting system standards serve as requirements to meet for developers to build systems. In the acquisition phase, they also provide a framework that state and local governments can use to evaluate systems. In the operations phase, they specify the necessary documentation for operating the systems.

Testing processes are conducted throughout the life cycle of a voting system. Voting system vendors conduct product testing during development of the system and its components. Federal certification testing of products submitted by system vendors is conducted by national voting system testing laboratories (VSTL). States may conduct evaluation testing before acquiring a system to determine how well products meet their state-specific specifications, or they may conduct certification testing to ensure that a system performs its functions as specified by state laws and requirements. Once a voting system is delivered by the system vendor, states and local jurisdictions may conduct acceptance testing to ensure that the system satisfies functional requirements. Finally, local jurisdictions typically conduct logic and accuracy tests related to each election and sometimes subject portions of the system to parallel testing during each election to ensure that the system components perform accurately.

Management processes ensure that each life cycle phase produces a desirable outcome. Typical management activities that span the system life cycle include planning, configuration management, system performance review and evaluation, problem tracking and correction, human capital management, and user training. These activities are conducted by the responsible parties in each life cycle phase.

In 2004, we reported that the performance of electronic voting systems, like any type of automated information system, can be judged on several bases, including their security, accuracy, ease of use, efficiency, and cost. We also reported that voting system performance depends on how the system was designed, developed, and implemented.

Laboratory Accreditation Plays an Important Role in Ensuring Accurate, Reliable, and Secure Voting Systems

Since the passage of HAVA, the use of electronic voting systems has increased and become the predominant method of voting. However, concerns have been raised about the security and reliability of these systems. As we have previously reported, testing and certifying voting systems is one critical step in acquiring, deploying, operating, and administering voting systems, which better ensures that they perform securely and reliably. Among other things, rigorous execution and careful documentation of system testing is a proven way to help ensure that system problems are found before the systems are deployed and used in an election. To accomplish this, it is vital that the organizations that test the systems be qualified and competent to do so. For voting systems, a key testing organization is a federally accredited, national VSTL.

In general, accreditation is the formal recognition that a laboratory is competent to carry out specific types of tests or calibrations. Federally accredited laboratories perform many different types of testing and related activities on various products, ranging from inspecting grain to certifying maritime cargo gear. The genesis of laboratory accreditation programs owes largely to agencies’ need to assure themselves of the competency of the organizations responsible for testing products or services that involve the use of federal funds.

To provide national recognition for competent laboratories, the NIST Director established the National Voluntary Laboratory Accreditation Program (NVLAP) in 1976 at the request of the private sector. Under this program, which is based on internationally accepted standards, NIST accredits laboratories that it finds competent to perform specific types of tests or calibrations. In June 2004, NVLAP announced the establishment, in accordance with HAVA, of an accreditation program for laboratories that test voting systems using standards determined by EAC.

HAVA Assigned EAC and NIST Responsibility for Accrediting VSTLs

Enacted in October 2002, HAVA affected nearly every aspect of the voting process, from voting technology to provisional ballots and from voter registration to poll worker training. In particular, the act authorized $3.86 billion in funding over several fiscal years to replace punch card and mechanical lever voting equipment, improve election administration and accessibility, train poll workers, and perform research and pilot studies.

HAVA also established EAC, provided for the appointment of four commissioners, and specified the process for selecting an executive director. Generally speaking, EAC is to assist in the administration of federal elections and provide assistance in administering certain federal election laws and programs.

Since the passage of HAVA in 2002, the federal government has taken steps to implement the act’s provisions. For example, after beginning operations in January 2004, EAC updated the existing federal voluntary standards for voting systems, including strengthening provisions related to security and reliability. Additionally, EAC established an interim VSTL accreditation program that leveraged a predecessor program run by the National Association of State Elections Directors, and EAC and NIST then established companion accreditation programs that replaced the interim program.

EAC Updated the Federal Voluntary Standards for Voting Systems

Federal standards for voting systems were first issued in 1990 when the Federal Election Commission published standards. These federal standards identified minimum functional and performance requirements, which states were free to adopt in whole, in part, or not at all, for electronic voting equipment, and specified test procedures to ensure that the equipment met those requirements. In 2002, the Federal Election Commission issued its Voting System Standards (VSS), which updated the 1990 standards to reflect more modern voting system technologies. In 2005, we reported that these standards identified minimum functional and performance requirements for voting systems but were not sufficient to ensure secure and reliable voting systems. As a result, we recommended that EAC work to define specific tasks, measurable outcomes, milestones, and resource needs to improve the voting system standards. Until then, election administrators were at risk of relying on voting systems that were not developed, acquired, tested, operated, or managed in accordance with rigorous security and reliability standards— potentially affecting the reliability of future elections and voter confidence in the accuracy of the vote count.

Following the enactment of HAVA in 2002 and the establishment of EAC in 2004, EAC adopted the Voluntary Voting System Guidelines (VVSG) in 2005. The VVSG specify the functional requirements, performance characteristics, documentation requirements, and test evaluation criteria for the national certification of voting systems. Accredited testing laboratories are to use the VVSG to develop test plans and procedures for the analysis and testing of systems in support of EAC’s voting system certification program. The VVSG are also used by voting system manufacturers as the basis for designing and deploying systems that can be federally certified.

EAC Established an Interim Accreditation Program

We reported in 2001 that the National Association of State Elections Directors was accrediting independent test authorities to test voting equipment against the Federal Election Commission standards. Under this program, three laboratories were accredited. Under HAVA, NIST is to recommend laboratories for EAC accreditation. In 2006, NIST notified EAC that its initial recommendations might not be available until sometime in 2007. As a result, EAC initiated an interim accreditation program and invited the three laboratories accredited by the state elections directors to apply. As part of the interim program, laboratories were required to attest to a set of EAC-required conditions and practices, including certifying the integrity of personnel, the absence of conflicts of interest, and the financial stability of the laboratory. In August and September 2006, EAC granted interim accreditation to two of the three laboratories invited to apply. EAC terminated its interim program in March 2007.

EAC and NIST Have Established Separate but Related Laboratory Accreditation Programs

HAVA assigned responsibilities for laboratory accreditation to both EAC and NIST. In general, to reach an accreditation decision, NIST is to focus on assessing laboratory technical qualifications, while EAC is to use those assessment results and recommendations and augment them with its own review of related laboratory capabilities. See table 1 for the two agencies’ HAVA responsibilities.

The tasks that NIST is to perform addressed in an annual interagency agreement executed between the institute and EAC each year. For example, the 2008 interagency agreem states that NVLAP will continue to assess VSTLs and will coordinate with EAC to continually monitor and review the performance of the laboratories. Additionally, the agreement states that the two agencies will coordinate to maintain continuity between their respective accreditation programs. in meeting HAVA’s requirements are The NIST and EAC accreditation programs can be viewed together as forming a federal VSTL accreditation process that consists of a series o complementary steps. These steps are depicted in figure 2, where the numbers correspond to a detailed narrative description below.

Status of Completed VSTL Accreditation Activities

As of May 2008, EAC has accredited four laboratories. These laboratories are SysTest Labs, LLC; Wyle Laboratories, Inc.; iBeta Quality Assurance; and InfoGard Laboratories, Inc. A fifth laboratory, CIBER Inc., has been granted NVLAP accreditation and has been recommended to, but not yet accredited by, EAC. InfoGard Laboratories, Inc., whose NVLAP accreditation expires in June 2008, has recently notified NIST and EAC that it would not apply to renew its accreditation, citing the volatility of the voting system environment as one reason. The timeline for each of these accreditations, and other accreditation program activities, is found in figure 3.

NIST Has Defined and Implemented an Accreditation Approach That Reflects Relevant Standards but Is Missing Details Needed for Consistent and Verifiable Implementation

NIST’s defined approach to accrediting voting system laboratories largely reflects applicable HAVA requirements and relevant international standards, both of which are necessary to an effective program. However, this approach is continuing to evolve based on issues realized during NIST’s implementation experience to date. In particular, because NIST’s defined program does not, for example, specify the nature and extent of assessment documentation to generate or retain or specify the version of the voting system standards to be used, our analysis of NIST’s efforts in accrediting four laboratories could not confirm that the agency has consistently followed its defined accreditation program. NIST officials stated that these limitations are due in part to the relative newness of the program and that they will be addressed by updating the accreditation program handbook. However, they said that they do not have documented plans to accomplish this. Until these limitations are addressed, NIST will be challenged in accrediting voting system laboratories in a consistent and verifiable manner.

NIST Voting System Accreditation Program Reflects HAVA Requirements

NIST has defined its voting system accreditation program to address relevant HAVA requirements. According to HAVA, NIST is to conduct reviews of independent, nonfederal voting system testing laboratories and submit to EAC a list of proposed voting system testing laboratories and monitor and review the performance of those proposed laboratories that EAC accredits, including making recommendations to EAC regarding accreditation continuance and revocation.

NIST’s defined voting system accreditation program satisfies both of these requirements. With respect to the first, NIST announced in June 2004 the establishment of its voting system testing laboratory accreditation program as part of NVLAP, a statutorily created program for unbiased, third parties to establish the competence of national independent laboratories. As such, NIST adopted its NVLAP handbook as the basis for its defined approach to reviewing VSTLs and has supplemented it with a handbook that is specific to voting system testing.

With respect to the second HAVA requirement, the supplemental handbook cited above states that the NIST Director will recommend NVLAP-accredited VSTLs to EAC for subsequent commission accreditation. Additionally, NIST’s handbooks provide for both monitoring accredited laboratories and for making recommendations regarding a laboratory’s continued accreditation. For example, the handbook states that a monitoring visit may occur at both scheduled and unscheduled times and the scope may be limited to a few items or include a full review. It also states that a reaccreditation review shall be conducted in accordance with the procedures used to initially accredit laboratories. Further, the handbook also identifies accreditation or reaccreditation decision options, including granting, denying, or modifying the scope of an accreditation.

According to NIST officials, these HAVA requirements are relevant and important to defining an effective voting system testing laboratory accreditation program. By incorporating them, NIST has reflected one key aspect of an effectively defined program.

NIST Has Incorporated Relevant International Accreditation Standards into Its VSTL Accreditation Program

NIST’s VSTL accreditation program reflects internationally recognized standards for establishing and conducting accreditation activities. These standards are published by the International Organization for Standardization (ISO), and the two that are germane to this accreditation program are (1) ISO/IEC 17011, which establishes general requirements for accreditation bodies and (2) ISO/IEC 17025, which establishes the general requirements for reviewing the competence of laboratories. According to NIST program documentation, this allows NVLAP to both operate as an unbiased, third party accreditation body and to utilize a quality management system compliant with international standards. As a result, NIST has incorporated key aspects of an effective accreditation body into its voting system accreditation program.

NIST Program Meets ISO Accreditation Body Requirements

ISO/IEC 17011 requires that an accrediting body have, among other things, (1) a management system for accreditation activities, (2) a policy defining the types of records to be retained and how those records will be maintained, (3) a clear description of the accreditation process that covers the rights and responsibilities of those seeking accreditation, and (4) a clear description of the accreditation activities to be performed.

NIST VSTL accreditation program-related documentation, including its program handbooks, satisfies each of these requirements. In fact, NIST has cross-referenced its documentation to each ISO/IEC 17011 requirement. Specifically, the first requirement is cross-referenced to the NVLAP Management System Manual, which describes the overall accreditation program’s management policies and control structure, and the second is cross-referenced to the program’s record keeping policy, which specifies what types of records should be maintained and how they should be maintained. The third and fourth requirements are cross-referenced to the accreditation process descriptions in both the Management System Manual and the general handbook. Together, these documents contain, for example, (1) the rights of laboratories applying for accreditation and (2) the scope of accreditation activities to be performed, including a preassessment review, an on-site review, and a final on-site assessment report.

NIST Program Meets ISO Laboratory Accreditation Review Requirements

ISO/IEC 17025 requires that accreditation reviews cover specific topics. These include (1) laboratory personnel independence and conflicts of interest; (2) a laboratory system for quality control (i.e., a framework for producing reliable results and continuous improvement to laboratory procedures); and (3) a laboratory mechanism for collecting and responding to customer complaints. Additionally, the standard establishes basic technical requirements that a laboratory has to meet, and thus that reviews are to cover, including (1) competent laboratory personnel who are capable of executing the planned tests, (2) appropriate tests and test methods, and (3) clear and accurate test result documentation.

NIST voting system testing laboratory accreditation program-related documents, including its program handbooks, satisfy these requirements. First, the general handbook defines the requirement for a laboratory to have personnel that are independent and free of any conflict of interest. Second, the handbook requires that a laboratory have a management quality control system and that this system provide for reliable results and continuous improvement to laboratory procedures. Third, the handbook requires that a laboratory have a mechanism for receiving and responding to customer complaints. Last, the handbook establishes certain technical requirements that a laboratory must meet, such as having competent laboratory personnel capable of executing the planned tests, using appropriate tests and test methods, and documenting test results in a clear and accurate manner.

For several of these requirements, NIST’s voting-specific supplemental handbook augments the general handbook. For example, this supplemental handbook requires laboratories to submit a quality control manual, as well as information to demonstrate the competence of laboratory administrative and technical staff. Further, it requires that a laboratory’s training program be updated so that staff can be retrained as new versions of voting system standards are issued.

NIST Voting System Accreditation Program Does Not Reflect Its Own Findings on the Need for Assessor Qualifications and Training and Key Assessment Criteria

NIST has reported on the importance of ensuring that those persons who perform accreditation assessments are sufficiently qualified and that the assessments themselves are based on explicitly defined criteria and are adequately documented. Nevertheless, NIST has not fully reflected key aspects of these findings in its defined approach to accrediting voting system testing laboratories. For example, it has not specified the basis for determining the qualifications of its accreditation assessors, and while a draft update to its handbook now includes the specific voting system standards to be used when performing an accreditation assessment, this handbook was only recently approved. According to NIST officials, these gaps are due to the newness of the accreditation program and will be addressed in the near future. Because these gaps have confused laboratories as to what standards they were to meet, and may have resulted in differences in how accreditations have been performed to date, it is important that the gaps be addressed.

NIST Has Not Specified Requirements for Assessor Qualifications and Training

NIST has reported on the importance of having competent and qualified human resources to support accreditation programs. According to these findings, an accreditation program should, among other things, provide for having experienced and qualified assessors to perform accreditation demonstrating an assessors’ qualifications using defined documentation and explicit criteria that encompass the person’s education, experience, and training; and training (initial and continuing) for assessors.

NIST’s defined approach to VSTL accreditation does not provide for all these requirements. To its credit, its program handbook identifies the need for experienced and qualified assessors in the execution of accreditation activities and provides for each assessor’s qualifications to be documented. Further, it has defined generic training that applies to all of its accreditation assessors. For example, the NVLAP Assessor Training Syllabus includes training on ISO/IEC 17011 and 17025, as well as training on the NVLAP general handbook. In addition, the VSTL accreditation program manager stated that new assessors receive training on the 2002 VSS and 2005 VVSG and that periodic training seminars are provided to assessors on changes to either the general handbook or the 2005 VVSG.

In addition, the program manager told us that candidate assessors must submit some form of documentation (e.g., a resume), and that this documentation is used to evaluate, rank, and select candidates that are best qualified. The NIST VSTL assessors that we interviewed confirmed that they were required to submit such documentation at NIST’s request.

However, NIST’s defined approach does not cite the explicit capabilities and qualifications that an assessor must meet or the associated documentation needed to demonstrate these capabilities and qualifications. According to the program manager, this is because the field of potential assessors in the voting system arena is small and specialized and because they focused on defining other aspects of the program that were higher priorities. Further, NIST has not defined and documented the specific training requirements needed to be a VSTL lead assessor or a technical assessor for the VSTL program. According to the program manager, this is because these assessors receive all the training they need by working on the job with more experienced assessors. Not specifying criteria governing assessor qualifications and training is of concern because differences in assessors’ capabilities could cause inconsistencies in how assessments are performed.

NIST’s Approach Does Not Fully Specify Criteria for Evaluating and Documenting VSTL Capabilities

NIST recognizes the importance of specifying explicit criteria against which all candidate laboratories will be assessed and fully documenting the assessments that are performed. Specifically, the general handbook provides the criteria and requirements that will be used to evaluate basic laboratory capabilities. It also states that technical requirements specific to a given field of accreditation are published in program-specific handbooks. To that end, NIST published a supplemental program-specific handbook in December 2005 that provided the voting-specific requirements to be used to evaluate VSTLs, additional guidance, and related interpretive information.

NIST’s 2005 supplemental handbook does not contain sufficient criteria against which to evaluate VSTLs. It identifies specific requirements that laboratories are to demonstrate relative to the 2002 VSS but not the 2005 VVSG. For example, the handbook states that laboratories are expected to develop, validate, and document test methods that meet the 2002 VSS. However, it does not refer to the 2005 VVSG. In addition, the program- specific checklist that accompanies this version of the handbook does not identify all the 2005 VVSG standards against which laboratories are evaluated. Specifically, this checklist makes reference to the VVSG in relation to just a few checklist requirements.

According to the NIST program manager, the 2005 handbook did not refer to the 2005 VVSG requirements because only the 2002 VSS requirements were mandatory at the time it was published. He further stated that, despite the fact that the 2005 VVSG requirements were not included in that handbook, NIST assessors were expected to use them when performing the first laboratory assessments. Representatives for two laboratories stated that because these requirements were not documented or identified in the NIST handbooks, they did not learn that they would be required to demonstrate 2005 VVSG-based capabilities until the NIST on-site assessment teams arrived.

In December 2007, NIST released draft revisions of the voting program- specific handbook and checklist, stating that labs are expected to meet both 2002 VSS and 2005 VVSG. In addition, the 2007 draft handbook clearly specifies that laboratories must demonstrate how developed test methods and planned tests trace back to and satisfy both the 2002 VSS and the 2005 VVSG. Taken together, the new handbook and checklist should better identify the requirements and criteria used to evaluate a laboratory and document the results. According to NIST, the new handbook and checklist have recently been finalized, and both are now in use.

Available Documentation Does Not Show That NIST Has Consistently Followed All Aspects of Its Defined Accreditation Approach

NIST has found that reliable and accurate documentation provides assurance that laboratory accreditation activities have been effectively fulfilled. However, in its efforts to date in accrediting four VSTLs, documentation of the assessments does not show that NIST has fully followed its defined accreditation approach. While we could not determine whether this is due to incomplete documentation of the steps performed and the decisions made during an assessment or due to steps not being performed as defined, this absence of verifiable evidence raises questions about the consistency of the assessments and the resultant accreditations. Without adequately documenting each assessment, including all steps performed and the basis for any steps not performed, such questions may continue to be raised.

To NIST’s credit, available documentation shows that it consistently followed some aspects of its defined approach in accrediting the four laboratories. For example, we verified that NIST received an application from each of the laboratories as required, and our review of completed checklists and summary reports shows that preassessment reviews and on-site assessments were performed for each laboratory, as was required. According to a lead assessor, this review usually focused on the laboratories’ quality assurance manuals. Moreover, the completed checklists identified whether the requirement was met or not for each listed requirement, and included comments, in some cases, as to how a laboratory addressed a requirement. Also as required, NIST received laboratory responses describing how unmet requirements were addressed within specified time frames, used the responses in making accreditation decisions, and notified EAC of its decisions via letters of recommendation. Furthermore, NIST has recently begun reaccreditation reviews at two laboratories, as required.

However, documentation does not show that NIST has consistently followed other aspects of its defined approach. Our analysis of the checklists that are to be used to both guide and document a given assessment, including identifying unmet requirements and capturing assessor comments and observations, shows some differences. For example: One type of checklist (the supplemental handbook checklist) was prepared for only two of the four laboratory assessments. According to the program manager, this is because even though a draft revision of this checklist was actually used to assess the other two laboratories, the assessment results were recorded on a different checklist (the general handbook checklist). While this is indicated on one of the two checklists, it is not indicated on the other.

On the checklist used for one laboratory, an assessor marked several sections as “TA” with no explanation as to what this means. Also, the checklist used for another laboratory did not identify whether most of the requirements were met or not met. Further, the checklist for a third laboratory had one section marked as “not applicable” but included no explanation as to why that section did not apply, while the checklist for a different laboratory marked the same section as “not applicable” but included a reason for doing so.

Notwithstanding these differences, the program manager told us that each laboratory was assessed using the same requirements and all assessments to date were performed in a consistent manner. On the basis of available documentation, however, we could not verify that this is the case. As a result, it is not clear that NIST has consistently followed its defined approach.

Available documentation also does not show that NIST followed other aspects of its approach. For example: The program handbook states that each laboratory is to identify the requested scope of accreditation in its application package. However, our analysis of the four application packages shows that two laboratories did not specify a requested scope of accreditation. According to the program manager, the scope of accreditation for all laboratories was the 2002 VSS and 2005 VVSG because, even though the latter standards were not yet in effect at the time, they were anticipated to be in effect in the near future. However, NIST did not have documentation that notified the laboratories of this scope of accreditation or that indicated whether this scope was established by EAC, NIST, or the laboratories.

The program handbook states that after receiving a laboratory’s application package, NIST will acknowledge its receipt in writing and will inform the laboratory of the next steps in the accreditation process. However, NIST did not have documentation demonstrating that this was done. According to the program manager, this was handled via telephone conversations. However, representatives for several laboratories noted that these calls did not clearly establish expectations, adding that some expectations were not communicated until the NIST team assessors arrived to conduct the on-site assessment.

The program manager stated that these deviations from the defined approach are attributable to the relative newness of the program, but despite these discrepancies, each laboratory was assessed consistently. However, we could not verify this, and thus it is not clear that NIST has consistently followed its defined approach. According to this official, future versions of the program handbook would address these limitations. However, documented plans for doing so have not been developed.

EAC’s Recently Drafted Accreditation Approach and Its Earlier Performed Laboratory Accreditations Lack Key Effectiveness Factors and Features

EAC has recently defined its voting system laboratory accreditation approach in a draft program manual. However, this draft manual omits important content. While addressing relevant HAVA requirements, the draft manual does not adequately define key accreditation factors that NIST has identified, and a key accreditation feature that we have previously reported as being integral to an effective accreditation program. Moreover, not all factors and features that the draft manual does include have been defined to a level that would ensure thorough, consistent, and verifiable implementation. Because this manual was not available for EAC to use on the four laboratory accreditations that it has completed, the accreditations were performed using a largely undocumented series of steps. As a result, the thoroughness and consistency of these accreditations is not clear. According to EAC officials, these gaps are due to the agency’s limited resources being focused on other issues, and will be addressed as its accreditation program evolves. However, they said that they do not yet have documented plans to accomplish this. Until EAC fully defines a repeatable VSTL accreditation approach, it will be challenged in its ability to treat all laboratories consistently and produce verifiable results.

EAC Has Defined a Draft Accreditation Approach that Meets HAVA Requirements

In February 2008, EAC issued a draft version of a VSTL accreditation program manual for public comment. According to HAVA, EAC’s accreditation program is to meet certain requirements. Specifically, it is to provide for voting system hardware and software testing, certification, decertification, and recertification by accredited laboratories. Additionally, it is to base laboratory accreditation decisions, including decisions to revoke an accreditation, on a vote of the commissioners, and it is to provide for a published explanation of any commission decision to accredit any laboratory that was not first recommended for accreditation by NIST.

To EAC’s credit, its draft accreditation program manual addresses each of these requirements. First, the manual defines the role that the laboratories are to play relative to voting system testing, certification, recertification and decertification, and it incorporates by reference an EAC companion voting system certification manual that defines requirements and process steps for voting system testing and certification-related activities.

With respect to the remaining three HAVA requirements, the draft EAC accreditation manual also requires (1) that the commissioners vote on the accreditation of laboratories recommended by NIST for accreditation, (2) that EAC publish an explanation for the accreditation of any laboratory not recommended by NIST for accreditation, and (3) that the commissioners vote on the proposed revocation of a laboratory’s accreditation.

According to EAC officials, its draft approach incorporates HAVA requirements because the commission is focused on meeting its legal obligations in all aspects of its operations, including VSTL accreditation. In doing so, EAC has addressed one important aspect of having an effective accreditation program.

EAC Draft Approach Does Not Adequately Define Key Accreditation-Related Steps and Decision Criteria

Beyond addressing relevant HAVA requirements, EAC’s draft accreditation manual defines an accreditation process, including program phases, requirements, and certain evaluation criteria. However, it does not do so in a manner that fully satisfies factors that NIST has reported can affect the effectiveness of accreditation programs. Moreover, it does not adequately address a set of features that our research shows are common to federal accreditation programs and that can influence a program’s effectiveness. According to EAC officials, these factors and features are not fully addressed in the draft program manual because its accreditation program is still in its early stages of development and is still evolving. Until they are fully addressed, EAC’s accreditation program’s effectiveness will be limited.

Key NIST Accreditation Factors Not Fully Addressed

According to NIST, having confidence in and ensuring appropriate use of an accredited testing laboratory requires that accreditation stakeholders have an adequate understanding of the accreditation process, scope, and related criteria. NIST further reports that confidence in the accreditation process can be traced to a number of factors that will influence the thoroughness and competence of accreditation programs, and thus these factors can be viewed as essential accreditation program characteristics. They include having published procedures governing how the accreditation program is to be executed, such as procedures for granting, maintaining, modifying, suspending, and withdrawing accreditation; specific instructions, steps, and criteria for those who conduct an accreditation assessment (assessors) to follow, such as a test methodology that is acceptable to the accreditation program; knowledgeable and experienced assessors to execute the instructions and steps and apply the related criteria; and complete records on the data collected, results found, and reports prepared relative to each assessment performed.

EAC’s draft accreditation program manual addresses one of these factors but it does not fully address the other three. (See table 2.) For example, while the manual requires that EAC maintain records, it only addresses the retention of records associated with the testing of voting systems and not those associated with the accreditation of laboratories. EAC officials told us that testing records are meant to include accreditation records, although they added that this is not explicit in the manual and needs to be clarified. Further, the manual is silent on the steps to be followed and criteria to be applied in reviewing a laboratory’s application and the qualifications required for accreditation reviewers. By not fully addressing these factors, EAC increases the risk that its accreditation reviews will not be performed consistently and comprehensively.

As we have previously reported, the nature and focus of federal programs for accrediting laboratories vary, but nevertheless include certain common features. In particular, these programs require laboratories to provide certain information to the accrediting body, and they provide for evaluation of this information by the accrediting body in making an accreditation determination. As we reported, the required information is to include, among other things, the laboratory’s (1) organizational information, (2) records and record-keeping policy, (3) test methods and procedures, (4) conflict of interest policy, and (5) financial stability.

To its credit, EAC’s draft accreditation manual provides for laboratories to submit information relative to each of these features that are common to federal accreditation programs. For example, it provides for laboratories to submit organizational information, such as location(s), ownership, and organizational chart; a written policy for maintaining accreditation-related records for 5 years; conflict of interest policies and procedures; test- related polices and procedures, as well as system-specific test plans; and financial information needed to demonstrate stability. Moreover, for four of the five features, the manual identifies the specific types of information needed for accreditation and how the information is to be evaluated, including the criteria that are to be used in evaluating it. However, for the financial stability feature, the manual does not describe what specific documents are required from the laboratory to satisfy this requirement, nor does the manual indicate how information provided by a laboratory will be evaluated.

At the time of our review, EAC’s Director of Voting System Testing and Certification told us that the draft accreditation manual was to be submitted for approval and that this draft did not address all of the limitations cited above. For example, it would not contain the information needed and the evaluation approach and criteria to be used in making determinations about financial stability because this decision is to be based on what the director referred to as a “reasonableness” test that involves EAC evaluation of the information relative to that provided by other laboratories. Further, while EAC officials said that they plan to evolve their approach to VSTL accreditation and to address these gaps, EAC does not have documented plans for accomplishing this. Without clearly defining information to be used and how it is to be used, EAC increases the risk that financial stability determinations will not be consistently and thoroughly made.

Available Documentation Does Not Demonstrate EAC’s Basis for Accrediting Laboratories to Date

As of May 2008, EAC has accredited four laboratories, but the documentation associated with each of these accreditations is not sufficient to recreate a meaningful understanding of how each evaluation was performed and how decisions were made, and thus, the bases for each accreditation were not clear. Specifically, each of the accreditations occurred before EAC had defined its approach for conducting them. Because of this, EAC performed each one using a broadly defined process outlined in a letter to each laboratory and an associated checklist that only indicated whether certain documents were received. Our analysis of these letters showed that the correspondence sent to each laboratory was all the same, identifying three basic review steps to be performed and citing a list of documents that the laboratories were to provide as part of their applications. However, the letters did not describe in any manner how EAC would review the submitted material, including the criteria to be used.

According to EAC officials, the review steps were not documented. Instead, they were derived by a single reviewer using (1) the applications and accompanying documents submitted by the laboratories, (2) familiarity with the materials used by the state election directors- sponsored accreditation program, and (3) the judgment of each reviewer. Further, while the reviews were supported by a checklist that covered each of the items that was to be included in the laboratory applications and provided space for the reviewer(s) to make notes relative to each of these items, the checklists did not include any guidance or methodology, including criteria, for evaluating the submitted items. Rather, the EAC accreditation program director told us that he was the reviewer on all the accreditations and he applied his own, but undocumented, tests for reasonableness in deciding on the submissions’ adequacy and acceptability.

Our analysis of the checklists for each laboratory accreditation showed that while the same checklist was used for each laboratory, the checklists did not provide a basis for evaluating and documenting the basis for the sufficiency of those documents. In some cases, additional communications occurred between the reviewer and the laboratory to obtain additional documents. However, no documentation was available to demonstrate what standards or other criteria the laboratories were held to or how their submissions were otherwise reviewed. For example, each of the checklists indicated that each laboratory provided “a copy of the laboratory’s conflict of interest policy.” However, they did not specify, for example, whether the policy adequately addressed particular requirements. Nevertheless, for three of the four accredited laboratories, documentation shows that EAC sought clarification on or modification to the policies provided, thus suggesting that some form of review was performed against more detailed requirements. Similarly, while the checklists indicate that the laboratories disclosed their respective coverage limits for general liability insurance policies, and in one case EAC communicated to the laboratory that the limits appeared to be low, no documentation specifies the expected coverage limits. According to the EAC Director of Voting System Testing and Certification, this determination was made after comparing limits among the laboratories and was not based on any predetermined threshold. Further, while the checklists indicate that each laboratory provided audited financial statements, there is no documentation indicating how these statements were reviewed.

According to the EAC program director, the lack of documentation demonstrating the basis for EAC’s laboratory accreditations is due to the need at the time to move quickly in accrediting the laboratories and the fact that use of the same individual to review the accreditation evaluation negated the need for greater documentation. Without such documentation, however, we could not fully establish how the accreditations were performed, including whether there was an adequate basis for the accreditation decisions reached and whether they were performed consistently.

Conclusions

The effectiveness of our nation’s overall election system depends on many interrelated and interdependent variables, including the security and reliability of voting systems. Both NIST and EAC play critical roles in ensuring that the laboratories that test these two variables have the capability, experience, and competence necessary to test a voting system against the relevant standards. NIST has recently established an accreditation program that largely accomplishes this, and while EAC is not as far along, it has a foundation upon which it can build.

However, important elements are still missing from both programs. Specifically, the current NIST approach does not define requirements for assessor qualifications and training or ensure that assessments are fully documented. Additionally, EAC has not developed program management practices that are fully consistent with what NIST has found to be hallmarks of an effective accreditation program, nor has the agency adequately specified how evaluations are to be performed and documented. As a result, opportunities exist for NIST and EAC to further define and implement their respective programs in ways that promote greater consistency, repeatability, and transparency—and thus improve the results achieved. It is also important for NIST and EAC to follow through on their stated intentions to evolve their respective programs, building on what they have already accomplished through the development and execution of well-defined plans of action. If they do not, both will be challenged in their ability to consistently provide the American people with adequate assurance that accredited laboratories are qualified to test the voting systems that will eventually be used in U.S. elections.

Recommendations for Executive Action

To help NIST in evolving its VSTL accreditation program, we recommend that the Director of NIST ensure that the accreditation program manager develops and executes plans that specify tasks, milestones, resources, and performance measures that provide for the following two actions: Establish and implement transparent requirements for the technical qualifications and training of accreditation assessors.

Ensure that each laboratory accreditation review is fully and consistently documented in accordance with NIST program requirements.

To help EAC in evolving its VSTL accreditation program, we recommend that the Chair of the EAC ensure that the EAC Executive Director develops and executes plans that specify tasks, milestones, resources, and performance measures that provide for the following action: Establish and implement practices for the VSTL accreditation program consistent with accreditation program management guidance published by NIST and GAO, including documentation of specific accreditation steps and criteria to guide assessors in conducting each laboratory review; transparent requirements for the qualifications of accreditation reviewers; requirements for the adequate maintenance of records related to the VSTL accreditation program; and requirements for determining laboratory financial stability.

Agency Comments and Our Evaluation

Both NIST and EAC provided written comments on a draft of this report, signed by the Deputy Director of NIST and the Executive Director of EAC, respectively. These comments are described below along with our response to them.

In its comments, NIST stated that it appreciates our careful review of its VSTL program and generally concurs with our conclusions that its program must continue to evolve and improve. However, NIST also provided comments to clarify the current status of the program relative to three of our findings.

With respect to our finding that NIST’s defined approach for accrediting VSTLs does not cite explicit qualifications for the persons who conduct the technical assessments, the institute stated that it does explicitly cite assessor qualifications for its overall national laboratory accreditation program, adding that this approach to specifying assessor qualifications has a proven record of success. It also stated that the overall program’s management manual requires all assessors to meet defined criteria in such areas as laboratory experience, assessment skills, and technical knowledge, and that candidate assessors must submit information addressing each of these areas as well as factors addressing technical competence in a given laboratory’s focus area (e.g., voting systems). Further, it stated that candidate assessors’ qualification ratings and rankings are captured in work sheets.

In response, we do not disagree with any of these statements. However, our finding is that NIST’s defined approach for VSTL accreditation does not specify requirements for persons who assess those laboratories that specifically test voting systems. In this regard, NIST’s own written comments confirm this, stating that specific requirements for assessors are not separately documented for each of its national laboratory accreditation programs, such as the VSTL program. Therefore, we have not modified this finding or the related recommendation.

Regarding our finding that NIST’s defined approach for accrediting VSTLs has not always cited the current voting system standards, the institute affirmed this in its comments by stating that the VSTL program handbook that it provided to us only cites the 2002 system standards, as these were the only standards in place when the handbook was published. However, NIST also noted that when the 2005 system guidelines were adopted in December 2005, it began the process of updating the handbook and associated assessment checklist, and that the handbook update was recently finalized for publication and is now being used.

In response, we stand by our finding that NIST’s defined approach has not always cited the current voting system standards, which NIST acknowledges in its comments. However, we also recognize that NIST has recently addressed this inconsistency by finalizing its new handbook and the associated assessment checklist. In light of NIST’s recent actions, we have updated the report to acknowledge the finalization of the handbook and checklist, and removed the associated recommendation that was contained in our draft report for NIST to ensure that its defined approach addresses all required voting system standards.

Regarding our finding that available documentation from completed accreditations does not show that NIST has consistently followed all aspects of its defined approach, the institute stated that, among other things, all required documents for its VSTL accreditation program are currently in use and reflect the recent update to its handbook and checklist, and that all these documents are securely maintained. In response, we do not question these statements; however, they are not pertinent to our finding. Specifically, our finding is that the four completed accreditations that we reviewed were not consistently documented. As we state in our report, we reviewed the documentation associated with the accreditation assessments for these four laboratories, and we found that all four were not documented in a similar manner, even though they were based on the same version of the program handbook. For example, neither the laboratory notifications of the scope of the assessment nor the next steps in the accreditation process were consistently documented. Therefore, we have not modified our finding, but have slightly modified our recommendation to make it clear that its intent is to ensure that all phases of the accreditation review are fully and consistently documented.

In its comments, EAC described our review and report as being helpful to the commission as it works to fully develop and implement its VSTL program. It also stated that it agrees with the report’s conclusions that additional written internal procedures, standards, and documentation are needed to ensure more consistent and repeatable implementation of the program. The commission added that it generally accepts our recommendations and will work hard to implement them. To assist it in doing so, it sought clarification about two of our recommendations, as discussed below.

EAC stated that the recommendation in our draft report for the commission to develop specific accreditation steps and criteria was broadly worded, and thus the recommendation’s intent was not clear. EAC also stated that it interpreted the recommendation to mean that it should define internal instructions to guide assessors in performing an accreditation, and that the recommendation was not intended to have any impact on its published requirements and procedures governing, for example, granting, suspending, or withdrawing an accreditation. We agree with EAC’s interpretation, as it is in line with the intent of our recommendation. To avoid the potential for any future misunderstanding, we have modified the wording of the recommendation to clarify its intent.

EAC stated that the recommendation in our draft report for the commission to develop transparent technical requirements for the qualifications of its assessors may be confusing because, as we state in our report, only NIST performs a technical accreditation review, as EAC’s review is administrative, non-technical in nature. To avoid the potential for any confusion, we have modified the wording of the recommendation to eliminate any reference to technical qualification requirements.

We are sending copies of this report to the Ranking Member of the House Committee on House Administration, the Chairman and Ranking Member of the Senate Committee on Rules and Administration, the Chairmen and Ranking Members of the Subcommittees on Financial Services and General Government, Senate and House Committees on Appropriations, and the Chairman and Ranking Member of the House Committee on Oversight and Government Reform. We are also sending copies to the Chair and Executive Director of EAC, the Secretary of Commerce, the Deputy Director of NIST, and other interested parties. We will also make copies available to others on request. In addition, this report will be available at no charge on the GAO Website at http://www.gao.gov.

Should you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine whether the National Institute of Standards and Technology (NIST) and the Election Assistance Commission (EAC) have defined effective voting system testing laboratory (VSTL) accreditation approaches, and whether each is following its defined approach.

To determine whether NIST has defined an effective accreditation approach, we reviewed documentation from its VSTL accreditation program, such as handbooks and program manuals for the National Voluntary Laboratory Accreditation Program (NVLAP), of which the VSTL accreditation program is a part. In doing so, we compared these documents with applicable statute, guidance, and best practices, primarily the Help America Vote Act of 2002 (HAVA), internationally recognized standards from the International Organization for Standardization (ISO), and federal accreditation program management guidance published by NIST. We compared program documentation with HAVA’s NIST-specific accreditation requirements to determine the extent to which the agency was fulfilling its HAVA responsibilities. We also reviewed program documentation against ISO/IEC 17011, which establishes general requirements for accreditation bodies, and ISO/IEC 17025, which establishes the general requirements for assessing the competence of laboratories, to determine the extent to which NIST’s accreditation program was based on internationally recognized standards. We also compared the documentation against NIST publication NISTIR 6014, which contains sections that provide guidance for laboratory accreditation programs, to determine whether the VSTL accreditation program had defined other elements of effective accreditation programs. We also interviewed the voting accreditation program manager to determine how these documents were used to guide the program.

To determine whether NIST has followed its defined approach, we examined artifacts from the accreditation assessments of five VSTLs, including one laboratory accredited by NVLAP, but not yet recommended to EAC. This material included completed assessment checklists derived from the accreditation program handbooks, additional documents supporting the assessments, and laboratory accreditation applications and supporting documentation. We compared artifacts from these assessments to program guidance to determine the extent to which the defined process was followed. In addition, we interviewed officials from NIST and NIST contract assessors and officials from EAC and the four EAC-accredited VSTLs to understand how the NIST process was implemented and how it related to the process managed by EAC.

To determine whether EAC has defined an effective accreditation approach, we reviewed documentation from its VSTL accreditation program, such as the draft Voting System Test Laboratory Accreditation Program Manual. In doing so, we compared this document with applicable statute and best practices, primarily HAVA and federal accreditation program management guidance published by NIST. We compared the draft program manual with HAVA’s EAC-specific accreditation requirements to determine the extent to which the agency was fulfilling its HAVA responsibilities. We also compared the documentation against the accreditation guidance in NISTIR 6014 to determine whether the accreditation program had defined other elements of effective accreditation programs. We also interviewed the EAC voting program director and executive director to determine how these documents were used to guide the program and to understand EAC’s defined accreditation approach prior to the development of the draft manual.

To determine whether EAC has followed its defined approach, we compared artifacts from the accreditation reviews of four VSTLs. We did not review a fifth laboratory, which had been accredited by NVLAP, but not yet recommended to EAC. The materials reviewed included checklists completed by EAC in the absence of an approved program manual. In doing so, we compared the review artifacts to accreditation program requirements, as communicated to the laboratories, to determine the extent to which the agency followed its process, as verbally described to us. We did not compare accreditation submissions or EAC review artifacts with the draft accreditation manual because agency officials stated that the draft manual had not been used in the review of any laboratory. In addition, we interviewed officials from NIST, EAC, and the four EAC- accredited VSTLs to understand how the EAC process was implemented and how it related to the process managed by NIST.

To assess data reliability, we reviewed program documentation to substantiate data provided in interviews with knowledgeable agency officials. We have also made appropriate attribution indicating the data’s sources.

We conducted this performance audit at EAC and NIST offices in Washington, D.C., and Gaithersburg, Maryland, respectively, from September 2007 to September 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the National Institute of Standards and Technology

Appendix III: Comments from the Election Assistance Commission

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

Paula Moore, Assistant Director; Justin Booth; Timothy Case; Neil Doherty; Timothy Eagle; Nancy Glover; Dave Hinchman; Rebecca LaPaze; Freda Paintsil; Nik Rapelje; and Jeffrey Woodward made key contributions to this report.