Elections: Federal Programs for Accrediting Laboratories That Test Voting Systems Need to Be Better Defined and Implemented
Published: 2008-09-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The 2002 Help America Vote Act (HAVA) created the Election Assistance Commission (EAC) and assigned both it and the National Institute of Standards and Technology (NIST) responsibilities for accrediting laboratories that test voting systems. NIST assesses a laboratory's technical qualifications and makes recommendations to EAC, which makes a final accreditation decision. In view of the continuing concerns about voting systems and the important roles that NIST and EAC play in accrediting the laboratories that test these systems, GAO was asked to determine whether each organization has defined an effective approach for accrediting laboratories that test voting systems and whether each is following its defined approach. To accomplish this, GAO compared NIST and EAC policies, guidelines, and procedures against applicable legislation and guidance, and reviewed both agencies' efforts to implement them.
section_title: What GAO Found
paragraphs: NIST has largely defined and implemented an approach for accrediting voting system testing laboratories that incorporates many aspects of an effective program. In particular, its approach addresses relevant HAVA requirements and reflects relevant laboratory accreditation guidance, including standards accepted by the international standards community. However, NIST's defined approach does not, for example, cite explicit qualifications for the persons who conduct accreditation technical assessments, as called for in federal accreditation program guidance. Instead, NIST officials said that they rely on individuals who have prior experience in reviewing such laboratories. Further, even though the EAC requires that laboratory accreditation be based on demonstrated capabilities to test against the latest voting system standards, NIST's defined approach has not always cited these current standards. As a result, two of the four laboratories accredited to date were assessed using assessment tools that were not linked to the latest standards. Moreover, available documentation for the four laboratory assessments was not sufficient to determine how the checklists were applied and how decisions were reached. According to NIST officials, the four laboratories were consistently assessed. Moreover, they said that they intend to evolve NIST's accreditation approach to, for example, clearly provide for sufficient documentation of how accreditation reviews are conducted and decisions are reached. However, they had yet to develop specific plans for accomplishing this. EAC recently developed a draft laboratory accreditation program manual, but this draft manual does not adequately define all aspects of an effective approach, and it was not used in the four laboratory accreditations performed to date. Specifically, while this draft manual addresses relevant HAVA requirements, such as the requirement for the commissioners to vote on the accreditation of any laboratory that NIST recommends for accreditation, it does not include a methodology governing how laboratories are to be evaluated or criteria for granting accreditation. Because the manual was not approved at the time EAC accredited four laboratories, these accreditations were governed by a more broadly defined accreditation review process that was described in correspondence sent to each laboratory and a related document receipt checklist. As a result, these accreditations were based on review steps that were not sufficiently defined to permit them to be executed in a repeatable manner. According to EAC officials, including the official who conducted the accreditation reviews for the four laboratories, using the same person to conduct the reviews ensured that the steps performed on the first laboratory were repeated on the other three. However, given that both the steps and the results were not documented, GAO could not verify this. EAC officials stated that they intend to evolve the program manual over time and apply it to future accreditations and reaccreditations. However, they did not have specific plans for accomplishing this. Further, although EAC very recently approved an initial version of its program manual, this did not occur until after EAC provided comments, and GAO had finalized, this report.

section_title: Letter
paragraphs: In the wake of the 2000 and 2004 general elections, we issued a series of  reports and testified on virtually every aspect of our nation’s overall  election system, including the many challenges and opportunities  associated with various types of voting systems. In this regard, we  emphasized that voting systems alone were neither the sole contributor  nor solution to the problems that were experienced during the 2000 and  2004 elections, and that the overall election system depended on the  effective interplay of people, process, and technology and involved all  levels of government. Among many things, we specifically reported in  2001 that no federal entity was responsible for accrediting the  laboratories that tested voting systems, and we raised the establishment of  such an entity as a matter for congressional consideration.
Subsequently, Congress passed the Help America Vote Act (HAVA), which  created the Election Assistance Commission (EAC) and assigned both it  and the National Institute of Standards and Technology (NIST) separate  but related responsibilities for accrediting laboratories that test voting  systems. In general, NIST is responsible for assessing a laboratory’s  technical qualifications and making an accreditation recommendation to  EAC, while EAC is to use the assessment results and recommendation,  along with its own review of related laboratory capabilities, to reach an  accreditation decision. In 2004 and 2007, NIST and EAC established voting  system testing laboratory accreditation programs, respectively. To date,  EAC has accredited four laboratories. In view of the continuing concerns  about voting systems and the important roles that both NIST and EAC play  in accrediting the laboratories that test these systems, you asked us to  determine whether NIST and EAC have each defined an effective  laboratory accreditation approach and whether each is following its  defined approach.
To accomplish this, we reviewed NIST and EAC policies, guidelines, and  procedures governing voting system testing laboratory accreditation,  deaccreditation, and reaccreditation and compared them, as appropriate,  to applicable statute, such as HAVA, and guidance published by NIST, the  International Organization for Standardization, and us. We then compared  NIST and EAC actions and artifacts that were used for accrediting four  voting system testing laboratories to their respective policies, guidelines,  and procedures. We did not review a fifth laboratory because NIST was in  the process of assessing it when we started our review, and had yet to  recommend the laboratory to EAC for final accreditation. In addition, we  interviewed officials from NIST, EAC, and the four laboratories to  understand and clarify approaches taken, documentation provided, and  decisions reached.
We conducted this performance audit at EAC and NIST offices in  Washington, D.C., and Gaithersburg, Maryland, respectively, from  September 2007 to September 2008 in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.  Further details of our objective, scope, and methodology are included in  appendix I.
subsections: section_title: Background
paragraphs: All levels of government share responsibility in the overall U.S. election  system. At the federal level, Congress has authority under the Constitution  to regulate presidential and congressional elections and to enforce  prohibitions against specific discriminatory practices in all federal, state,  and local elections. Congress has passed legislation that addresses voter  registration, absentee voting, accessibility provisions for the elderly and  persons with disabilities, and prohibitions against discriminatory  practices. At the state level, individual states are responsible for the  administration of both federal elections and their own elections. States  regulate the election process, including, for example, the adoption of  voluntary voting system guidelines, the state certification and acceptance  testing of voting systems, ballot access, registration procedures, absentee  voting requirements, the establishment of voting places, the provision of  election day workers, and the counting and certification of the vote.
In total, the overall U.S. election system can be seen as an assemblage of  55 distinct election systems—those of the 50 states, 4 U.S. territories, and  the District of Columbia. Further, although election policy and procedures  are legislated primarily at the state level, states typically have  decentralized election systems, so that the details of administering  elections are carried out at the city or county levels, and voting is done at  the local level. As we reported in 2001, local election jurisdictions number  more than 10,000, and their sizes vary enormously—from a rural county  with about 200 voters to a large urban county, such as Los Angeles County,  where the total number of registered voters for the 2000 elections  exceeded the registered voter totals in 41 states. Further, these thousands  of jurisdictions rely on many different types of voting methods that employ  a wide range of voting system makes, models, and versions. Because of the  prominent role played by electronic voting systems, testing these systems  against national standards is critical to ensuring their security and  reliability. Equally critical is ensuring that the laboratories that perform  these tests are competent to carry out testing activities.
subsections: section_title: The Overall U.S. Election System Depends on Effective Interactions among People, Processes, and Technology
paragraphs: In the United States today, most votes are cast and counted by electronic  voting systems, and many states require use of systems that have been  certified nationally or by state authorities. However, voting systems are  but one facet of a multifaceted, continuous overall election system that  involves the interplay of people, processes, and technology during the  entire life of a system. All levels of government, as well as commercial  voting system manufacturers and system testing laboratories, play key  roles in ensuring that voting systems perform as intended.
Electronic voting systems are typically developed by manufacturers, then  purchased as commercial, off-the-shelf products and operated by state and  local election administrators. Viewed at a high level, these activities make  up three phases of a system life cycle: product development, acquisition,  and operations. (See fig. 1.) Key processes that span these life cycle  phases include managing the people, processes, and technologies within  each phase and across phases, and testing the systems and components  during and at the end of each phase. Additionally, voting system standards  are important through all of the phases because they provide criteria for  developing, testing, and acquiring voting systems, and they specify the  necessary documentation for operating the systems.
The product development phase includes activities such as establishing  requirements for the system, designing a system architecture, developing  software, and integrating components. Activities in this phase are  performed by the system vendor.
The acquisition phase includes activities such as publishing a solicitation,  evaluating offers, choosing a voting technology and a vendor, and  awarding and administering contracts. For voting systems, activities in this  phase are primarily the responsibility of state and local governments but  entail some responsibilities that are shared with the system vendor (e.g.,  entering into the contract).
The operations phase consists of activities such as ballot design and  programming, setup of systems before voting, pre-election testing, vote  capture and counting during elections, recounts and system audits after  elections, and storage of systems between elections. Responsibility for  activities in this phase typically resides with local jurisdictions, whose  officials may, in turn, rely on or obtain assistance from system vendors for  aspects of these activities.
Standards for voting systems, as will be discussed in a later section, were  developed at the national level by the Federal Election Commission in  1990 and 2002 and were updated by EAC in 2005. In the product  development phase, voting system standards serve as requirements to  meet for developers to build systems. In the acquisition phase, they also  provide a framework that state and local governments can use to evaluate  systems. In the operations phase, they specify the necessary  documentation for operating the systems.
Testing processes are conducted throughout the life cycle of a voting  system. Voting system vendors conduct product testing during  development of the system and its components. Federal certification  testing of products submitted by system vendors is conducted by national  voting system testing laboratories (VSTL). States may conduct evaluation  testing before acquiring a system to determine how well products meet  their state-specific specifications, or they may conduct certification testing  to ensure that a system performs its functions as specified by state laws  and requirements. Once a voting system is delivered by the system vendor,  states and local jurisdictions may conduct acceptance testing to ensure  that the system satisfies functional requirements. Finally, local  jurisdictions typically conduct logic and accuracy tests related to each  election and sometimes subject portions of the system to parallel testing  during each election to ensure that the system components perform  accurately.
Management processes ensure that each life cycle phase produces a  desirable outcome. Typical management activities that span the system life  cycle include planning, configuration management, system performance  review and evaluation, problem tracking and correction, human capital  management, and user training. These activities are conducted by the  responsible parties in each life cycle phase.
In 2004, we reported that the performance of electronic voting systems,  like any type of automated information system, can be judged on several  bases, including their security, accuracy, ease of use, efficiency, and cost.  We also reported that voting system performance depends on how the  system was designed, developed, and implemented.
section_title: Laboratory Accreditation Plays an Important Role in Ensuring Accurate, Reliable, and Secure Voting Systems
paragraphs: Since the passage of HAVA, the use of electronic voting systems has  increased and become the predominant method of voting. However,  concerns have been raised about the security and reliability of these  systems. As we have previously reported, testing and certifying voting  systems is one critical step in acquiring, deploying, operating, and  administering voting systems, which better ensures that they perform  securely and reliably. Among other things, rigorous execution and careful  documentation of system testing is a proven way to help ensure that  system problems are found before the systems are deployed and used in  an election. To accomplish this, it is vital that the organizations that test  the systems be qualified and competent to do so. For voting systems, a key  testing organization is a federally accredited, national VSTL.
In general, accreditation is the formal recognition that a laboratory is  competent to carry out specific types of tests or calibrations. Federally  accredited laboratories perform many different types of testing and related  activities on various products, ranging from inspecting grain to certifying  maritime cargo gear. The genesis of laboratory accreditation programs  owes largely to agencies’ need to assure themselves of the competency of  the organizations responsible for testing products or services that involve  the use of federal funds.
To provide national recognition for competent laboratories, the NIST  Director established the National Voluntary Laboratory Accreditation  Program (NVLAP) in 1976 at the request of the private sector. Under this  program, which is based on internationally accepted standards, NIST  accredits laboratories that it finds competent to perform specific types of  tests or calibrations. In June 2004, NVLAP announced the establishment, in  accordance with HAVA, of an accreditation program for laboratories that  test voting systems using standards determined by EAC.
section_title: HAVA Assigned EAC and NIST Responsibility for Accrediting VSTLs
paragraphs: Enacted in October 2002, HAVA affected nearly every aspect of the voting  process, from voting technology to provisional ballots and from voter  registration to poll worker training. In particular, the act authorized $3.86  billion in funding over several fiscal years to replace punch card and  mechanical lever voting equipment, improve election administration and  accessibility, train poll workers, and perform research and pilot studies.
HAVA also established EAC, provided for the appointment of four  commissioners, and specified the process for selecting an executive  director. Generally speaking, EAC is to assist in the administration of  federal elections and provide assistance in administering certain federal  election laws and programs.
Since the passage of HAVA in 2002, the federal government has taken  steps to implement the act’s provisions. For example, after beginning  operations in January 2004, EAC updated the existing federal voluntary  standards for voting systems, including strengthening provisions related to  security and reliability. Additionally, EAC established an interim VSTL  accreditation program that leveraged a predecessor program run by the  National Association of State Elections Directors, and EAC and NIST then  established companion accreditation programs that replaced the interim  program.
subsections: section_title: EAC Updated the Federal Voluntary Standards for Voting Systems
paragraphs: Federal standards for voting systems were first issued in 1990 when the  Federal Election Commission published standards. These federal  standards identified minimum functional and performance requirements,  which states were free to adopt in whole, in part, or not at all, for  electronic voting equipment, and specified test procedures to ensure that  the equipment met those requirements. In 2002, the Federal Election  Commission issued its Voting System Standards (VSS), which updated the  1990 standards to reflect more modern voting system technologies. In  2005, we reported that these standards identified minimum functional  and performance requirements for voting systems but were not sufficient  to ensure secure and reliable voting systems. As a result, we  recommended that EAC work to define specific tasks, measurable  outcomes, milestones, and resource needs to improve the voting system  standards. Until then, election administrators were at risk of relying on  voting systems that were not developed, acquired, tested, operated, or  managed in accordance with rigorous security and reliability standards— potentially affecting the reliability of future elections and voter confidence  in the accuracy of the vote count.
Following the enactment of HAVA in 2002 and the establishment of EAC in  2004, EAC adopted the Voluntary Voting System Guidelines (VVSG) in  2005. The VVSG specify the functional requirements, performance  characteristics, documentation requirements, and test evaluation criteria  for the national certification of voting systems. Accredited testing  laboratories are to use the VVSG to develop test plans and procedures for  the analysis and testing of systems in support of EAC’s voting system  certification program. The VVSG are also used by voting system  manufacturers as the basis for designing and deploying systems that can  be federally certified.
section_title: EAC Established an Interim Accreditation Program
paragraphs: We reported in 2001 that the National Association of State Elections  Directors was accrediting independent test authorities to test voting  equipment against the Federal Election Commission standards. Under this  program, three laboratories were accredited. Under HAVA, NIST is to  recommend laboratories for EAC accreditation. In 2006, NIST notified  EAC that its initial recommendations might not be available until  sometime in 2007. As a result, EAC initiated an interim accreditation  program and invited the three laboratories accredited by the state  elections directors to apply. As part of the interim program, laboratories  were required to attest to a set of EAC-required conditions and practices,  including certifying the integrity of personnel, the absence of conflicts of  interest, and the financial stability of the laboratory. In August and  September 2006, EAC granted interim accreditation to two of the three  laboratories invited to apply. EAC terminated its interim program in March  2007.
section_title: EAC and NIST Have Established Separate but Related Laboratory Accreditation Programs
paragraphs: HAVA assigned responsibilities for laboratory accreditation to both EAC  and NIST. In general, to reach an accreditation decision, NIST is to focus  on assessing laboratory technical qualifications, while EAC is to use those  assessment results and recommendations and augment them with its own  review of related laboratory capabilities. See table 1 for the two agencies’  HAVA responsibilities.
The tasks that NIST is to perform  addressed in an annual interagency agreement executed between the  institute and EAC each year. For example, the 2008 interagency agreem states that NVLAP will continue to assess VSTLs and will coordinate with  EAC to continually monitor and review the performance of the  laboratories. Additionally, the agreement states that the two agencies will  coordinate to maintain continuity between their respective accreditation  programs.  in meeting HAVA’s requirements are  The NIST and EAC accreditation programs can be viewed together as  forming a federal VSTL accreditation process that consists of a series o complementary steps. These steps are depicted in figure 2, where the  numbers correspond to a detailed narrative description below.
section_title: Status of Completed VSTL Accreditation Activities
paragraphs: As of May 2008, EAC has accredited four laboratories. These laboratories  are SysTest Labs, LLC; Wyle Laboratories, Inc.; iBeta Quality Assurance;  and InfoGard Laboratories, Inc. A fifth laboratory, CIBER Inc., has been  granted NVLAP accreditation and has been recommended to, but not yet  accredited by, EAC. InfoGard Laboratories, Inc., whose NVLAP  accreditation expires in June 2008, has recently notified NIST and EAC  that it would not apply to renew its accreditation, citing the volatility of  the voting system environment as one reason. The timeline for each of  these accreditations, and other accreditation program activities, is found  in figure 3.
section_title: NIST Has Defined and Implemented an Accreditation Approach That Reflects Relevant Standards but Is Missing Details Needed for Consistent and Verifiable Implementation
paragraphs: NIST’s defined approach to accrediting voting system laboratories largely  reflects applicable HAVA requirements and relevant international  standards, both of which are necessary to an effective program. However,  this approach is continuing to evolve based on issues realized during  NIST’s implementation experience to date. In particular, because NIST’s  defined program does not, for example, specify the nature and extent of  assessment documentation to generate or retain or specify the version of  the voting system standards to be used, our analysis of NIST’s efforts in  accrediting four laboratories could not confirm that the agency has  consistently followed its defined accreditation program. NIST officials  stated that these limitations are due in part to the relative newness of the  program and that they will be addressed by updating the accreditation  program handbook. However, they said that they do not have documented  plans to accomplish this. Until these limitations are addressed, NIST will  be challenged in accrediting voting system laboratories in a consistent and  verifiable manner.
subsections: section_title: NIST Voting System Accreditation Program Reflects HAVA Requirements
paragraphs: NIST has defined its voting system accreditation program to address  relevant HAVA requirements. According to HAVA, NIST is to    conduct reviews of independent, nonfederal voting system testing  laboratories and submit to EAC a list of proposed voting system testing  laboratories and   monitor and review the performance of those proposed laboratories that  EAC accredits, including making recommendations to EAC regarding  accreditation continuance and revocation.
NIST’s defined voting system accreditation program satisfies both of these  requirements. With respect to the first, NIST announced in June 2004 the  establishment of its voting system testing laboratory accreditation  program as part of NVLAP, a statutorily created program for unbiased,  third parties to establish the competence of national independent  laboratories. As such, NIST adopted its NVLAP handbook as the basis for  its defined approach to reviewing VSTLs and has supplemented it with a  handbook that is specific to voting system testing.
With respect to the second HAVA requirement, the supplemental  handbook cited above states that the NIST Director will recommend  NVLAP-accredited VSTLs to EAC for subsequent commission  accreditation. Additionally, NIST’s handbooks provide for both monitoring  accredited laboratories and for making recommendations regarding a  laboratory’s continued accreditation. For example, the handbook states  that a monitoring visit may occur at both scheduled and unscheduled  times and the scope may be limited to a few items or include a full review.  It also states that a reaccreditation review shall be conducted in  accordance with the procedures used to initially accredit laboratories.  Further, the handbook also identifies accreditation or reaccreditation  decision options, including granting, denying, or modifying the scope of an  accreditation.
According to NIST officials, these HAVA requirements are relevant and  important to defining an effective voting system testing laboratory  accreditation program. By incorporating them, NIST has reflected one key  aspect of an effectively defined program.
section_title: NIST Has Incorporated Relevant International Accreditation Standards into Its VSTL Accreditation Program
paragraphs: NIST’s VSTL accreditation program reflects internationally recognized  standards for establishing and conducting accreditation activities. These  standards are published by the International Organization for  Standardization (ISO), and the two that are germane to this accreditation  program are (1) ISO/IEC 17011, which establishes general requirements  for accreditation bodies and (2) ISO/IEC 17025, which establishes the  general requirements for reviewing the competence of laboratories.  According to NIST program documentation, this allows NVLAP to both  operate as an unbiased, third party accreditation body and to utilize a  quality management system compliant with international standards. As a  result, NIST has incorporated key aspects of an effective accreditation  body into its voting system accreditation program.
subsections: section_title: NIST Program Meets ISO Accreditation Body Requirements
paragraphs: ISO/IEC 17011 requires that an accrediting body have, among other things,  (1) a management system for accreditation activities, (2) a policy defining  the types of records to be retained and how those records will be  maintained, (3) a clear description of the accreditation process that covers  the rights and responsibilities of those seeking accreditation, and (4) a  clear description of the accreditation activities to be performed.
NIST VSTL accreditation program-related documentation, including its  program handbooks, satisfies each of these requirements. In fact, NIST has  cross-referenced its documentation to each ISO/IEC 17011 requirement.  Specifically, the first requirement is cross-referenced to the NVLAP  Management System Manual, which describes the overall accreditation  program’s management policies and control structure, and the second is  cross-referenced to the program’s record keeping policy, which specifies  what types of records should be maintained and how they should be  maintained. The third and fourth requirements are cross-referenced to the  accreditation process descriptions in both the Management System  Manual and the general handbook. Together, these documents contain, for  example, (1) the rights of laboratories applying for accreditation and (2)  the scope of accreditation activities to be performed, including a  preassessment review, an on-site review, and a final on-site assessment  report.
section_title: NIST Program Meets ISO Laboratory Accreditation Review Requirements
paragraphs: ISO/IEC 17025 requires that accreditation reviews cover specific topics.  These include (1) laboratory personnel independence and conflicts of  interest; (2) a laboratory system for quality control (i.e., a framework for  producing reliable results and continuous improvement to laboratory  procedures); and (3) a laboratory mechanism for collecting and  responding to customer complaints. Additionally, the standard establishes  basic technical requirements that a laboratory has to meet, and thus that  reviews are to cover, including (1) competent laboratory personnel who  are capable of executing the planned tests, (2) appropriate tests and test  methods, and (3) clear and accurate test result documentation.
NIST voting system testing laboratory accreditation program-related  documents, including its program handbooks, satisfy these requirements.  First, the general handbook defines the requirement for a laboratory to  have personnel that are independent and free of any conflict of interest.  Second, the handbook requires that a laboratory have a management  quality control system and that this system provide for reliable results and  continuous improvement to laboratory procedures. Third, the handbook  requires that a laboratory have a mechanism for receiving and responding  to customer complaints. Last, the handbook establishes certain technical  requirements that a laboratory must meet, such as having competent  laboratory personnel capable of executing the planned tests, using  appropriate tests and test methods, and documenting test results in a clear  and accurate manner.
For several of these requirements, NIST’s voting-specific supplemental  handbook augments the general handbook. For example, this  supplemental handbook requires laboratories to submit a quality control  manual, as well as information to demonstrate the competence of  laboratory administrative and technical staff. Further, it requires that a  laboratory’s training program be updated so that staff can be retrained as  new versions of voting system standards are issued.
section_title: NIST Voting System Accreditation Program Does Not Reflect Its Own Findings on the Need for Assessor Qualifications and Training and Key Assessment Criteria
paragraphs: NIST has reported on the importance of ensuring that those persons who  perform accreditation assessments are sufficiently qualified and that the  assessments themselves are based on explicitly defined criteria and are  adequately documented. Nevertheless, NIST has not fully reflected key  aspects of these findings in its defined approach to accrediting voting  system testing laboratories. For example, it has not specified the basis for  determining the qualifications of its accreditation assessors, and while a  draft update to its handbook now includes the specific voting system  standards to be used when performing an accreditation assessment, this  handbook was only recently approved. According to NIST officials, these  gaps are due to the newness of the accreditation program and will be  addressed in the near future. Because these gaps have confused  laboratories as to what standards they were to meet, and may have  resulted in differences in how accreditations have been performed to date,  it is important that the gaps be addressed.
subsections: section_title: NIST Has Not Specified Requirements for Assessor Qualifications and Training
paragraphs: NIST has reported on the importance of having competent and qualified  human resources to support accreditation programs. According to these  findings, an accreditation program should, among other things, provide for    having experienced and qualified assessors to perform accreditation    demonstrating an assessors’ qualifications using defined documentation  and explicit criteria that encompass the person’s education, experience,  and training; and  training (initial and continuing) for assessors.
NIST’s defined approach to VSTL accreditation does not provide for all  these requirements. To its credit, its program handbook identifies the need  for experienced and qualified assessors in the execution of accreditation  activities and provides for each assessor’s qualifications to be  documented. Further, it has defined generic training that applies to all of  its accreditation assessors. For example, the NVLAP Assessor Training  Syllabus includes training on ISO/IEC 17011 and 17025, as well as training  on the NVLAP general handbook. In addition, the VSTL accreditation  program manager stated that new assessors receive training on the 2002  VSS and 2005 VVSG and that periodic training seminars are provided to  assessors on changes to either the general handbook or the 2005 VVSG.
In addition, the program manager told us that candidate assessors must  submit some form of documentation (e.g., a resume), and that this  documentation is used to evaluate, rank, and select candidates that are  best qualified. The NIST VSTL assessors that we interviewed confirmed  that they were required to submit such documentation at NIST’s request.
However, NIST’s defined approach does not cite the explicit capabilities  and qualifications that an assessor must meet or the associated  documentation needed to demonstrate these capabilities and  qualifications. According to the program manager, this is because the field  of potential assessors in the voting system arena is small and specialized  and because they focused on defining other aspects of the program that  were higher priorities. Further, NIST has not defined and documented the  specific training requirements needed to be a VSTL lead assessor or a  technical assessor for the VSTL program. According to the program  manager, this is because these assessors receive all the training they need  by working on the job with more experienced assessors. Not specifying  criteria governing assessor qualifications and training is of concern  because differences in assessors’ capabilities could cause inconsistencies  in how assessments are performed.
section_title: NIST’s Approach Does Not Fully Specify Criteria for Evaluating and Documenting VSTL Capabilities
paragraphs: NIST recognizes the importance of specifying explicit criteria against  which all candidate laboratories will be assessed and fully documenting  the assessments that are performed. Specifically, the general handbook  provides the criteria and requirements that will be used to evaluate basic  laboratory capabilities. It also states that technical requirements specific  to a given field of accreditation are published in program-specific  handbooks. To that end, NIST published a supplemental program-specific  handbook in December 2005 that provided the voting-specific  requirements to be used to evaluate VSTLs, additional guidance, and  related interpretive information.
NIST’s 2005 supplemental handbook does not contain sufficient criteria  against which to evaluate VSTLs. It identifies specific requirements that  laboratories are to demonstrate relative to the 2002 VSS but not the 2005  VVSG. For example, the handbook states that laboratories are expected to  develop, validate, and document test methods that meet the 2002 VSS.  However, it does not refer to the 2005 VVSG. In addition, the program- specific checklist that accompanies this version of the handbook does not  identify all the 2005 VVSG standards against which laboratories are  evaluated. Specifically, this checklist makes reference to the VVSG in  relation to just a few checklist requirements.
According to the NIST program manager, the 2005 handbook did not refer  to the 2005 VVSG requirements because only the 2002 VSS requirements  were mandatory at the time it was published. He further stated that,  despite the fact that the 2005 VVSG requirements were not included in that  handbook, NIST assessors were expected to use them when performing  the first laboratory assessments. Representatives for two laboratories  stated that because these requirements were not documented or identified  in the NIST handbooks, they did not learn that they would be required to  demonstrate 2005 VVSG-based capabilities until the NIST on-site  assessment teams arrived.
In December 2007, NIST released draft revisions of the voting program- specific handbook and checklist, stating that labs are expected to meet  both 2002 VSS and 2005 VVSG. In addition, the 2007 draft handbook clearly  specifies that laboratories must demonstrate how developed test methods  and planned tests trace back to and satisfy both the 2002 VSS and the 2005  VVSG. Taken together, the new handbook and checklist should better  identify the requirements and criteria used to evaluate a laboratory and  document the results. According to NIST, the new handbook and checklist  have recently been finalized, and both are now in use.
section_title: Available Documentation Does Not Show That NIST Has Consistently Followed All Aspects of Its Defined Accreditation Approach
paragraphs: NIST has found that reliable and accurate documentation provides  assurance that laboratory accreditation activities have been effectively  fulfilled. However, in its efforts to date in accrediting four VSTLs,  documentation of the assessments does not show that NIST has fully  followed its defined accreditation approach. While we could not determine  whether this is due to incomplete documentation of the steps performed  and the decisions made during an assessment or due to steps not being  performed as defined, this absence of verifiable evidence raises questions  about the consistency of the assessments and the resultant accreditations.  Without adequately documenting each assessment, including all steps  performed and the basis for any steps not performed, such questions may  continue to be raised.
To NIST’s credit, available documentation shows that it consistently  followed some aspects of its defined approach in accrediting the four  laboratories. For example, we verified that NIST received an application  from each of the laboratories as required, and our review of completed  checklists and summary reports shows that preassessment reviews and  on-site assessments were performed for each laboratory, as was required.  According to a lead assessor, this review usually focused on the  laboratories’ quality assurance manuals. Moreover, the completed  checklists identified whether the requirement was met or not for each  listed requirement, and included comments, in some cases, as to how a  laboratory addressed a requirement. Also as required, NIST received  laboratory responses describing how unmet requirements were addressed  within specified time frames, used the responses in making accreditation  decisions, and notified EAC of its decisions via letters of recommendation.  Furthermore, NIST has recently begun reaccreditation reviews at two  laboratories, as required.
However, documentation does not show that NIST has consistently  followed other aspects of its defined approach. Our analysis of the  checklists that are to be used to both guide and document a given  assessment, including identifying unmet requirements and capturing  assessor comments and observations, shows some differences. For  example:    One type of checklist (the supplemental handbook checklist) was  prepared for only two of the four laboratory assessments. According to the  program manager, this is because even though a draft revision of this  checklist was actually used to assess the other two laboratories, the  assessment results were recorded on a different checklist (the general  handbook checklist). While this is indicated on one of the two checklists,  it is not indicated on the other.
On the checklist used for one laboratory, an assessor marked several  sections as “TA” with no explanation as to what this means. Also, the  checklist used for another laboratory did not identify whether most of the  requirements were met or not met. Further, the checklist for a third  laboratory had one section marked as “not applicable” but included no  explanation as to why that section did not apply, while the checklist for a  different laboratory marked the same section as “not applicable” but  included a reason for doing so.
Notwithstanding these differences, the program manager told us that each  laboratory was assessed using the same requirements and all assessments  to date were performed in a consistent manner. On the basis of available  documentation, however, we could not verify that this is the case. As a  result, it is not clear that NIST has consistently followed its defined  approach.
Available documentation also does not show that NIST followed other  aspects of its approach. For example:    The program handbook states that each laboratory is to identify the  requested scope of accreditation in its application package. However, our  analysis of the four application packages shows that two laboratories did  not specify a requested scope of accreditation. According to the program  manager, the scope of accreditation for all laboratories was the 2002 VSS  and 2005 VVSG because, even though the latter standards were not yet in  effect at the time, they were anticipated to be in effect in the near future.  However, NIST did not have documentation that notified the laboratories  of this scope of accreditation or that indicated whether this scope was  established by EAC, NIST, or the laboratories.
The program handbook states that after receiving a laboratory’s  application package, NIST will acknowledge its receipt in writing and will  inform the laboratory of the next steps in the accreditation process.  However, NIST did not have documentation demonstrating that this was  done. According to the program manager, this was handled via telephone  conversations. However, representatives for several laboratories noted  that these calls did not clearly establish expectations, adding that some  expectations were not communicated until the NIST team assessors  arrived to conduct the on-site assessment.
The program manager stated that these deviations from the defined  approach are attributable to the relative newness of the program, but  despite these discrepancies, each laboratory was assessed consistently.  However, we could not verify this, and thus it is not clear that NIST has  consistently followed its defined approach. According to this official,  future versions of the program handbook would address these limitations.  However, documented plans for doing so have not been developed.
section_title: EAC’s Recently Drafted Accreditation Approach and Its Earlier Performed Laboratory Accreditations Lack Key Effectiveness Factors and Features
paragraphs: EAC has recently defined its voting system laboratory accreditation  approach in a draft program manual. However, this draft manual omits  important content. While addressing relevant HAVA requirements, the  draft manual does not adequately define key accreditation factors that  NIST has identified, and a key accreditation feature that we have  previously reported as being integral to an effective accreditation  program. Moreover, not all factors and features that the draft manual does  include have been defined to a level that would ensure thorough,  consistent, and verifiable implementation. Because this manual was not  available for EAC to use on the four laboratory accreditations that it has  completed, the accreditations were performed using a largely  undocumented series of steps. As a result, the thoroughness and  consistency of these accreditations is not clear. According to EAC  officials, these gaps are due to the agency’s limited resources being  focused on other issues, and will be addressed as its accreditation  program evolves. However, they said that they do not yet have  documented plans to accomplish this. Until EAC fully defines a repeatable  VSTL accreditation approach, it will be challenged in its ability to treat all  laboratories consistently and produce verifiable results.
subsections: section_title: EAC Has Defined a Draft Accreditation Approach that Meets HAVA Requirements
paragraphs: In February 2008, EAC issued a draft version of a VSTL accreditation  program manual for public comment. According to HAVA, EAC’s  accreditation program is to meet certain requirements. Specifically, it is to  provide for voting system hardware and software testing, certification,  decertification, and recertification by accredited laboratories.  Additionally, it is to base laboratory accreditation decisions, including  decisions to revoke an accreditation, on a vote of the commissioners, and  it is to provide for a published explanation of any commission decision to  accredit any laboratory that was not first recommended for accreditation  by NIST.
To EAC’s credit, its draft accreditation program manual addresses each of  these requirements. First, the manual defines the role that the laboratories  are to play relative to voting system testing, certification, recertification  and decertification, and it incorporates by reference an EAC companion  voting system certification manual that defines requirements and process  steps for voting system testing and certification-related activities.
With respect to the remaining three HAVA requirements, the draft EAC  accreditation manual also requires (1) that the commissioners vote on the  accreditation of laboratories recommended by NIST for accreditation, (2)  that EAC publish an explanation for the accreditation of any laboratory  not recommended by NIST for accreditation, and (3) that the  commissioners vote on the proposed revocation of a laboratory’s  accreditation.
According to EAC officials, its draft approach incorporates HAVA  requirements because the commission is focused on meeting its legal  obligations in all aspects of its operations, including VSTL accreditation. In  doing so, EAC has addressed one important aspect of having an effective  accreditation program.
section_title: EAC Draft Approach Does Not Adequately Define Key Accreditation-Related Steps and Decision Criteria
paragraphs: Beyond addressing relevant HAVA requirements, EAC’s draft accreditation  manual defines an accreditation process, including program phases,  requirements, and certain evaluation criteria. However, it does not do so in  a manner that fully satisfies factors that NIST has reported can affect the  effectiveness of accreditation programs. Moreover, it does not adequately  address a set of features that our research shows are common to federal  accreditation programs and that can influence a program’s effectiveness.  According to EAC officials, these factors and features are not fully  addressed in the draft program manual because its accreditation program  is still in its early stages of development and is still evolving. Until they are  fully addressed, EAC’s accreditation program’s effectiveness will be  limited.
subsections: section_title: Key NIST Accreditation Factors Not Fully Addressed
paragraphs: According to NIST, having confidence in and ensuring appropriate use of  an accredited testing laboratory requires that accreditation stakeholders  have an adequate understanding of the accreditation process, scope, and  related criteria. NIST further reports that confidence in the accreditation  process can be traced to a number of factors that will influence the  thoroughness and competence of accreditation programs, and thus these  factors can be viewed as essential accreditation program characteristics.  They include having    published procedures governing how the accreditation program is to be  executed, such as procedures for granting, maintaining, modifying,  suspending, and withdrawing accreditation;    specific instructions, steps, and criteria for those who conduct an  accreditation assessment (assessors) to follow, such as a test methodology  that is acceptable to the accreditation program;   knowledgeable and experienced assessors to execute the instructions and  steps and apply the related criteria; and    complete records on the data collected, results found, and reports  prepared relative to each assessment performed.
EAC’s draft accreditation program manual addresses one of these factors  but it does not fully address the other three. (See table 2.) For example,  while the manual requires that EAC maintain records, it only addresses the  retention of records associated with the testing of voting systems and not  those associated with the accreditation of laboratories. EAC officials told  us that testing records are meant to include accreditation records,  although they added that this is not explicit in the manual and needs to be  clarified. Further, the manual is silent on the steps to be followed and  criteria to be applied in reviewing a laboratory’s application and the  qualifications required for accreditation reviewers. By not fully addressing  these factors, EAC increases the risk that its accreditation reviews will not  be performed consistently and comprehensively.
As we have previously reported, the nature and focus of federal programs  for accrediting laboratories vary, but nevertheless include certain common  features. In particular, these programs require laboratories to provide  certain information to the accrediting body, and they provide for  evaluation of this information by the accrediting body in making an  accreditation determination. As we reported, the required information is to  include, among other things, the laboratory’s (1) organizational  information, (2) records and record-keeping policy, (3) test methods and  procedures, (4) conflict of interest policy, and (5) financial stability.
To its credit, EAC’s draft accreditation manual provides for laboratories to  submit information relative to each of these features that are common to  federal accreditation programs. For example, it provides for laboratories  to submit organizational information, such as location(s), ownership, and  organizational chart; a written policy for maintaining accreditation-related  records for 5 years; conflict of interest policies and procedures; test- related polices and procedures, as well as system-specific test plans; and  financial information needed to demonstrate stability. Moreover, for four  of the five features, the manual identifies the specific types of information  needed for accreditation and how the information is to be evaluated,  including the criteria that are to be used in evaluating it. However, for the  financial stability feature, the manual does not describe what specific  documents are required from the laboratory to satisfy this requirement,  nor does the manual indicate how information provided by a laboratory  will be evaluated.
At the time of our review, EAC’s Director of Voting System Testing and  Certification told us that the draft accreditation manual was to be  submitted for approval and that this draft did not address all of the  limitations cited above. For example, it would not contain the  information needed and the evaluation approach and criteria to be used in  making determinations about financial stability because this decision is to  be based on what the director referred to as a “reasonableness” test that  involves EAC evaluation of the information relative to that provided by  other laboratories. Further, while EAC officials said that they plan to  evolve their approach to VSTL accreditation and to address these gaps,  EAC does not have documented plans for accomplishing this. Without  clearly defining information to be used and how it is to be used, EAC  increases the risk that financial stability determinations will not be  consistently and thoroughly made.
section_title: Available Documentation Does Not Demonstrate EAC’s Basis for Accrediting Laboratories to Date
paragraphs: As of May 2008, EAC has accredited four laboratories, but the  documentation associated with each of these accreditations is not  sufficient to recreate a meaningful understanding of how each evaluation  was performed and how decisions were made, and thus, the bases for each  accreditation were not clear. Specifically, each of the accreditations  occurred before EAC had defined its approach for conducting them.  Because of this, EAC performed each one using a broadly defined process  outlined in a letter to each laboratory and an associated checklist that only  indicated whether certain documents were received. Our analysis of these  letters showed that the correspondence sent to each laboratory was all the  same, identifying three basic review steps to be performed and citing a list  of documents that the laboratories were to provide as part of their  applications. However, the letters did not describe in any manner how  EAC would review the submitted material, including the criteria to be  used.
According to EAC officials, the review steps were not documented.  Instead, they were derived by a single reviewer using (1) the applications  and accompanying documents submitted by the laboratories, (2)  familiarity with the materials used by the state election directors- sponsored accreditation program, and (3) the judgment of each reviewer.  Further, while the reviews were supported by a checklist that covered  each of the items that was to be included in the laboratory applications  and provided space for the reviewer(s) to make notes relative to each of  these items, the checklists did not include any guidance or methodology,  including criteria, for evaluating the submitted items. Rather, the EAC  accreditation program director told us that he was the reviewer on all the  accreditations and he applied his own, but undocumented, tests for  reasonableness in deciding on the submissions’ adequacy and  acceptability.
Our analysis of the checklists for each laboratory accreditation showed  that while the same checklist was used for each laboratory, the checklists  did not provide a basis for evaluating and documenting the basis for the  sufficiency of those documents. In some cases, additional communications  occurred between the reviewer and the laboratory to obtain additional  documents. However, no documentation was available to demonstrate  what standards or other criteria the laboratories were held to or how their  submissions were otherwise reviewed. For example, each of the checklists  indicated that each laboratory provided “a copy of the laboratory’s conflict  of interest policy.” However, they did not specify, for example, whether  the policy adequately addressed particular requirements. Nevertheless, for  three of the four accredited laboratories, documentation shows that EAC  sought clarification on or modification to the policies provided, thus  suggesting that some form of review was performed against more detailed  requirements. Similarly, while the checklists indicate that the laboratories  disclosed their respective coverage limits for general liability insurance  policies, and in one case EAC communicated to the laboratory that the  limits appeared to be low, no documentation specifies the expected  coverage limits. According to the EAC Director of Voting System Testing  and Certification, this determination was made after comparing limits  among the laboratories and was not based on any predetermined  threshold. Further, while the checklists indicate that each laboratory  provided audited financial statements, there is no documentation  indicating how these statements were reviewed.
According to the EAC program director, the lack of documentation  demonstrating the basis for EAC’s laboratory accreditations is due to the  need at the time to move quickly in accrediting the laboratories and the  fact that use of the same individual to review the accreditation evaluation  negated the need for greater documentation. Without such documentation,  however, we could not fully establish how the accreditations were  performed, including whether there was an adequate basis for the   accreditation decisions reached and whether they were performed  consistently.
section_title: Conclusions
paragraphs: The effectiveness of our nation’s overall election system depends on many  interrelated and interdependent variables, including the security and  reliability of voting systems. Both NIST and EAC play critical roles in  ensuring that the laboratories that test these two variables have the  capability, experience, and competence necessary to test a voting system  against the relevant standards. NIST has recently established an  accreditation program that largely accomplishes this, and while EAC is not  as far along, it has a foundation upon which it can build.
However, important elements are still missing from both programs.  Specifically, the current NIST approach does not define requirements for  assessor qualifications and training or ensure that assessments are fully  documented. Additionally, EAC has not developed program management  practices that are fully consistent with what NIST has found to be  hallmarks of an effective accreditation program, nor has the agency  adequately specified how evaluations are to be performed and  documented. As a result, opportunities exist for NIST and EAC to further  define and implement their respective programs in ways that promote  greater consistency, repeatability, and transparency—and thus improve  the results achieved. It is also important for NIST and EAC to follow  through on their stated intentions to evolve their respective programs,  building on what they have already accomplished through the  development and execution of well-defined plans of action. If they do not,  both will be challenged in their ability to consistently provide the  American people with adequate assurance that accredited laboratories are  qualified to test the voting systems that will eventually be used in U.S.  elections.
section_title: Recommendations for Executive Action
paragraphs: To help NIST in evolving its VSTL accreditation program, we recommend  that the Director of NIST ensure that the accreditation program manager  develops and executes plans that specify tasks, milestones, resources, and  performance measures that provide for the following two actions:    Establish and implement transparent requirements for the technical  qualifications and training of accreditation assessors.
Ensure that each laboratory accreditation review is fully and consistently  documented in accordance with NIST program requirements.
To help EAC in evolving its VSTL accreditation program, we recommend  that the Chair of the EAC ensure that the EAC Executive Director  develops and executes plans that specify tasks, milestones, resources, and  performance measures that provide for the following action:    Establish and implement practices for the VSTL accreditation program  consistent with accreditation program management guidance published by  NIST and GAO, including   documentation of specific accreditation steps and criteria to guide  assessors in conducting each laboratory review; transparent requirements for the qualifications of accreditation  reviewers; requirements for the adequate maintenance of records related to the  VSTL accreditation program; and requirements for determining laboratory financial stability.
section_title: Agency Comments and Our Evaluation
paragraphs: Both NIST and EAC provided written comments on a draft of this report,  signed by the Deputy Director of NIST and the Executive Director of EAC,  respectively. These comments are described below along with our  response to them.
In its comments, NIST stated that it appreciates our careful review of its  VSTL program and generally concurs with our conclusions that its  program must continue to evolve and improve. However, NIST also  provided comments to clarify the current status of the program relative to  three of our findings.
With respect to our finding that NIST’s defined approach for accrediting  VSTLs does not cite explicit qualifications for the persons who conduct  the technical assessments, the institute stated that it does explicitly cite  assessor qualifications for its overall national laboratory accreditation  program, adding that this approach to specifying assessor qualifications  has a proven record of success. It also stated that the overall program’s  management manual requires all assessors to meet defined criteria in such  areas as laboratory experience, assessment skills, and technical  knowledge, and that candidate assessors must submit information  addressing each of these areas as well as factors addressing technical  competence in a given laboratory’s focus area (e.g., voting systems).  Further, it stated that candidate assessors’ qualification ratings and  rankings are captured in work sheets.
In response, we do not disagree with any of these statements. However,  our finding is that NIST’s defined approach for VSTL accreditation does  not specify requirements for persons who assess those laboratories that  specifically test voting systems. In this regard, NIST’s own written  comments confirm this, stating that specific requirements for assessors  are not separately documented for each of its national laboratory  accreditation programs, such as the VSTL program. Therefore, we have  not modified this finding or the related recommendation.
Regarding our finding that NIST’s defined approach for accrediting VSTLs  has not always cited the current voting system standards, the institute  affirmed this in its comments by stating that the VSTL program handbook  that it provided to us only cites the 2002 system standards, as these were  the only standards in place when the handbook was published. However,  NIST also noted that when the 2005 system guidelines were adopted in  December 2005, it began the process of updating the handbook and  associated assessment checklist, and that the handbook update was  recently finalized for publication and is now being used.
In response, we stand by our finding that NIST’s defined approach has not  always cited the current voting system standards, which NIST  acknowledges in its comments. However, we also recognize that NIST has  recently addressed this inconsistency by finalizing its new handbook and  the associated assessment checklist. In light of NIST’s recent actions, we  have updated the report to acknowledge the finalization of the handbook  and checklist, and removed the associated recommendation that was  contained in our draft report for NIST to ensure that its defined approach  addresses all required voting system standards.
Regarding our finding that available documentation from completed  accreditations does not show that NIST has consistently followed all  aspects of its defined approach, the institute stated that, among other  things, all required documents for its VSTL accreditation program are  currently in use and reflect the recent update to its handbook and  checklist, and that all these documents are securely maintained.  In response, we do not question these statements; however, they are not  pertinent to our finding. Specifically, our finding is that the four completed  accreditations that we reviewed were not consistently documented. As we  state in our report, we reviewed the documentation associated with the  accreditation assessments for these four laboratories, and we found that  all four were not documented in a similar manner, even though they were  based on the same version of the program handbook. For example, neither  the laboratory notifications of the scope of the assessment nor the next  steps in the accreditation process were consistently documented.  Therefore, we have not modified our finding, but have slightly modified  our recommendation to make it clear that its intent is to ensure that all  phases of the accreditation review are fully and consistently documented.
In its comments, EAC described our review and report as being helpful to  the commission as it works to fully develop and implement its VSTL  program. It also stated that it agrees with the report’s conclusions that  additional written internal procedures, standards, and documentation are  needed to ensure more consistent and repeatable implementation of the  program. The commission added that it generally accepts our  recommendations and will work hard to implement them. To assist it in  doing so, it sought clarification about two of our recommendations, as  discussed below.
EAC stated that the recommendation in our draft report for the  commission to develop specific accreditation steps and criteria was  broadly worded, and thus the recommendation’s intent was not clear. EAC  also stated that it interpreted the recommendation to mean that it should  define internal instructions to guide assessors in performing an  accreditation, and that the recommendation was not intended to have any  impact on its published requirements and procedures governing, for  example, granting, suspending, or withdrawing an accreditation. We agree  with EAC’s interpretation, as it is in line with the intent of our  recommendation. To avoid the potential for any future misunderstanding,  we have modified the wording of the recommendation to clarify its intent.
EAC stated that the recommendation in our draft report for the  commission to develop transparent technical requirements for the  qualifications of its assessors may be confusing because, as we state in our  report, only NIST performs a technical accreditation review, as EAC’s  review is administrative, non-technical in nature. To avoid the potential for  any confusion, we have modified the wording of the recommendation to  eliminate any reference to technical qualification requirements.
We are sending copies of this report to the Ranking Member of the House  Committee on House Administration, the Chairman and Ranking Member  of the Senate Committee on Rules and Administration, the Chairmen and  Ranking Members of the Subcommittees on Financial Services and  General Government, Senate and House Committees on Appropriations,  and the Chairman and Ranking Member of the House Committee on  Oversight and Government Reform. We are also sending copies to the  Chair and Executive Director of EAC, the Secretary of Commerce, the  Deputy Director of NIST, and other interested parties. We will also make  copies available to others on request. In addition, this report will be  available at no charge on the GAO Website at http://www.gao.gov.
Should you or your staffs have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine whether the National Institute of  Standards and Technology (NIST) and the Election Assistance  Commission (EAC) have defined effective voting system testing laboratory  (VSTL) accreditation approaches, and whether each is following its  defined approach.
To determine whether NIST has defined an effective accreditation  approach, we reviewed documentation from its VSTL accreditation  program, such as handbooks and program manuals for the National  Voluntary Laboratory Accreditation Program (NVLAP), of which the VSTL  accreditation program is a part. In doing so, we compared these  documents with applicable statute, guidance, and best practices, primarily  the Help America Vote Act of 2002 (HAVA), internationally recognized  standards from the International Organization for Standardization (ISO),  and federal accreditation program management guidance published by  NIST. We compared program documentation with HAVA’s NIST-specific  accreditation requirements to determine the extent to which the agency  was fulfilling its HAVA responsibilities. We also reviewed program  documentation against ISO/IEC 17011, which establishes general  requirements for accreditation bodies, and ISO/IEC 17025, which  establishes the general requirements for assessing the competence of  laboratories, to determine the extent to which NIST’s accreditation  program was based on internationally recognized standards. We also  compared the documentation against NIST publication NISTIR 6014,  which contains sections that provide guidance for laboratory accreditation  programs, to determine whether the VSTL accreditation program had  defined other elements of effective accreditation programs. We also  interviewed the voting accreditation program manager to determine how  these documents were used to guide the program.
To determine whether NIST has followed its defined approach, we  examined artifacts from the accreditation assessments of five VSTLs,  including one laboratory accredited by NVLAP, but not yet recommended  to EAC. This material included completed assessment checklists derived  from the accreditation program handbooks, additional documents  supporting the assessments, and laboratory accreditation applications and  supporting documentation. We compared artifacts from these assessments  to program guidance to determine the extent to which the defined process  was followed. In addition, we interviewed officials from NIST and NIST  contract assessors and officials from EAC and the four EAC-accredited  VSTLs to understand how the NIST process was implemented and how it  related to the process managed by EAC.
To determine whether EAC has defined an effective accreditation  approach, we reviewed documentation from its VSTL accreditation  program, such as the draft Voting System Test Laboratory Accreditation  Program Manual. In doing so, we compared this document with  applicable statute and best practices, primarily HAVA and federal  accreditation program management guidance published by NIST. We  compared the draft program manual with HAVA’s EAC-specific  accreditation requirements to determine the extent to which the agency  was fulfilling its HAVA responsibilities. We also compared the  documentation against the accreditation guidance in NISTIR 6014 to  determine whether the accreditation program had defined other elements  of effective accreditation programs. We also interviewed the EAC voting  program director and executive director to determine how these  documents were used to guide the program and to understand EAC’s  defined accreditation approach prior to the development of the draft  manual.
To determine whether EAC has followed its defined approach, we  compared artifacts from the accreditation reviews of four VSTLs. We did  not review a fifth laboratory, which had been accredited by NVLAP, but  not yet recommended to EAC. The materials reviewed included checklists  completed by EAC in the absence of an approved program manual. In  doing so, we compared the review artifacts to accreditation program  requirements, as communicated to the laboratories, to determine the  extent to which the agency followed its process, as verbally described to  us. We did not compare accreditation submissions or EAC review artifacts  with the draft accreditation manual because agency officials stated that  the draft manual had not been used in the review of any laboratory. In  addition, we interviewed officials from NIST, EAC, and the four EAC- accredited VSTLs to understand how the EAC process was implemented  and how it related to the process managed by NIST.
To assess data reliability, we reviewed program documentation to  substantiate data provided in interviews with knowledgeable agency  officials. We have also made appropriate attribution indicating the data’s  sources.
We conducted this performance audit at EAC and NIST offices in  Washington, D.C., and Gaithersburg, Maryland, respectively, from  September 2007 to September 2008 in accordance with generally accepted  government auditing standards. Those standards require that we plan and  perform the audit to obtain sufficient, appropriate evidence to provide a  reasonable basis for our findings and conclusions based on our audit  objectives. We believe that the evidence obtained provides a reasonable  basis for our findings and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the National Institute of Standards and Technology
section_title: Appendix III: Comments from the Election Assistance Commission
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: Paula Moore, Assistant Director; Justin Booth; Timothy Case; Neil  Doherty; Timothy Eagle; Nancy Glover; Dave Hinchman; Rebecca LaPaze;  Freda Paintsil; Nik Rapelje; and Jeffrey Woodward made key contributions  to this report.