Elections: Federal Program for Certifying Voting Systems Needs to Be Further Defined, Fully Implemented, and Expanded
Published: 2008-09-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The 2002 Help America Vote Act (HAVA) created the Election Assistance Commission (EAC) and, among other things, assigned the commission responsibility for testing and certifying voting systems. In view of concerns about voting systems and the important role EAC plays in certifying them, GAO was asked to determine whether EAC has (1) defined an effective approach to testing and certifying voting systems, (2) followed its defined approach, and (3) developed an effective mechanism to track problems with certified systems and use the results to improve its approach. To accomplish this, GAO compared EAC guidelines and procedures with applicable statutes, guidance, and best practices, and examined the extent to which they have been implemented.
section_title: What GAO Found
paragraphs: EAC has defined an approach to testing and certifying voting systems that follows a range of relevant practices and statutory requirements associated with a product certification program, including those published by U.S. and international standards organizations, and those reflected in HAVA. EAC, however, has yet to define its approach in sufficient detail to ensure that certification activities are performed thoroughly and consistently. This lack of definition also has caused voting system manufacturers and test laboratories to interpret program requirements differently, and the resultant need to reconcile these differences has contributed to delays in certifying systems that several states were intending to use in the 2008 elections. According to EAC officials, these definitional gaps can be attributed to the program's youth and the commission's limited resources being devoted to other priorities. Nevertheless, they said that they intend to address these gaps, but added that they do not yet have written plans for doing so. EAC has largely followed its defined approach for each of the dozen systems it is in the process of certifying, with one major exception. Specifically, it has not established an effective and efficient repository for certified versions of voting system software, or related procedures and tools, for states and local jurisdictions to use in verifying that their acquired voting systems are identical to what EAC has certified. Further, EAC officials told GAO that they do not have a documented plan or requirements for a permanent solution. As an interim solution, they stated that they will maintain copies of certified versions in file cabinets and mail copies of these versions upon their request by states and local jurisdictions. In GAO's view, this process puts states and local jurisdictions at increased risk of using a version of a system during an election that differs from the certified version. Under its voting system testing and certification program, EAC has broadly described an approach for tracking problems with certified voting systems and using this information to improve its certification program. While this approach is consistent with some aspects of relevant guidance, key elements are either missing or inadequately defined. According to EAC officials, while they intend to address some of these gaps, they do not have documented plans for doing so. In addition, even if EAC defines and implements an effective approach, it would not affect the vast majority of voting systems that are to be used in the 2008 elections. This is because the commission's approach only applies to those voting systems that it has certified, and it is unlikely that any voting systems will be certified in time to be used in the upcoming elections. Moreover, because most states do not currently require EAC certification for their voting systems, it is uncertain if this situation will change relative to future elections. As a result, states and other election jurisdictions are on their own to discover, disclose, and address any shared problems with these noncertified systems.

section_title: Letter
paragraphs: Following the 2000 and 2004 general elections, we issued a series of  reports and testified on virtually every aspect of our nation’s election  system, including the many challenges and opportunities associated with  various types of voting systems. In this regard, we emphasized that the  voting systems alone were neither the sole contributor nor solution to the  problems that were experienced during the 2000 and 2004 elections, and  that the overall election system depended on the effective interplay of  people, process, and technology and involved all levels of government.  Among other things, we specifically reported in 2001 that no federal entity  was responsible for developing voting system standards and for testing  and certifying these systems against such standards, and we raised the  establishment of such an entity as a matter for congressional  consideration.
Subsequently, Congress passed the Help America Vote Act (HAVA), which  created the Election Assistance Commission (EAC) and assigned it  responsibilities for, among other things, the testing, certification,  decertification, and recertification of voting system hardware and  software. In 2004, we testified on the challenges facing EAC in meeting its  responsibilities, adding that the commission’s ability to meet these  challenges depended in part on having adequate resources. In 2007, EAC  established and began implementing its voting system certification  program. In view of the continuing concerns about voting systems and the  important role the commission plays in certifying them, you asked us to  determine whether EAC has (1) defined an effective approach to testing  and certifying voting systems, (2) followed its defined approach, and   (3) developed an effective mechanism to track problems with certified  systems and use the results to improve its certification program.
To accomplish this, we reviewed EAC policies, procedures, and standards  for testing, certifying, decertifying, and recertifying voting systems and  compared them, as appropriate, with applicable statutory requirements  and leading practices, such as HAVA and guidance published by the  National Institute of Standards and Technology (NIST), International  Organization for Standardization (ISO), and International Electrotechnical  Commission (IEC). We also compared EAC actions and artifacts for  executing the voting system certification program with its policies,  guidelines, and procedures. In addition, we interviewed officials from  EAC, NIST, voting system test laboratories (VSTL), voting system  manufacturers, and the National Association of State Election Directors  (NASED).
We conducted this performance audit at EAC offices in Washington, D.C.,  from September 2007 to September 2008 in accordance with generally  accepted government auditing standards. Those standards require that we  plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objectives. Further details of our objectives, scope, and methodology are  included in appendix I.
subsections: section_title: Background
paragraphs: All levels of government share responsibility in the overall U.S. election  system. At the federal level, Congress has authority under the Constitution  to regulate presidential and congressional elections and to enforce  prohibitions against specific discriminatory practices in all federal, state,  and local elections. Congress has passed legislation that addresses voter  registration, absentee voting, accessibility provisions for the elderly and  handicapped, and prohibitions against discriminatory practices. At the  state level, individual states are responsible for the administration of both  federal elections and their own elections. States regulate the election  process, including, for example, the adoption of voluntary voting system  guidelines, the state certification and acceptance testing of voting systems,  ballot access, registration procedures, absentee voting requirements, the  establishment of voting places, the provision of election day workers, and  the counting and certification of the vote.
In total, the overall U.S. election system can be seen as an assemblage of  55 distinct election systems—those of the 50 states, the District of  Columbia, and the 4 U.S. territories. Further, although election policy and  procedures are legislated primarily at the state level, states typically  decentralize election administration, so that it is carried out at the city or  county levels, and voting is done at the local level. As we reported in 2001,  local election jurisdictions number more than 10,000, and their sizes vary  enormously—from a rural county with about 200 voters to a large urban  county, such as Los Angeles County, where the total number of registered  voters for the 2000 elections exceeded the registered voter totals in   41 states. Further, these thousands of jurisdictions rely on many different  types of voting methods that employ a wide range of voting system makes,  models, and versions.
subsections: section_title: The U.S. Election System Depends on Effective Interactions among People, Processes, and Technology
paragraphs: Voting systems are but one facet of a multifaceted, continuous election  system that involves the interplay of people, processes, and technology.  All levels of government, as well as commercial voting system  manufacturers and VSTLs, play key roles in ensuring that voting systems  perform as intended.
Electronic voting systems are typically developed by manufacturers,  purchased as commercial off-the-shelf products, and operated by state and  local election administrators. These activities can be viewed as three  phases in a system’s life cycle: product development, acquisition, and  operations (see fig. 1). Spanning these life cycle phases are key processes,  including managing the interplay of people, processes, and technologies,  and testing the systems and components. In addition, voting system  standards are important through all of these phases because they provide  the criteria for developing, testing, and acquiring the systems, and they  specify the necessary documentation for operating the systems. We  discuss each of these phases after figure 1.
The product development phase includes such activities as establishing  requirements for the system, designing a system architecture, and  developing software and integrating components. Activities in this phase  are performed by the system manufacturer.
The acquisition phase includes such procurement-related activities as  publishing a solicitation, evaluating offers, choosing a voting technology,  choosing a vendor, and awarding and administering contracts. Activities in  this phase are primarily the responsibility of state and local governments,  but include responsibilities that are shared with the vendor, such as  establishing contracts.
The operations phase consists of such activities as ballot design and  programming, setup of systems before voting, pre-election testing, vote  capture and counting during elections, recounts and system audits after  elections, and storage of systems between elections. Responsibility for  activities in this phase typically resides with local jurisdictions, whose  officials may, in turn, rely on or obtain assistance from system vendors for  aspects of these activities.
Standards for voting systems were developed at the national level by the  Federal Election Commission (FEC) in 1990 and 2002 and were updated  by EAC in 2005. Voting system standards serve as guidance for product  developers in building systems, a framework for state and local  governments to evaluate systems, and as the basis for documentation  needed to operate the systems.
Testing processes are conducted throughout the life cycle of a voting  system. For example, manufacturers conduct product testing during  development of the system. Also, national certification testing of products  submitted by system manufacturers is conducted by nationally accredited  VSTLs. States and local jurisdictions also perform a range of system tests.
Management processes help to ensure that each life cycle phase produces  desirable outcomes. Typical management activities include planning,  configuration management, system performance review and evaluation,  problem tracking and correction, human capital management, and user  training.
section_title: Testing and Certification Are Important to Ensuring Voting System Security and Reliability
paragraphs: Testing electronic voting systems for conformance with requirements and  standards is critical to ensuring their security and reliability, and an  essential means to ensuring that systems perform as intended. In addition,  such testing can help find and correct errors in systems before they are  used in elections. If done properly, testing provides voters with assurance  and confidence that their voting systems will perform as intended.
Testing is particularly important for electronic voting systems because  these systems have become our Nation’s predominant method of voting,  and concerns have been raised about their security and reliability. As we  reported in 2005, these concerns include weak security controls, system  design flaws, inadequate system version control, inadequate security  testing, incorrect system configuration, poor security management, and  vague or incomplete voting system standards. Further, security experts  and some election officials have expressed concerns that tests performed  under the NASED program by independent testing authorities and state  and local election officials did not adequately assess voting systems’  security and reliability. Consistent with these concerns, most of the  security weaknesses that we identified in our prior report related to  systems that NASED had previously qualified. Our report also recognized  that security experts and others pointed to these weaknesses as an  indication that both the standards and the NASED testing program were  not rigorous enough with respect to security, and that these concerns  were amplified by what some described as a lack of transparency in the  testing process.
section_title: EAC’s Responsibilities under HAVA include Certifying Voting Systems
paragraphs: Enacted in October 2002, HAVA affects nearly every aspect of the election  system, from voting technology to provisional ballots and from voter  registration to poll worker training. Among other things, the act  authorized $3.86 billion in funding over several fiscal years for states to  replace punch card and mechanical lever voting equipment, improve  election administration and accessibility, and perform research and pilot  studies. In addition, the act established EAC and assigned it responsibility  for, among other things, (1) updating voting system standards, (2) serving  as a clearinghouse for election-related information, (3) accrediting  independent test laboratories, and (4) certifying voting systems. EAC  began operations in January 2004. In 2004, we testified on the challenges  facing EAC in meeting its responsibilities. For example, we reported that  EAC needed to move swiftly to strengthen voting system standards and  the testing associated with these standards. We also reported that the  commission’s ability to meet its responsibilities depended, in part, on the  adequacy of the resources at its disposal.
Updating standards: HAVA requires EAC to adopt a set of federal voting  system standards, referred to as the Voluntary Voting System Guidelines  (VVSG). In December 2005, the commission adopted the VVSG, which  defines a set of specifications and requirements against which voting  systems are to be designed, developed, and tested to ensure that they  provide the functionality, accessibility, and security capabilities required  to ensure the integrity of voting systems. As such, the VVSG specifies the  functional requirements, performance characteristics, documentation  requirements, and test evaluation criteria for the national certification of  voting systems. In 2007, the Technical Guidelines Development  Committee submitted its recommendations for the next iteration of the  VVSG to EAC. The commission has yet to establish a date when the update  will be approved and issued.
Serving as an information clearinghouse: HAVA requires EAC to  maintain a clearinghouse of information on the experiences of state and  local governments relative to, among other things, implementing the VVSG  and operating voting systems. As part of this responsibility, EAC posts  voting system reports and studies that have been conducted or  commissioned by a state or local government on its Web site. These  reports must be submitted by a state or local government that certifies that  the report reflects its experience in operating a voting system or  implementing the VVSG. EAC does not review the information for quality  and does not endorse the reports and studies.
Accrediting independent test laboratories: HAVA assigned responsibilities  for laboratory accreditation to both EAC and NIST. In general, NIST  focuses on assessing laboratory technical qualifications and recommends  laboratories to EAC for accreditation. EAC uses NIST’s assessment results  and recommendations, and augments them with its own review of related  laboratory capabilities to reach an accreditation decision. As we have  previously reported, EAC and NIST have defined their respective  approaches to accrediting laboratories that address relevant HAVA  requirements. However, neither approach adequately defines all aspects of  an effective program to the degree needed to ensure that laboratories are  accredited in a consistent and verifiable manner. Accordingly, we recently  made recommendations to NIST and EAC aimed at addressing these  limitations.
Certifying voting systems: HAVA requires EAC to provide for the testing,  certification, decertification, and recertification of voting system hardware  and software. EAC’s voting system testing and certification program is  described in detail in the following section.
Prior to HAVA, no federal agency was assigned or assumed responsibility  for testing and certifying voting systems against the federal standards.  Instead, NASED, through its voting systems committee, assumed this  responsibility by accrediting independent test authorities, which in turn  tested equipment against the standards. When testing was successfully  completed, the independent test authorities notified NASED that the  equipment satisfied testing requirements. NASED would then qualify the  system for use in elections. According to a NASED official, the committee  has neither qualified any new or modified systems, nor taken any actions  to disqualify noncompliant systems, since the inception of EAC’s testing  and certification program in January 2007.
section_title: Overview of EAC’s Voting System Testing and Certification Program
paragraphs: EAC implemented its voting system testing and certification program in  January 2007. According to the commission’s Testing and Certification  Program Manual, EAC certification means that a voting system has been  successfully tested by an accredited VSTL, meets requirements set forth in  a specific set of federal voting system standards, and performs according  to the manufacturer’s specifications.
The process of EAC’s voting system testing and certification program  consists of seven major phases. Key stakeholders that are involved in this  process include voting system manufacturers, accredited VSTLs, and state  and local election officials. These seven phases are described in the  following text and depicted in figure 2.
subsections: section_title: 1. Manufacturer Registration
paragraphs: All manufacturers must be registered to submit a voting system for  certification. To register, a manufacturer must provide such information as  organizational structure and contact(s); quality assurance, configuration  management, and document retention procedures; and identification of all  manufacturing and assembly facilities. In registering, the manufacturer  agrees to certain duties and requirements at the outset of its participation  in the program. These requirements include properly using and  representing EAC’s certification label, notifying the commission of any  changes to a certified system, permitting EAC to verify the manufacturer’s  quality control procedures by inspecting fielded systems and  manufacturing facilities, cooperating with any inquiries and investigations  about certified systems, reporting any known malfunction of a system, and  otherwise adhering to all procedural requirements of the program manual.
Once a manufacturer submits a completed application form and all  required attachments, EAC reviews the submission for sufficiency using a  checklist that maps to the application requirements listed in the program  manual. If the application passes the review, EAC provides the  manufacturer with a unique identification code and posts the applicant as  a registered manufacturer on the commission’s Web site, along with  relevant documentation.
section_title: 2. Voting System Certification Application
paragraphs: For each voting system that a manufacturer wishes to have certified, it  submits an application package. The package includes an application form  requiring the following: manufacturer information, accredited VSTL  selection, applicable voting system standard(s), nature of submission,  system name and version number, all system components and  corresponding version numbers, and system configuration information.  The package also includes the following documentation: system  implementation statement, functional diagram, and system overview.  EAC reviews the submission for completeness and accuracy and, if it is  acceptable, notifies the manufacturer and assigns a unique application  number to the system.
section_title: 3. Test Plan Development and Approval
paragraphs: Once the certification application is accepted, the accredited VSTL  prepares and submits to EAC a test plan defining how it will ensure that  the system meets applicable standards and functions as intended. When a  laboratory submits its test plan, EAC’s technical reviewers assess it for  adequacy. If the plan is deemed not acceptable, the commission provides  written notice to the laboratory that includes a description of the problems  identified and the steps required to remedy the test plan. The laboratory  may take remedial action and resubmit the test plan until it is accepted by  EAC reviewers.
section_title: 4. Test Plan Execution
paragraphs: The VSTL executes the approved test plan and notifies EAC directly of any  test anomalies or failures, along with any changes or modifications to the  test plan as a result of testing. The laboratory then prepares a test results  report.
section_title: 5. Test Report Review
paragraphs: The VSTL submits the test results report to EAC’s Program Director who  reviews it for completeness. If it is complete, the technical reviewers  analyze the report in conjunction with related technical documents and  the test plan for completeness, appropriateness, and adequacy. The  reviewers submit their findings to the Program Director, who either  recommends certification of the system to the Decision Authority, EAC’s  Executive Director, or refers the matter back to the reviewers for  additional specified action and resubmission.
section_title: 6. Initial Certification Decision and Appeal
paragraphs: EAC’s Decision Authority reviews the recommendation of the Program  Director and supporting materials and issues a written decision to the  manufacturer. If certification is denied, the manufacturer may request an  opportunity to correct the basis for the denial or may request  reconsideration of the decision after submitting supporting written  materials, data, and a rationale for its position. The Decision Authority  considers the request and issues a written decision. If the decision is to  deny certification, the manufacturer may request an appeal in writing to  the Program Director. The Appeal Authority, which consists of two or  more EAC Commissioners or other individuals appointed by the  Commissioners who have not previously served as the initial or  reconsideration authority, consider the appeal. The Appeal Authority may  overturn the decision if it finds that the manufacturer has demonstrated by  clear and convincing evidence that its system met all substantive and  procedural requirements for certification.
section_title: 7. Final Certification
paragraphs: The initial decision becomes final and EAC issues a Certificate of  Conformance to the manufacturer, and posts the system on the list of  certified voting systems on its Web site, when the manufacturer and VSTL  successfully demonstrate that the voting system under test has been:    Subject to a trusted build: The voting system’s source code is converted to  executable code in the presence of at least one VSTL representative and  one manufacturer representative, using security measures to ensure that  the executable code is a verifiable and faithful representation of the  source code. This demonstrates that (1) the software was built as  described in the technical documentation, (2) the tested and approved  source code was actually used to build the executable code on the system,  and (3) no other elements were introduced in the software build. It also  serves to document the configuration of the certified system for future  reference.
Placed in a software repository: The VSTL delivers the following to one or  more trusted repositories designated by EAC: (1) source code used for the  trusted build and its file signatures; (2) disk image of the prebuild, build  environment, and any file signatures to validate that it is unmodified;   (3) disk image of the postbuild, build environment, and any file signatures  to validate that it is unmodified; (4) executable code produced by the  trusted build and its file signatures of all files produced; and   (5) installation device(s) and its file signatures.
Verified using system identification tools: The manufacturer creates and  makes available system identification tools that federal, state, and local  officials can use to verify that their voting systems are unmodified from  the system that was certified. These tools are to provide the means to  identify and verify hardware and software.
section_title: EAC’s Approach Largely Follows Many Accepted Practices, but Needs to Be Further Defined
paragraphs: To its credit, EAC has taken steps to develop an approach to testing and  certifying voting systems that follows statutory requirements and many  recognized and accepted practices. However, the commission has not  developed its approach in sufficient detail to ensure that its certification  activities are performed thoroughly and consistently. It has not, for  example, defined procedures or specific criteria for many of its review  activities, and for ensuring that the decisions made, and their basis, are  properly documented. According to EAC officials, these gaps exist  because the program is still new and evolving and resources are limited.  Officials further stated that they do not yet have written plans for  addressing these gaps. Until these gaps are addressed, EAC cannot  adequately ensure that its approach is repeatable and verifiable across all  manufacturers and systems. Moreover, this lack of definition has caused  EAC stakeholders to interpret certification requirements differently, and  the resultant need to reconcile these differences has contributed to delays  in certifying systems that several states were planning on using in the 2008  elections.
subsections: section_title: EAC’s Approach Reflects Key Practices and Statutory Requirements
paragraphs: Product certification or conformance testing is a means by which a third  party provides assurance that a product conforms to specific standards. In  the voting environment, EAC is the third party that provides assurance to  the buyer (e.g., state or local jurisdictions) that the manufacturer’s voting  system conforms to the federal voting standards set forth in FEC’s 2002  Voting System Standards (VSS) or EAC’s 2005 VVSG.
Several organizations, such as NIST, ISO, and IEC, have individually or  jointly developed guidance for product certification and conformance  testing programs. This guidance includes, among other things,   (1) defining roles and responsibilities for all parties involved in the  certification process, (2) defining a clear and transparent process for  applicants to follow, (3) ensuring that persons involved in the process are  impartial and independent, (4) establishing a process for handling  complaints and appeals, and (5) having testing conducted by competent  laboratories. Further, HAVA established statutory requirements for a  federal testing and certification program. These requirements include  ensuring that the program covers testing, certification, decertification, and  recertification of voting system hardware and software.
EAC’s defined voting system certification approach reflects these key  practices. Specifically:    EAC has defined the roles and responsibilities for itself, the VSTLs, and  manufacturers in its Testing and Certification Program Manual. These  roles and responsibilities are described in table 1.
EAC’s testing and certification process is documented in its program  manual. Among other things, the manual clearly defines the program’s  administrative requirements that manufacturers and VSTLs are to follow.  EAC has made the program manual, along with supporting policies and  clarifications, publicly available on its Web site, and has made program- related news and correspondence publicly accessible as they have come  available.
EAC’s certification program addresses impartiality and independence. For  example, EAC policy states that all personnel and contractors involved in  the certification program are subject to conflict-of-interest reporting and  review. In addition, the policy mandates conflict-of-interest and conduct  statements for the technical reviewers that support the program, requires  conflict-of-interest reporting and reviews to ensure the independence of  EAC personnel assigned to the program, and requires that all VSTLs  maintain and enforce policies that prevent conflict-of-interest or the  appearance of a conflict-of-interest, or other prohibited practices.
EAC’s program manual outlines its process for the resolution of  complaints, appeals, and disputes received from manufacturers and  laboratories. These can be about matters relating to the certification  process, such as test methods, procedures, test results, or program  administration. Specifically, the program manual contains policies and  procedures for submitting a Request for Interpretation, which is a means  by which a registered manufacturer or accredited laboratory seeks  clarification on a specific voting system standard, including any  misunderstandings or disputes about a standard’s interpretation or  implementation. The manual also contains policies, requirements, and  procedures for a manufacturer to file an appeal on a decision denying  certification, request an opportunity to correct a problem, and request  reconsideration of a decision. In addition, EAC provides for Notices of  Clarification, which offer guidance and explanation on the requirement  and procedure of the program. Notices may be issued pursuant to a  clarification request from a laboratory or manufacturer. EAC may also  issue a notice or interpretation if it determines that any general  clarifications are necessary.
EAC has a VSTL accreditation program. This program is supported by  NIST’s National Voluntary Laboratory Accreditation Program (NVLAP),  which is a long established and recognized laboratory accreditation  program. According to EAC’s program manual, all certification testing is to  be performed by a laboratory accredited by NIST and EAC. Further, all  subcontracted testing is to be performed by a laboratory accredited by  either NIST or the American Association of Laboratory Accreditation for  the specific scope of needed testing. Finally, any prior testing will only be  accepted if it was conducted or overseen by an accredited laboratory and  was reviewed and approved by EAC.
HAVA also established certain requirements for EAC’s voting system  testing and certification program. Under HAVA, EAC is to provide for the  testing, certification, decertification, and recertification of voting system  hardware and software by accredited laboratories. EAC’s defined  approach addresses each of these areas.
According to program officials, EAC’s certification program reflects many  leading practices because the commission consciously sought out these  best practices during program development. Officials stated that their  intention is to develop a program that stringently tests voting systems to  the applicable standards; therefore, they consulted with experts to assist  with drafting the program manual. For example, according to EAC  officials, they met with officials from other federal agencies that conduct  certification testing in order to benefit from their lessons learned. By  reflecting relevant practices, standards, and legislative requirements in its  defined approach, EAC has provided an important foundation for having  an effective voting system testing and certification program.
section_title: EAC’s Approach Does Not Sufficiently Define the Details for Certifications to Be Performed Thoroughly and Consistently
paragraphs: EAC has yet to define its approach for testing and certifying electronic  voting systems in sufficient detail to ensure that its certification activities  are performed thoroughly and consistently. It has not, for example,  defined procedures or specific criteria for many of its review activities and  for ensuring that the decisions made are properly documented. EAC  officials attributed this lack of definition to the fact that the program is  still new and evolving, and they stated that available resources are  constrained by competing priorities. Until these details are defined, EAC  will be challenged to ensure that testing and review activities are  repeatable across different systems and manufacturers, and that the  activities it performs are verifiable. Moreover, this lack of definition is  likely to result in different interpretations of program requirements by  stakeholders, which has already resulted in the need to reconcile different  interpretations and thereby caused delays in certifying systems that  several states intended to use in the 2008 elections. In such cases, the  delays are forcing states to either not require EAC certification or rely on  an alternative system.
According to federal and international guidance, having well-defined and  sufficiently detailed program management controls help to ensure that  programs are executed effectively and efficiently. Relative to a testing and  certification program, such management controls include, among other  things, having (1) defined procedures and established criteria for  performing evaluation activities so that they will be performed in a  comparable, unambiguous, and repeatable manner for each system and (2)  required documentation to demonstrate that procedural evaluation steps  and related decisions have been effectively performed, including  provisions for review and approval of such documentation by authorized  personnel.
EAC’s defined approach for voting system testing and certification lacks  such detail and definition. With respect to the first management control,  the commission has not defined procedures or specific criteria for many of  its review activities, instead it relies on the personal judgment of the  reviewers. Specifically, the program manual states that EAC, with the  assistance of its technical experts, as necessary, will review manufacturer  registration applications, system certification applications, test plans, and  test reports, but it does not define procedures or criteria for conducting  these reviews. For example:    The program manual states that upon receipt of a completed manufacturer  registration application, EAC will review the information for sufficiency.  However, it does not define what constitutes sufficiency and what this  sufficiency review should entail. Rather, EAC officials said that this is left  up to the individual reviewer’s judgment.
The program manual lists the information that manufacturers are required  to submit as part of their certification applications and states that EAC will  review the submission for completeness and accuracy. While the  commission has developed a checklist for determining whether the  required information was included in the application, neither the program  manual nor the checklist describe how reviewers should perform the  review or assess the adequacy of the information provided. For example,  EAC requires certification applications to include a functional diagram  depicting how the components for the voting system function and interact,  as well as a system overview that includes a description of the functional  and physical interfaces between components. Although the checklist  provides for determining whether these items are part of the application  package, it does not, for example, provide for checking them for  completeness and consistency. Moreover, we identified issues with  completeness and consistency of these documents for approved  certification application packages. Again, EAC officials said that these  determinations are to be based on each reviewer’s judgment.
The program manual states that test plans are to be reviewed for  adequacy. However, it does not define adequacy or how such reviews are  to be performed. This lack of detail is particularly problematic because  EAC officials told us that the VSS and VVSG contain many vague and  undefined requirements. According to these officials, reviewers have been  directed to ensure that VSTLs stress voting systems during testing, based  on what they believe are the most difficult and stringent conditions likely  to be encountered in an election environment that are permissible under  the standards.
The program manual states that EAC technical experts will assess test  results reports for completeness, appropriateness, and adequacy.  However, it does not define appropriateness and adequacy or the  procedural steps for conducting the review.
The program manual requires VSTLs to use all applicable test suites issued  by EAC when developing test plans. However, program officials stated that  they currently do not have defined test suites and NIST officials said that  they are focused on preparing test suites for the forthcoming version of  the VVSG and not the 2005 version. As a result, each laboratory develops  its own unique testing approach, which requires each to interpret what is  needed to test for compliance with the standards and increases the risk of  considerable variability in how testing is performed. To address this void,  EAC has tasked NIST with developing test suites for both the 2005 VVSG  and the yet-to-be-released update to these guidelines. Until then, EAC  officials acknowledge that they will be challenged to ensure that testing  conducted on different systems or at different VSTLs is consistent and  comparable.
With respect to the second management control, the commission has not  defined the documentation requirements that would demonstrate that  procedural steps, evaluations, and related decision making have been  performed in a thorough, consistent, and verifiable manner. Specifically,  while the program manual requires the Program Director to maintain  documentation to demonstrate that procedures and evaluations were  effectively performed, EAC has yet to specify the nature or content of this  documentation. For example:    The program manual requires technical reviewers to assess test plans and  test reports prepared by laboratories and then to submit reports of their  findings to the Program Director. However, EAC does not require  documentation of how these reviews were performed beyond completion  of a recently developed checklist, and this checklist does not provide for  capturing how decisions were reached, including steps performed and  criteria applied. For example, the VVSG requires that systems permit  authorized access and prevent unauthorized access, and lists examples of  measures to accomplish this, such as computer-generated password keys  and controlled access security. While the checklist cites this requirement  and provides for the reviewer to indicate whether the test plan satisfies it,  it does not provide specific guidance on how to determine whether the  access control measures are adequate, and it does not provide for  documenting how the reviewer made such a decision.
The program manual does not require supervisory review of work  conducted by EAC personnel. Moreover, it does not require that the  reviewers be identified.
According to EAC officials, its approach does not yet include such details  because it is still new and evolving and because the commission’s limited  resources have been devoted to other priorities. To address these gaps,  EAC officials stated that they intend to undertake a number of activities.  For example, the Program Director stated that in the near-term, the  technical reviewers will collaborate and share views on each test plan and  test report under review as a way to provide consistency, and they will use  a recently finalized checklist for test plan and test report reviews. In the  longer term, EAC intends to define more detailed procedures for each step  in its process. However, it has yet to establish documented plans,  including the level of resources needed to accomplish this. Until these  plans are developed and executed, EAC will be challenged to ensure that  its testing and certification activities are performed thoroughly and  consistently across different systems, manufacturers, and VSTLs.
Moreover, this lack of program definition surrounding certification testing  and review requirements has already caused, and could continue to cause,  differences in how EAC reviewers, VSTLs, and manufacturers interpret the  requirements. For example, the program requires sufficient and adequate  testing, but it does not define what constitutes sufficient and adequate. As  a result, laboratory officials and manufacturer representatives told us that  EAC reviewers have interpreted these requirements more stringently than  they have, and that reconciling these different interpretations has already  caused delays in the approval of test plans, and they will likely prevent  EAC from certifying any systems in time for use in the upcoming 2008  elections. This is especially problematic for those states that have  statutory or other requirements to use federally certified systems and that  have a need to acquire or upgrade existing systems for these elections. In  this regard, 18 states reported to us that they were relying on EAC to  certify systems for use in the 2008 elections, but now will have to adopt  different strategies for meeting their states’ respective EAC certification  requirements. For example, officials for several states said that they would  use the same system as in 2006, while officials for other states described  plans to either undertake upgrades to existing systems without federal  certification, or change state requirements to no longer require EAC  certification.
section_title: EAC Has Largely Followed Its Defined Certification Approach, but a Key System Verification Capability to Support States and Local Jurisdictions Is Not Yet in Place
paragraphs: EAC has largely followed its defined certification approach for each of the  dozen voting systems that it is in the process of certifying, with one major  exception. Specifically, it has not established sufficient means for states  and local jurisdictions to verify that the voting systems that each receives  from its manufacturer have system configurations that are identical to  those of the system that the EAC certified, and it has not established plans  or time frames for doing so. This means that states and local jurisdictions  are at increased risk of using a version of a system during an election that  differs from the certified version. This lack of an effective and efficient  verification capability could diminish the value of an EAC system  certification.
subsections: section_title: EAC Has Largely Followed Its Defined Certification Approach
paragraphs: EAC has largely executed its voting system testing and certification  program as defined. While no system has yet to complete all major steps of  the certification process discussed in the Background section of this  report, and thus receive certification, 12 different voting systems have  completed at least the first step of the process, and some have completed  several steps. Specifically, as of May 2008, EAC had received, reviewed,  and approved 12 manufacturer registration applications, and these  approved manufacturers have collectively submitted certification  applications for 12 different systems, of which EAC has accepted 9. For  these 9 systems, manufacturers have submitted 7 test plans, of which EAC  has reviewed and approved 2 plans. In 1 of these 2 cases, EAC has  received a test results report, which it is currently in the process of  reviewing. At the same time, EAC has responded to 8 requests for  interpretations of standards from manufacturers and laboratories. EAC  has also issued 6 notices of clarification, which provide further guidance  and explanation on the program requirements and procedures.
Our analysis of available certification-related documentation showed that  in each of the 12 systems submitted for certification, all elements of each  executed step in the certification process were followed. With respect to  the manufacturer registration step, EAC reviewed and approved all   12 applications, as specified in its program manual. For the certification  application step, EAC reviewed and approved 9 applications. In doing so,  EAC issued 3 notices of noncompliance to manufacturers for failure to  comply with program requirements. In each notice, it identified the area of  noncompliance, and described what requested relevant information or  corrective action(s) was needed in order to participate in the program. In   1 of the cases, EAC terminated the certification application due to the  manufacturer’s failure to respond within the established time frame, which  is consistent with the program manual. These actions were generally  consistent with its program manual.
section_title: EAC Has Not Established a Sufficient Means for States and Local Jurisdictions to Verify That Their Systems Match the Certified Version
paragraphs: Notwithstanding EAC’s efforts to follow its defined approach, it has not  yet established a sufficient mechanism for states and local jurisdictions to  use in verifying that the voting systems that they receive from  manufacturers for use in elections are identical to the systems that were  actually tested and certified. According to EAC’s certification program  manual, final certification is conditional upon (1) testing laboratories  depositing certified voting system software into an EAC-designated  repository and (2) manufacturers creating and making available system  identification tools for states and local jurisdictions to use in verifying that  their respective systems’ software configurations match that of the  software for the system that was certified and deposited into the  repository. However, EAC has yet to establish a designated repository or  procedures and review criteria for evaluating the manufacturer-provided  tools, and has not established plans or time frames for doing so. While  none of the ongoing system certifications have progressed to the point  where these aspects of EAC’s defined approach is applicable, they will be  needed when the first system’s test results are approved. Until both  aspects are in place, state and local officials will likely face difficulties in  determining whether the systems that they receive from manufacturers are  the same as the systems that EAC certified.
EAC’s program requires the use of a designated software repository for  certified voting systems. Specifically, the certification program manual  states that final certification will be conditional upon, among other things,  the manufacturer and VSTL creating and documenting a trusted software  build, and the laboratory depositing the build in a designated repository.  In its 2005 VVSG, EAC designated NIST’s National Software Reference  Library (NSRL) as its repository and required its use. However, program  officials stated that the commission does not intend to use the NSRL as its  designated repository because the library cannot perform all required  functions. While these officials added that they may use the NSRL for  portions of the certification program, they said it will not serve as EAC’s  main repository.
Nevertheless, the commission has not established plans to identify and  designate another repository, and has yet to define minimum requirements  (functional, performance, or interface) for what it requires in a repository  to efficiently support states and local jurisdictions. As an interim measure,  an EAC official stated that they will store the trusted builds on compact  disks and keep the disks in fireproof filing cabinets in their offices.  According to the official, this approach is consistent with established  program requirements because the program manual merely refers to the  use of a trusted archive or repository designated by EAC. Under this  measure, state and local election officials will have to request physical  copies of material from EAC and wait for the materials to be physically  packaged and delivered. This interim approach is problematic for several  reasons, including the demand for EAC resources to keep up with the  requests during a time when the Executive Director told us that a more  permanent repository solution has not been a commission focus because  its limited resources have been focused on other priorities.
EAC has also not defined how it will ensure that manufacturers develop  and provide to states and local jurisdictions tools to verify their respective  systems’ software against the related trusted software builds. According  to the program manual, final certification of a voting system is also  conditional upon manufacturers creating and making available to states  and local jurisdictions tools to compare their systems’ software with the  trusted build in the repository. In doing so, the program manual states that  manufacturers shall develop and make available tools of their choice, and  that the manufacturer must submit a letter certifying the creation of the  tools and include a copy and description of the tools to EAC. The  commission may choose to review the tools. Further, the 2005 VVSG  provides some requirements for software verification tools, for example,  that the tools provide a method to comprehensively list all software files  that are installed on the voting system. However, EAC has yet to specify  exactly what needs to be done to ensure that manufacturers provide  effective and efficient system identification tools and processes, and it has  not developed plans for ensuring that this occurs. Instead, EAC has stated  that until it defines procedures to supplement the program manual, it will  review each tool submitted. However, the commission has yet to establish  specific criteria for the assessment.
Without an established means for effectively and efficiently verifying that  acquired systems have the same system configurations as the version that  EAC certified, states and local jurisdictions will not know whether they  are using federally certified voting systems. The absence of such tools  unnecessarily increases the risk of a system bearing EAC’s mark of  certification differing from the certified version.
section_title: EAC’s Mechanism for Tracking and Resolving Problems with Certified Voting Systems Lacks Sufficient Definition, and Its Scope Is Limited
paragraphs: As part of its voting system testing and certification program, EAC has  broadly described an approach for tracking and resolving problems with  certified voting systems, and using the information about these problems  to improve its program. This approach reflects some key aspects of  relevant guidance. However, other aspects are either missing or not  adequately defined, and although EAC officials stated that they intend to  address some of these gaps, the commission does not have defined plans  or time frames for doing so. Commission officials cited limited resources  and competing priorities as reasons for these gaps.
In addition, EAC’s problem tracking and resolution approach does not  extend to any of the voting systems that are likely to be used in the 2008  elections, and it is uncertain when, and to what extent, this situation will  change. This is because its defined scope only includes EAC-certified  systems; it does not include NASED-qualified systems or any other  systems to be used in elections. EAC officials stated that the reason for  this is because HAVA does not explicitly assign the commission  responsibility for systems other than those it certifies. This means that no  federal entity is currently responsible for tracking and facilitating the  resolution of problems found with the vast majority of voting systems that  are used across the country today and that could be used in the future, and  thus states and local jurisdictions must deal with problems with their  systems on their own.
subsections: section_title: EAC’s Approach for Tracking and Resolving Problems with Certified Systems Partially Reflects Relevant Guidance
paragraphs: According to published guidance, tracking and resolving problems with  certified products is important. This includes, among other things,   (1) withdrawing certification if a product becomes noncompliant;   (2) regularly monitoring the continued compliance of products being  produced and distributed; (3) investigating the validity and scope of  reports of noncompliance; (4) requiring the manufacturer to take  corrective actions when defects are discovered, and ensuring that such  actions are taken; and (5) using information gathered from these activities  to improve the certification program.
EAC’s approach for tracking and resolving problems with certified  systems reflects some, but not all aspects of these five practices. First, its  certification program includes provisions for withdrawing certification for  noncompliant voting systems. For example, the program manual describes  procedures for decertifying a noncompliant voting system if the  manufacturer does not take timely or sufficient action to correct instances  of noncompliance. According to these procedures, the decertification  decision cannot be made until the manufacturer is formally alerted to the  noncompliance issue and provided with an opportunity to correct the  issue (problem) or to submit additional information for consideration.  Also, a manufacturer can dispute the decision by requesting an appeal. The  procedures also state that upon decertification, the manufacturer cannot  represent the system as certified and the system may not be labeled with a  mark of certification. In addition, EAC is to remove the system from its list  of certified systems, alert state and local election officials to the system’s  decertification via monthly newsletters and e-mail updates, and post all  correspondence regarding the decertification on its Web site.
Second, EAC’s certification program includes provisions for  postcertification oversight of voting systems and manufacturers.  Specifically, the program manual provides for reviewing and retesting  certified voting systems to ensure that they have not been modified, and  that they continue to comply with applicable standards and program  requirements. In addition, the program manual calls for periodic  inspections of manufacturers’ facilities to evaluate their production  quality, internal test procedures, and overall compliance with program  requirements. However, the program manual states that reviewing and  retesting of certified systems is an optional step, and does not specify the  conditions under which this option is to be exercised. Further, while the  program manual provides for conducting periodic inspections of  manufacturers’ facilities, it does not define, for example, who is to conduct  the inspections, what procedures and evaluation criteria are to be used in  conducting them, and how they are to be documented.
Third, EAC’s certification program includes provisions for investigating  reports of system defects. According to the program manual,  investigations of reports alleging defects with certified systems begin as  informal inquiries that can potentially become formal investigations.  Specifically, the Program Director is to conduct an informal inquiry in  which a determination is made regarding whether the reported defect  information is both credible and deserving of system decertification if  found to be credible. If both conditions are met, then a formal  investigation is to be conducted. Depending on the outcome,  decertification of the system could result. However, the program manual  does not call for assessing the scope or impact of any defects identified,  such as whether a defect is confined to an individual unit or whether it  applies to all such units. Further, the program manual does not include  procedures or criteria for determining the credibility of reported defects,  or any other aspect of the inquiry or investigation, such as how EAC will  gain access to systems once they are purchased and fielded by states and  local jurisdictions. This is particularly important because EAC does not  have regulatory authority over state election authorities, and thus, it  cannot compel their cooperation during an inquiry or investigation.
Fourth, EAC’s certification program does not address how it will verify  that manufacturers take required corrective actions to fix problems  identified with certified systems. Specifically, the program manual states  that the manufacturer is to provide EAC with a compliance plan describing  how it will address identified defects. However, the program manual does  not define an approach for evaluating the compliance plan and confirming  that a manufacturer actually implements the plan. According to EAC  officials, they see their role as certifying a system and informing states of  modifications. As a result, they do not intend to monitor how or whether  manufacturers implement changes to fielded systems. In their view, the  state ultimately decides if a system will be fielded.
Fifth, EAC’s certification program provides for using information  generated by its problem tracking and resolution activities to improve the  program. According to the program manual, information gathered during  quality monitoring activities will be used to, among other things, identify  improvements to the certification process and to inform the related  standards-setting process. Further, the program manual states that  information gathered from these activities will be used to inform relevant  stakeholders of issues associated with operating a voting system in a real- world environment and to share information with jurisdictions that use  similar systems. However, the program manual does not describe how  EAC will compile and analyze the information gathered to improve the  program, or how it will coordinate these functions with information  gathered in performing its HAVA-assigned clearinghouse function.
EAC officials attributed the state of their problem tracking and resolution  approach to the newness of the certification program and the fact that the  commission’s limited resources have been devoted to other priorities. In  addition, while these officials said that they intend to address some of  these gaps, they do not have defined plans or time frames for doing so. For  example, while EAC officials stated that they plan to develop procedures  for investigating voting system problems and for inspecting manufacturing  facilities, they said that it is the states’ responsibility to ensure that  corrective actions are implemented on fielded systems. To illustrate their  resource challenges, these officials told us that three staff are assigned to  the testing and certification program and each is also supporting other  programs. In addition, they said that the commission’s technical  reviewers are experts who, under Office of Personnel Management  regulation, work no more than one-half of the time of the year.
Given that EAC has not yet certified a system, the impact of these  definitional limitations has yet to be realized. Nevertheless, with   12 systems currently undergoing certification, it is important for the  commission to address them quickly. If it does not, EAC will be challenged  in its ability to effectively track and resolve problems with the systems  that it certifies.
section_title: EAC’s Problem Tracking and Resolution Efforts Do Not Include Most Voting Systems to Be Used in Elections
paragraphs: The scope of EAC’s efforts to track and resolve problems with certified  voting systems does not extend to those systems that were either qualified  by NASED or were not endorsed by any national authority. According to  program officials, the commission does not have the authority or the  resources needed to undertake such a responsibility. Instead of tracking  and resolving problems with these systems, EAC anticipates that they will  eventually be replaced or upgraded with certified systems. Our review of  HAVA confirmed that the act does not explicitly assign EAC any  responsibilities for noncertified systems, although it also does not  preclude EAC from tracking and facilitating the resolution of problems  with these systems.
As a result, the commission’s efforts to track and resolve problems with  voting systems do not include most of the voting systems that will be used  in the 2008 elections. More specifically, while EAC has efforts under way  relative to the certification of 12 voting systems, as we have previously  described in this report, commission officials stated that it will be difficult  to field any system that EAC anticipates certifying before 2008 in time for  the 2008 elections. Thus, voting systems used in national elections will  likely be either those qualified under the now-discontinued NASED  program, or those not endorsed by any national entity. Moreover, this will  continue to be the case until states voluntarily begin to adopt EAC- certified systems, which is currently unclear and uncertain because only  18 states reported having requirements to use EAC-certified voting  systems. Restated, most states’ voting systems will not be covered by  EAC’s problem tracking and resolution efforts, and when and if they will is  not known. Moreover, manufacturers may or may not upgrade existing,  noncertified systems, and they may or may not seek EAC certification of  those systems. Thus, it is likely that many states, and their millions of  voters, will not use EAC-certified voting systems for the foreseeable  future.
Nevertheless, EAC has initiated efforts under the auspices of its HAVA- assigned clearinghouse responsibility to receive information that is  volunteered by states and local jurisdictions on problems and experiences  with systems that it has not certified, and to post this information on the  commission’s Web site to inform other states and jurisdictions about the  problems. In doing so, EAC’s Web site states that the commission does not  review the information for quality and does not endorse the reports and  studies. Notwithstanding this clearinghouse activity, this means that no  national entity is currently responsible for tracking and facilitating the  resolution of problems found with the vast majority of voting systems that  are in use across the country. This in turn leaves state and local  jurisdictions on their own to discover, disclose, and address any shared  problems with systems. While this increases the chances of states and  local jurisdictions duplicating efforts to get problems fixed, it also  increases the chances that problems addressed by one state or jurisdiction  may not even be known to another. A key to overcoming this situation will  be strong central leadership.
section_title: Conclusions
paragraphs: The effectiveness of our nation’s overall election system depends on many  interrelated and interdependent variables. Among these are the security  and reliability of the voting systems that are used to cast and count votes,  which in turn depend largely on the effectiveness with which these  systems are tested and certified. EAC plays a pivotal role in testing and  certifying voting systems. To its credit, EAC has recently established and  begun implementing a voting system testing and certification program that  is to both improve the quality of voting systems in use across the country,  and help foster public confidence in the electoral process. While EAC has  made important progress in defining and executing its program, more can  be done. Specifically, key elements of its defined approach, such as the  extent to which certification activities are to be documented, are vague,  while other elements are wholly undefined—such as threshold criteria for  making certification-related decisions. Moreover, a key element that is  defined—namely, giving states and local jurisdictions an effective and  efficient means to access the certified version of a given voting system  software—has yet to be implemented. While EAC acknowledges the need  to address these gaps, it has yet to develop specific plans or time frames  for completing them that, among other things, ensure that adequate  resources for accomplishing them are sought.
Addressing these gaps is very important because their existence not only  increases the chances of testing and certification activities being  performed in a manner that is neither repeatable nor verifiable, they also  can create misunderstanding among manufacturers and VSTLs that can  lead to delays in the time needed to certify systems. Such delays have  already been experienced, to the point that needed upgrades to current  systems will likely not be fielded in time for use in the 2008 elections. Such  situations ultimately detract from, and do not enhance, election integrity  and voter confidence. Moreover, by not having established an effective  means for states and local jurisdictions to verify that the systems each  acquires are the same as the EAC-certified version, EAC is increasing the  risk of noncertified versions ultimately getting used in an election.
Beyond the state of EAC’s efforts to define and follow an approach to  testing and certifying voting systems, including efforts to track and resolve  problems with certified systems and use this information to improve the  commission’s testing and certification program, a void exists relative to  having a national focus on tracking and resolving problems with voting  systems that EAC has not certified, and thus has not been assigned explicit  responsibility or has the resources to address. Unless this void is filled,  state and local governments will likely continue to be on their own for  resolving performance and maintenance issues for the vast majority of  voting systems in use today and the near future.
section_title: Recommendations for Executive Action
paragraphs: To assist EAC in building upon and evolving its voting systems testing and  certification program, we recommend that the Chair of the EAC direct the  commission’s Executive Director to ensure that plans are prepared,  approved, and implemented for developing and implementing    detailed procedures, review criteria, and documentation requirements to  ensure that voting system testing and certification review activities are  conducted thoroughly, consistently, and verifiably;    an accessible and available software repository for testing laboratories to  deposit certified versions of voting system software, as well as procedures  and review criteria for evaluating related manufacturer-provided tools to  support stakeholders in comparing their systems with this repository; and   detailed procedures, review criteria, and documentation requirements to  ensure that problems with certified voting systems are effectively tracked  and resolved, and that the lessons learned are effectively used to improve  the certification program.
section_title: Matter for Congressional Consideration
paragraphs: To address the potentially longstanding void in centrally facilitated  problem identification and resolution for non-EAC-certified voting  systems, we are raising for congressional consideration expanding EAC’s  role under HAVA such that, consistent with both the commission’s  nonregulatory mission and the voluntary nature of its voting system  standards and certification program, EAC is assigned responsibility for  providing resources and services to facilitate understanding and resolution  of common voting system problems that are not otherwise covered under  EAC’s certification program, and providing EAC with the resources  needed to accomplish this.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, signed by the EAC Executive  Director, and reprinted in appendix II, the commission stated that it agrees  with the report’s conclusion that more can be done to build on the existing  voting system certification program and ensure that certifications are  based on consistently performed reviews. In addition, EAC stated that it  has found our review and report helpful in its efforts to fully implement  and improve this program. It also stated that it generally accepts our three  recommendations with little comment, adding that it will work hard to  implement them. In this regard, it cited efforts that are planned or  underway to address the recommendations.
EAC provided additional comments on the findings that underlie each of  the recommendations, which it described as needed to clarify and avoid  confusion about some aspects of its certification program. According to  EAC, these comments are intended to allay some of the concerns raised in  our findings. We summarize and evaluate these comments below, and to  avoid any misunderstanding of our findings and the recommendation  associated with one of them, we have modified the report, as appropriate,  in response to EAC’s comments.
EAC also provided comments on our matter for congressional  consideration, including characterizing it as intending “to affect a sea  change in the way that EAC operates its testing and certification” program.  We agree that EAC does not have the authority to compel manufacturers  or states and local jurisdictions to submit to its testing and certification  program and that the wording of the matter in our draft inadvertently led  EAC to believe that our proposal would require the commission to assume  a more regulatory role. In response, we have modified the wording that we  used to clarify any misunderstanding as to our intent.
With respect to our first recommendation for developing and  implementing plans for ensuring that voting system testing and  certification review activities are governed by detailed procedures,  criteria, and documentation requirements, EAC stated that it is committed  to having a program that is rigorous and thorough, and that its program  manual creates such a program. It also stated that it agrees with our  recommendation and that it will work to further the process in its manual  by implementing detailed procedures. In this regard, however, it took issue  with five aspects of our finding.
With respect to our point that criteria and procedures have not been  adequately defined for reviewing the information in the manufacturer  registration application package, EAC stated that such criteria are not  necessary because the package does not require a determination of  sufficiency. We do not agree. According to section 2.4.2 of the program  manual, EAC is to review completed registration applications for  sufficiency. However, as our report states, the manual does not define  criteria as to what constitutes sufficiency and what this sufficiency review  should entail, and EAC officials stated this determination is left up to the  individual reviewer’s judgment.
Concerning our point that criteria and procedures have not been  adequately defined for reviewing the information in the system  certification application package, EAC stated that a technical review of the  package is not required. Rather, it said that the package simply requires a  determination that all necessary information is present, adding that it has a  checklist to assist a reviewer in determining this. We do not agree with this  comment for two reasons. First, our report does not state that the package  review is technical in nature, but rather is a review to determine a  package’s completeness and accuracy. Second, the checklist does not  include any criteria upon which to base a completeness and accuracy  determination. As EAC’s comments confirm, this is important because the  information in the package is to be used by technical reviewers as they  review test plans and test reports to ensure that the testing covers all  aspects of the voting system. For example, EAC requires certification  applications to include a functional diagram depicting how the  components for the voting system function and interact, as well as a  system overview that includes a description of the functional and physical  interfaces between components. Although the checklist provides for  determining whether these items are part of the application package, it  does not provide for checking them for completeness and consistency. We  have clarified this finding in our report by including this example.
As to our point that EAC has not defined how technical reviewers are to  determine the adequacy of system test plans and reports, the commission  stated that our report does not take into account what it described as a  certification requirements traceability matrix that its technical reviewers  use to assess the completeness and adequacy of the plans and reports.  However, EAC also acknowledged in its comments that procedures have  yet to be established relative to the use of the matrix. Further, we  reviewed this matrix, which we refer to in our report as a checklist, and as  we state in our report, this checklist does not provide for capturing how  decisions were reached, including steps performed and criteria applied.  For example, the VVSG requires that systems permit authorized access  and prevent unauthorized access, and lists examples of measures to  accomplish this, such as computer-generated password keys and  controlled access security. While the checklist cites this requirement and  provides for the reviewer to indicate whether the test plan satisfies it, it  does not provide specific guidance on how to determine whether the  access control measures are adequate, and it does not provide for  documenting how the reviewer made such a decision. In response to  EAC’s comments, we have added this access control example to clarify  our finding.
With regard to our point that the program manual does not include defined  test suites, EAC commented on the purpose of these test suites and stated  that it would not be appropriate to include them in the manual because the  manual is not intended to define technical requirements for testing. We  agree that the program manual should not include actual test suites, and it  was not our intent to suggest that it should. Rather our point is that test  suites do not yet exist. Accordingly, we have modified our report to more  clearly reflect this. In addition, we acknowledge EAC’s comment that NIST  is currently in the process of developing test suites, and that it recently  sent several test suites to the VSTLs and other stakeholders for review.  However, as we state in our report, NIST officials said that they are  focused on preparing test suites for the yet-to-be-released update to the  VVSG and not the 2005 version. Further, EAC has not yet established plans  or time frames for finalizing test suites for either versions of these  guidelines, and the program manual does not make reference to the  development of these test suites.
In commenting on our point that differences in interpretation of program  requirements have resulted in test plan and report approval delays, EAC  stated that its interpretation process provides a means for VSTLs and  manufacturers to request clarification of the voting system standards that  are ambiguous. We agree with this statement. However, we also believe  that having the kind of defined procedures and established criteria that are  embodied in our recommendation will provide a common understanding  among EAC stakeholders around testing and certification expectations,  which should minimize the need to reconcile differences in interpretations  later in the process.
With respect to our second recommendation for developing and  implementing plans for an accessible and available software repository for  certified versions of voting system software, as well as the related  manufacturer-provided procedures and tools to support stakeholders in  using this repository, EAC stated that it agrees that implementation of a  repository is needed. However, it stated that there is some  misunderstanding regarding the purpose of the repository and the creation  of software identification tools. Specifically, the commission stated that  the repository is intended for the commission’s own use when conducting  investigations of fielded systems, while the manufacturer-provided system  identification tools are for use by state and local election officials to  confirm that their systems are the same as the one certified by EAC. In  addition, it described steps taken or under way to ensure that a repository  and identification tools are in place when needed. This includes “placing  the onus” on system manufacturers to create verification tools,  investigating software storage options, and discussing with another  government agency and outside vendors the possibility of providing secure  storage for certified software.
We agree with EAC that its repository serves as a tool for its internal use.  However, the repository is also to serve state and local election officials in  verifying that their respective systems are identical to the EAC-certified  versions of the systems. According to the 2005 VVSG software distribution  and setup validation requirements, the process for voting system  purchasers in verifying that the version of the software that they receive  from a manufacturer is the same as the version certified by EAC is to be  performed by comparing it with the reference information generated by  the designated repository. Further, commission officials told us that EAC’s  repository needs to be accessible and easy to use by state and local  election officials. While we understand that the manufacturer-provided  system identification tools serve a separate function from the repository,  both tools together are required for state and local election officials to  verify their systems. We also acknowledge that the commission has  initiated steps relative to establishing a repository and identification tools.
However, our point is that EAC does not have any plans or time frames for  accomplishing this. Further, while we agree that the manufacturers are  responsible for creating the identification tools, as we stated in our report,  EAC has not defined how it will evaluate the manufacturer-provided tools.  To avoid any misunderstanding as to these points, we have slightly  modified our finding and related recommendation.
Concerning our third recommendation for developing and implementing  detailed procedures, review criteria, and documentation requirements for  tracking and resolving problems with certified voting systems and  applying lessons learned to improve the certification program, EAC stated  that the report does not correctly represent its role in confirming that  manufacturers actually correct anomalies in all fielded systems, and it  added that the commission does not have the authority or the human  capital to do so. Accordingly, EAC stated that it informs affected  jurisdictions of system changes, but that it is at the discretion of the states  and local jurisdictions, and beyond the scope of the commission, to  determine whether fixes are made to individual systems in the field.
We agree that the states and local jurisdictions have the responsibility and  authority to determine whether they will implement EAC-approved fixes in  the systems that they own. However, as we state in our report, published  ISO guidance on tracking and resolving problems with certified products  recognizes the importance of the certification body’s decision to require  manufacturers to take corrective actions when defects are discovered, and  to ensure that such actions are taken. Although this guidance  acknowledges the difficulty in ensuring corrective actions are  implemented on all affected units, it states that products should be  corrected “to the maximum degree feasible.” Given EAC’s authority over  registered manufacturers, it can play a larger role in ensuring that  problems with fielded system are in fact resolved, while maintaining the  voluntary nature of its program, by monitoring the manufacturers’ efforts  to fix systems for those jurisdictions that choose to implement such  corrections, and holding manufacturers accountable for doing so. To avoid  any confusion about this point, we have slightly modified our finding.
As to our matter for congressional consideration to amend HAVA to give  EAC certain additional responsibilities relative to problem resolution on  voting systems not certified by EAC, the commission voiced several  concerns. Among other things, it stated that our proposal would “affect a  sea change in the way that EAC operates its testing and certification”  program, changing it from voluntary to mandatory. Further, it stated that it  would, in effect, place EAC in a position to act in a regulatory capacity  without having the specific authority to do so, as it would necessitate  making both the voluntary voting system guidelines and the testing and  certification program mandatory for all states. It also stated that it would  require EAC to have specific authority to compel manufacturers of these  noncertified voting systems to submit their systems for testing, and to  compel states and local jurisdictions to report and resolve any identified  system problems.
We recognize that both the voting system guidelines and the testing and  certification program are voluntary, and that EAC does not have the  authority to compel manufacturers or states and local jurisdictions to  submit to its testing and certification program, or to force them to correct  any known problems or report future problems. We further acknowledge  that the wording of the matter for congressional consideration in our draft  report resulted in EAC interpreting accomplishment of it as requiring such  unintended measures. Therefore, we have modified it to clarify our intent  and to avoid any possible misunderstanding. In doing so, we have  emphasized our intent for EAC to continue to serve its existing role as a  facilitator and provider of resources and services to assist states and local  jurisdictions in understanding shared problems, as well as the voluntary  nature of both the system guidelines and the testing and certification  program. Further, we seek to capitalize on EAC’s unique role as a national  coordination entity to address a potentially longstanding, situational  awareness void as it pertains to voting systems in use in our nation’s  elections. As we state in our report, this void increases the chances of  states and local jurisdictions duplicating efforts to fix common system  problems, and of problems addressed by one state or local jurisdiction  being unknown to others. We believe that a key to overcoming this will be  strong central leadership, and that with the appropriate resources, EAC is  in the best position to serve this role.
We are sending a copy of this report to the Ranking Member of the House  Committee on House Administration, the Chairman and Ranking Member  of the Senate Committee on Rules and Administration, the Chairmen and  Ranking Members of the Subcommittees on Financial Services and  General Government, Senate and House Committees on Appropriations,  and the Chairman and Ranking Member of the House Committee on  Oversight and Government Reform. We are also sending copies to the  Chair and Executive Director of the EAC, the Secretary of Commerce, the  Acting Director of the NIST, and other interested parties. We will also  make copies available to others on request. In addition, this report will be  available at no charge on the GAO Web site at www.gao.gov.
Should you or your staff have any questions on matters discussed in this  report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine whether the Election Assistance  Commission (EAC) has (1) defined an effective approach to testing and  certifying voting systems, (2) followed its defined approach, and   (3) developed an effective mechanism to track problems with certified  systems and use the results to improve its certification program.
To address the first and third objectives, we researched leading practices  relevant to certification testing/conformity assessment and tracking and  resolving problems with certified products, including published guidance  from the National Institute of Standards and Technology (NIST),  International Organization for Standardization, and International  Electrotechnical Commission, and legal requirements in the Help America  Vote Act. We obtained and reviewed relevant EAC policies and procedures  for testing, certifying, decertifying, and recertifying voting systems.  Specifically, we reviewed the EAC Certification Program Manual and other  EAC-provided documents. We interviewed EAC officials and their  technical reviewers, NIST officials, representatives from the industry trade  association for voting system manufacturers, representatives from the  voting system test laboratories, and the National Association of State  Election Directors’ point-of-contact for qualified systems. We then  compared this body of evidence with the leading practices and related  guidance we had researched, as well as applicable legal requirements, to  determine whether EAC’s program had been effectively defined. In  addition, for the third objective, we reviewed the contents and policy of  EAC’s clearinghouse.
To address our second objective, we obtained and reviewed actions and  artifacts from EAC’s execution of its certification program to date. We  assessed this information against the policies, procedures, and standards  outlined in the EAC Certification Program Manual and the 2005 Voluntary  Voting System Guidelines, and after discussing and confirming our  findings with EAC officials, we determined whether EAC had followed its  defined approach.
In addition, to determine the impact of federal certification time frames,  we included a question about EAC certification on a survey of officials  from all 50 states, the District of Columbia, and 4 territories. We also  contacted officials from states that indicated their intent to use EAC  certification for the 2008 elections to better understand how they plan to  address voting system certification in their state relative to EAC’s  program. To develop our survey, we reviewed related previous and  ongoing GAO work, and developed a questionnaire in collaboration with  GAO’s survey and subject matter experts. We conducted pretests in person  and by telephone with election officials from 5 states to refine and clarify  our questions. Our Web-based survey was conducted from December 2007  through April 2008. We received responses from 47 states, the District of  Columbia, and all 4 territories (a 95 percent response rate). Differences in  the interpretation of our questions among election officials, the sources of  information available to respondents, and the types of people who do not  respond may have introduced unwanted variability in the responses. We  examined the survey results and performed analyses to identify  inconsistencies and other indications of error, which were reviewed by an  independent analyst. We also contacted officials from those states whose  survey response indicated their intent to use EAC certification for the 2008  elections to identify their plans and approaches for state certification in  the event that federal certification could not be completed to meet their  election preparation schedules.
We conducted this performance audit at EAC offices in Washington, D.C.,  from September 2007 to September 2008 in accordance with generally  accepted government auditing standards. Those standards require that we  plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objectives.
section_title: Appendix II: Comments from the Election Assistance Commission
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Paula Moore, Assistant Director;  Mathew Bader; Neil Doherty; Nancy Glover; Dan Gordon; David  Hinchman; Valerie Hopkins; Rebecca LaPaze; Jeanne Sung; and Shawn  Ward made key contributions to this report.