Report to Congressional Requesters

United States General Accounting Office

GAO

October 2001 ELECTIONS

Status and Use of

Federal Voting

Equipment Standards

GAO-02-52
Page i GAO-02-52 Voting Equipment Standards

Letter 1

Results in Brief 2

Background 4

FEC Has Developed Voting Equipment Standards but Has Not

Maintained Them 6

NASED’s Process for Testing and Qualifying Voting Equipment 12

Conclusions 15

Matters for Congressional Consideration 16

Recommendations for Executive Action 16

Agency Comments and Our Evaluation 16

Appendix I Objectives, Scope, and Methodology 19

Appendix II Comments From the Federal Election Commission 21

Appendix III System Requirements and Supporting Life-Cycle

Processes and Our Analysis 27

Tables

Table 1: Types of Requirements and Processes Satisfied in FEC’s

1990 Voting Equipment Standards 10

Table 2: Types of Requirements and Processes Not Satisfied in

FEC’s 1990 Standards but Satisfied in the Draft Standards 11

Table 3: NASED Qualified Equipment by Voting Equipment

Category 15

Figures

Figure 1: States That Require the Use of FEC’s Voting Equipment

Standards 7

Figure 2: FEC’s Revised Standards Development Time Frame 8

Figure 3: Simplified Diagram of the Voting Equipment Standards

Development and Implementation Process 13

Contents
Page 1 GAO-02-52 Voting Equipment Standards

October 15, 2001

The Honorable Thomas Daschle

Majority Leader

The Honorable Trent Lott

Minority Leader

United States Senate

The Honorable Christopher J. Dodd

Chairman

The Honorable Mitch McConnell

Ranking Minority Member

Committee on Rules and Administration

United States Senate

Events surrounding the last presidential election have brought to light a

number of issues about the conduct of elections that extend to the people,

processes, and technology used to administer these elections. We were

asked by several congressional committees and members to review

aspects of elections throughout the United States. In response, we are

issuing a series of reports. To date, we have issued reports on the scope of

congressional authority in election administration and voting assistance to

military and overseas citizens. 1 Other forthcoming reports will examine

voting accessibility for people with disabilities; the factors that affected

the uncounted votes in the November 2000 presidential election; and the

people, processes, and technology challenges affecting the conduct and

administration of elections. We are also issuing a capping report that

draws upon our extensive body of work to identify the main issues and

challenges confronting our nation’s election system and to delineate an

analytical framework that Congress could use as it weighs the merits of

various reform proposals.

This report focuses on the status and use of federal voting equipment

standards. These standards define minimum functional and performance

requirements for voting equipment, such as accurately recording votes

cast. The standards also define minimum life-cycle management processes

 1

Elections: The Scope of Congressional Authority in Election Administration (GAO-01-

470, March 13, 2001) and Elections: Voting Assistance to Military and Overseas Citizens

Should Be Improved (GAO-01-1026, September 28, 2001).

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-02-52 Voting Equipment Standards

for voting equipment developers to follow, such as quality assurance. As

agreed with your offices, our objectives were to (1) identify the Federal

Election Commission’s (FEC) role regarding voting equipment and assess

how well FEC is fulfilling its role and (2) identify the National Association

of State Election Directors’ (NASED) process for testing and qualifying

voting equipment against FEC’s standards. Details of our objectives,

scope, and methodology are contained in appendix I.

No federal agency has been assigned explicit statutory responsibility for

developing voting equipment standards; however, the Federal Election

Commission assumed this role by developing voluntary standards in 1990

for computer-based systems,2

 and the Congress has supported this role

with appropriations. These standards describe specific performance

benchmarks, and address many—but not all—types of system

requirements. In 1997, the Commission initiated efforts to evaluate the

1990 standards to identify areas to be updated and in 1999, it initiated

efforts to update the standards. We reviewed available drafts of these

updated standards and found they describe most—but again not all—types

of system requirements. The Commission plans to issue revised standards

in 2002. This update is necessary because the Commission has not

proactively maintained them, thus allowing them to become out of date.

According to Federal Election Commission officials, the Commission has

not proactively maintained the standards because it has not been assigned

explicit responsibility to do so. Without current, relevant, and complete

voting equipment standards, states may choose not to follow them,

resulting in the adoption of disparate standards that could drive up the

cost of voting equipment and produce unevenness among states in the

capabilities of their respective equipment.

No federal agency has been assigned responsibility for or assumed the role

of testing voting equipment against the federal standards. Instead, the

National Association of State Election Directors, through its Voting

Systems Committee, has assumed responsibility for implementing the

federal voting equipment standards by accrediting independent test

authorities,3

 which in turn, test equipment against the standards. To this

 2

Performance and Test Standards for Punchcard, Marksense, and Direct Recording

Electronic Voting Systems (January 1990).

3

Independent test authorities are contractors independent from the voting equipment

vendors and are responsible for testing voting equipment to ensure that they meet the

Commission’s standards.

Results in Brief
Page 3 GAO-02-52 Voting Equipment Standards

end, the committee has developed procedures to accredit the independent

test authorities.4

 According to the test authorities, testing is generally

iterative, in which the voting equipment vendors are provided an

opportunity to correct deficiencies identified during testing and resubmit

the modified equipment for retesting. When testing is successfully

completed, the independent test authorities notify the National

Association of State Election Directors that the equipment has satisfied

testing requirements.5

 As of July 3, 2001, the Association had qualified 21

different voting equipment, representing 10 vendors.

Because development, maintenance, and implementation of voting

equipment standards are very important responsibilities, we are raising

matters for congressional consideration regarding the explicit assignment

of responsibility in these areas. Additionally, we are making

recommendations to the Federal Election Commission’s commissioners

aimed at improving its efforts to update its 1990 standards.

In written comments on a draft of this report, the Commission stated that

it generally agreed with most of our observations and recommendations,

and that it agreed with our matter for congressional consideration.

However, the Commission commented that it disagreed with our

conclusions that it has not proactively maintained its voting equipment

standards and that its draft updates to those standards do not satisfy

security certification and quality assurance requirements. While the

Commission deserves credit for recognizing the need over a decade ago

for voting equipment standards and developing and updating them, despite

a lack of explicit statutory responsibility to do so, we disagree that its

efforts to update its 1990 standards have been proactive. Nine years

elapsed from the time the Commission issued the standards and the time

the Commission began updating them; this is too long a period between

updates given advances in the field and is the major reason that the

current standards are out of date. Further, while we agree that the draft

standards address quality assurance, we are concerned that they omit

 4

According to the National Association of State Election Directors, accreditation signifies

formal recognition that the independent test authority possesses or will acquire the

competence to design and perform specific test methods applicable to voting equipment

hardware and software and that the test authority has adequately demonstrated its

competence for voting equipment testing.

5

Independent test authorities notify the Election Center, which serves as the National

Association of State Election Directors’ Secretariat and maintains the list of the

association’s approved voting equipment.
Page 4 GAO-02-52 Voting Equipment Standards

essential aspects of quality assurance, such as quality assurance plans and

process reviews. Regarding the draft standards satisfaction of security

requirements, subsequent to commenting on a draft of this report, the

Commission provided us additional draft standards, which address the

requirements for security certification. Therefore, we have modified this

report to recognize this new information. Last, we acknowledge the

Commission’s position, as stated in its comments, for not including certain

quality assurance standards areas in the revised draft standards was the

result of deliberative and collaborative interaction among NASED and

Commission staff, and we have modified this report to reflect this change.

In 1975 Congress created the FEC to administer and enforce the Federal

Election Campaign Act. To carry out this role, FEC discloses campaign

finance information, enforces provisions of the law such as limits and

prohibitions on contributions, and oversees the public funding of

presidential elections.

Within FEC, the Office of Election Administration (OEA) serves as a

national clearinghouse for information regarding the administration of

federal elections. As such, OEA assists state and local election officials by

developing voluntary voting equipment standards, responding to inquiries,

publishing research on election issues, and conducting workshops on all

matters related to election administration. In addition, it answers

questions from the public and briefs foreign delegations on the U.S.

election process, including voter registration and voting statistics.

FEC consists of six voting members, appointed by the President and

confirmed by the Senate. To encourage nonpartisan decisions, no more

than three commissioners can be members of the same political party, and

at least four votes are required for most official Commission actions.

FEC’s budget for fiscal year 2001 is $40.4 million, and of that amount,

$804,000 is allocated to support OEA functions. FEC has 357 full-time staff,

of which 5 are allocated to OEA functions.

The voting methods used in the United States can be placed into five

categories: paper ballots, mechanical lever machines, punch cards, optical

scan, and direct recording electronic. The last three methods use

Background

Voting Equipment Used in

the United States
Page 5 GAO-02-52 Voting Equipment Standards

computer-based equipment. Three of the five—paper ballots, punch cards,

and optical scan—use some kind of paper ballot to record voters’ choices.6

• Paper Ballot. Voters use a paper ballot listing the names of the candidates

and issues and record their choice by placing a mark in a box next to the

candidate’s name or issue. After making their choices, the ballots are

dropped into a sealed ballot box to be manually tabulated.

• Mechanical Lever. Voters pull a lever next to the candidate’s name or issue

and the machine records and tallies the votes using a counting mechanism.

Write-in votes must be recorded on a separate document. Election officials

tally the votes by reading the counting mechanism totals on each lever

voting machine.

• Punch Card. Voters can use one of two basic types of punch cards—

Votomatic or Datavote. In both instances, voters use a computer-readable

card to cast their vote. The Votomatic uses a computer-readable card with

numbered boxes that correspond to a particular ballot choice. The choices

corresponding to those numbered boxes are indicated to the voter in a

booklet attached to a vote recording machine, with the appropriate places

to punch indicated for each candidate and ballot choice. The voter uses a

simple stylus to punch out the box corresponding to each candidate and

ballot choice. In the Datavote, the names of the candidates and issues are

printed on the card itself—there is no ballot booklet. The voter uses a

stapler-like punching device to punch out the box corresponding to each

candidate and ballot choice. To tally the votes in both instances, the

ballots are fed into a computerized tabulation machine that records the

vote by reading the holes in the ballots.

• Optical Scan. Voters use a computer-readable paper ballot listing the

names of the candidates and issues. The voters record their choices by

using an appropriate writing instrument to fill in a box or oval, or

complete an arrow next to the candidate’s name or issue. The ballot is

then fed into a computerized tabulation machine, which senses or reads

the marks on the ballot, and records the vote.

• Direct Recording Electronic. Voters use a ballot that is printed and posted

on the voting machine or displayed on a computer screen listing the names

of the candidates and issues. Voters record their choices by pushing a

button or touching the screen next to the candidate’s name or issue. When

a voter is finished, the vote is submitted by pressing a vote button, which

 6

A more detailed description of each type of voting method and the associated equipment

can be found in Elections: Perspectives on Activities and Challenges Across the Nation

(GAO-02-03, October, 2001).
Page 6 GAO-02-52 Voting Equipment Standards

stores the vote in a computer memory chip. Election officials tally the

votes by reading the votes totaled on each machine’s computer chip.

While neither FEC nor any other federal agency has explicit statutory

responsibility to develop voting equipment standards, the Congress has

appropriated funds for FEC to develop and update the standards. FEC first

issued voting equipment standards in 1990. These standards identify

minimum functional and performance requirements for punch card,

optical scan, and direct recording electronic voting equipment, and specify

test procedures to ensure that the equipment meet these requirements.7

The functional and performance requirements address what voting

equipment should do and delineate minimum performance thresholds,

documentation provisions, and security and quality assurance

requirements. The test procedures describe three stages of testing:

qualification, certification, and acceptance. According to FEC’s standards

document:

• Qualification testing is the process by which a voting equipment is shown

to comply with the requirements of its own design specification and with

the requirements of FEC standards.

• Certification testing, generally conducted by individual states, determines

how well voting equipment conform to individual state laws and

requirements.

• Acceptance testing is generally performed by the local jurisdictions

procuring voting equipment and demonstrates that the equipment, as

delivered and installed, satisfies all the jurisdiction’s functional and

performance requirements.

The standards are voluntary; states are free to adopt them in whole, in

part, or reject them entirely. To date, 38 states require that voting

equipment used in the state meet FEC standards either in total or in part.8

Figure 1 shows these states.

 7

The FEC standards address only computer-based systems; therefore, the standards do not

address the paper and mechanical lever machine voting methods.

8

This includes the District of Columbia. Four of the 38 states reported that they followed

the FEC standards in part.

FEC Has Developed

Voting Equipment

Standards but Has

Not Maintained Them
Page 7 GAO-02-52 Voting Equipment Standards

Figure 1: States That Require the Use of FEC’s Voting Equipment Standards

Source: GAO survey results.

In September 1997, FEC initiated efforts to evaluate its voting equipment

standards and identify areas to be updated, and in July 1999, FEC initiated

efforts to revise the standards.9

 As part of this revision, FEC has been

working closely with state and local election officials and vendors to

incorporate industry comments on the draft standards. FEC plans to issue

the revised standards in multiple volumes: volume I is to include the

 9

FEC has contracted with American Management Systems to support FEC in revising the

voting equipment standards.
Page 8 GAO-02-52 Voting Equipment Standards

functional and performance requirements for voting equipment; volume II

is to provide the detailed test procedures, including information to be

submitted by the vendor, tests to be conducted to ensure compliance with

the standards, and the criteria to be applied to pass the individual tests.

Figure 2 depicts FEC’s time frames for revising the standards.

Figure 2: FEC’s Revised Standards Development Time Frame

Source: FEC data.

Organizations such as the Department of Defense and the Institute of

Electrical and Electronics Engineers have developed guidelines for

various types of systems requirements and for the processes that are

important to managing the development of any system throughout its life

cycle. These types of systems requirements and processes include, for

example:

• Security and Privacy Protection. Requirements defining the

security/privacy environment, types of security needed (e.g., data

confidentiality and fraud prevention), risks the system must withstand,

safeguards required to withstand those risks, security/privacy policies that

must be met, accountability (i.e., audit trails), and criteria for security

certification.10

 10Security certification is the technical evaluation that establishes the extent to which a

computer system, application, or network design and implementation meets specified

security requirements.

FEC’s Voting Equipment

Standards Address Most

Key Types of Systems

Requirements
Page 9 GAO-02-52 Voting Equipment Standards

• Human Factors. Requirements defining the usability of the system,

including considerations for human capabilities and limitations, and the

use and accessibility of the system by persons with disabilities.

• Documentation. Processes for recording information produced during the

system development life cycle, which includes identifying documents to be

produced; identifying the format, content, and presentation items for each

document; and developing a process for reviewing and approving each

document.

• Configuration Management. Processes to establish and maintain the

integrity of work products through the system development life cycle,

including developing a configuration management plan, identifying work

products to be maintained and controlled, establishing a repository to

maintain and control them, evaluating and approving changes to the work

products, accounting for changes to the products, and managing the

release and delivery of products.

• Quality Assurance. Processes to provide independent verification of the

requirements and processes used to develop and produce the system,

which include developing a quality assurance plan, determining what

system development product and process standards are supposed to be

followed, and conducting reviews to ensure that the product and process

standards are followed.

While FEC’s 1990 standards satisfy most of these areas, they do not satisfy

all. For example, in the area of security, the standards do not address the

security/privacy environment in which the voting equipment must operate,

the types of security to be provided, the risks the equipment must

withstand, the security/privacy policies that must be met, or the criteria

for security certification. Further, the standards do not specify

requirements for voting equipment usability, taking into account human

capabilities and limitations, or the use and accessibility of the voting

equipment by persons with disabilities. Table 1 summarizes the types of

requirements and processes satisfied in FEC’s 1990 voting equipment

standards.
Page 10 GAO-02-52 Voting Equipment Standards

Table 1: Types of Requirements and Processes Satisfied in FEC’s 1990 Voting

Equipment Standards

Types of requirements and processes Satisfied in the 1990 standards?

Functional and performance Yes

System quality Yes

Security and privacy protection No

Human factors No

System external interface Yes

Installation-dependent data Yes

System environment Yes

Design constraints Yes

Physical Yes

Safety Yes

Personnel Yes

Training Yes

Logistics Yes

Documentation No

Configuration management No

Quality assurance No

Source: GAO analysis.

As part of FEC’s current effort to revise the 1990 standards, it has made

improvements in all five of the areas in which we identified missing types

of requirements and processes. For example, in the area of human factors,

the draft standards now include requirements for the use and accessibility

of voting equipment by persons with disabilities.11 Further, for

documentation, the draft standards include requirements for identifying

documents to be produced; defining the format, content, and presentation

items for each document; and developing a process for reviewing and

approving each document. In addition, in the area of security, the

standards now address security types, risks, safeguards, policies,

accountability, and certification.

While FEC has made improvements, the draft standards do not satisfy two

areas—human factors and quality assurance. For example, in the area of

human factors, the draft standards do not address requirements for

equipment usability, including considerations for human capabilities and

 11Our review of these standards did not include validating that the requirements are correct

and complete, beyond determining whether the standards addressed all the requirements

and process elements.
Page 11 GAO-02-52 Voting Equipment Standards

limitations. Finally, the draft standards do not yet specify the development

of a quality assurance plan or the performance of quality assurance

reviews to ensure that the equipment development process requirements

are being met. Table 2 summarizes the types of requirements and

processes not satisfied in FEC’s 1990 voting equipment standards but

satisfied in the draft standards. Appendix III provides a detailed

description of the requirements and process types and our complete

analysis of FEC’s 1990 voting standards and draft standards.

Table 2: Types of Requirements and Processes Not Satisfied in FEC’s 1990

Standards but Satisfied in the Draft Standards

Type of requirements and processes Satisfied in the draft standards?

Security and privacy protection Yes

Human factors No

Documentation Yes

Configuration management Yes

Quality assurance No

Source: GAO analysis.

In the area of quality assurance, FEC stated in its written comments on a

draft of this report that its decision to not include quality assurance

process reviews in the revised standards was the result of deliberative and

collaborative interaction among NASED’s Voting System Committee and

FEC staff. In addition, FEC did not include equipment usability because it

was determined not to be an area of immediate concern by the election

community during FEC’s evaluation of the standards to identify areas to

be updated. FEC agrees that equipment usability should be addressed in

the standards and has stated that it will fully do so once resources are

available. Beyond this stated commitment, FEC has not established any

specific plans or allocated specific resources for doing so. Until FEC

addresses these missing requirements, the voting equipment standards’

value and utility will be diminished.

Given the pace of today’s technological advances, standards must be

proactively maintained to ensure that they remain current, relevant, and

complete. Standards-setting bodies, such as the American National

Standards Institute and the National Institute of Standards and

Technology, require that standards be revised or reaffirmed at least once

every 5 years. This is particularly important with voting equipment

standards, which must respond to technological developments if they are

FEC Has Not Proactively

Maintained the Voting

Equipment Standards
Page 12 GAO-02-52 Voting Equipment Standards

to be current, complete, and relevant, and are to be useful to state and

local election officials in assuring the public that their voting equipment

are reliable.

FEC has not proactively maintained its voting equipment standards. As

previously stated, FEC is only now updating the 1990 standards. Because

FEC has not proactively maintained the standards, they have become out

of date. Vendors are using new technology and expanding voting

equipment functions that are not sufficiently covered by the 1990

standards. For example, the 1990 standards do not address election

management systems, which are used to prepare ballots and programs for

use in casting and tallying votes, and to consolidate, report, and display

election results. According to a NASED committee representative and the

Independent Test Authority (ITA) responsible for testing election

management systems, the lack of adequate standards to address election

management software has forced them to interpret the current voting

equipment standards to accommodate the development of this new

software. Further, according to these representatives, these

interpretations have not been documented and formally shared with FEC.

As mentioned earlier, FEC is updating its standards, and the draft

standards now address election management systems.

FEC officials acknowledge the need to actively maintain the standards, but

state that they have not done so because they have not been assigned

explicit responsibility. By not ensuring that voting equipment standards

are current, complete, and relevant, states may choose not to follow them,

resulting in states adopting disparate standards. In turn, this could drive

up the cost of voting equipment being designed to multiple standards and

produce unevenness among states in the capabilities of voting equipment.

No federal agency, including FEC, has been assigned explicit

responsibility for testing voting equipment against FEC standards, and no

federal agency has assumed this role. Rather, NASED has assumed

responsibility for implementing the standards.12 To do so, NASED

established a voting systems committee, which comprises selected state

and local election officials and technical advisers. This committee

 12NASED, which comprises chief election officials from each state and territory of the

United States, provides a forum for state election officials to share information about their

duties, responsibilities, methods of operation, and suggestions for improving election laws.

NASED’s Process for

Testing and Qualifying

Voting Equipment
Page 13 GAO-02-52 Voting Equipment Standards

accredits ITAs to test and qualify voting equipment against FEC standards.

Figure 3 illustrates the voting equipment standards program, from the

development of voting equipment standards through the testing and

qualification of voting equipment.

Figure 3: Simplified Diagram of the Voting Equipment Standards Development and

Implementation Process

Source: GAO, based on FEC and NASED information.

To accredit the ITAs, the NASED committee has developed requirements

and procedures, which include provisions for NASED to periodically

reaccredit the ITAs and conduct on-site inspection visits, both of which
Page 14 GAO-02-52 Voting Equipment Standards

are important to ensuring that the accredited laboratories continue to

comply with all requirements.13 To date, the committee has not

reaccredited or inspected ITAs because, according to NASED committee

representatives, they rely on the committee’s technical advisers’ ongoing

conversations with ITA officials and the officials’ participation in

committee meetings to ensure that the ITAs are fulfilling their

responsibilities effectively.

Currently, three ITAs are approved to test voting equipment against the

FEC standards. In 1994, the NASED committee accredited Wyle Labs to

test the hardware and machine-resident software components of

proprietary vote cast and tally equipment. In February 2001, Metamor

(previously PSINet) applied for accreditation to conduct qualification

testing of vote tabulation and election management software.14 Also in

2001, SysTest applied for accreditation to conduct qualification testing of

voting tabulation and election management software. While both Metamor

and SysTest have been granted an interim approval to test voting

equipment, NASED has not yet accredited either.

To test voting equipment, voting equipment vendors submit requests for

testing to the ITAs, who then prepare a test procedure. The test procedure

details the software and hardware testing requirements that the voting

equipment will be tested against and is based on both the FEC voting

equipment standards and the vendors’ design specifications.

According to ITA officials responsible for testing voting equipment, the

testing process is generally an iterative one. Vendors are provided an

opportunity to correct deficiencies identified during testing and resubmit

the modified voting equipment for retesting. At the end of testing, the ITA

completes a test report and notifies the Election Center that the voting

equipment has successfully satisfied testing requirements.15 The Election

 13NASED, Accreditation of Independent Testing Authorities For Voting System

Qualification Testing Handbook (May 1, 1992).

14Software qualification testing does not include testing the software which is in a

permanent machine resident status (e.g., programmed on a read only memory chip) nor the

hardware and operating systems on which the software operates. Also, in 1997, the

committee accredited Nichols Research to test software and the integration of the software

with vote cast and tally equipment. However, as of December 2000, this responsibility was

moved to PSINet.

15The Election Center is a nonprofit organization dedicated to training local election

officials in election administration and serves as the NASED voting systems committee

secretariat, assisting NASED in implementing national voting equipment standards.
Page 15 GAO-02-52 Voting Equipment Standards

Center then assigns a NASED number to the specific equipment model and

firmware release that was tested and maintains the list of qualified voting

equipment.16 Each time a vendor issues a new model or software release,

the vendor is to submit a request for testing to the ITAs in order to qualify

the new model or release. As of July 3, 2001, NASED had qualified 21

models of voting equipment and 7 election management systems,

representing 10 vendors. See table 3 for a breakout of the types of

equipment qualified.

Table 3: NASED Qualified Equipment by Voting Equipment Category

Equipment category Number qualified

Punch card 1

Optical scan 6

Direct recording electronic 14

Election management systemsa 7

Total 28

a

Not specifically a separate voting equipment category. Rather these systems support other voting

equipment categories by preparing ballots and programs for use in casting and tallying votes, and

consolidating, reporting, and displaying election results.

Source: NASED data, as of July 3, 2001.

The ITAs stated that the testing process generally takes about 2 to 3

months. This is contingent, however, upon the vendors having the proper

documentation in order. If documentation is missing or incomplete, the

process may take longer. According to the ITAs, the cost of qualification

testing ranges from $40,000 for vote cast and tally equipment to $75,000 for

vote tabulation and election management software.

While not explicitly provided for in legislation, FEC and NASED have

assumed and are performing important roles by developing voting

equipment standards and testing and qualifying equipment against these

standards, respectively. Given the current pace of technological change for

voting equipment, the degree to which these standards are actively

maintained and the extent to which they are appropriately applied, can

have a direct bearing on the capabilities of voting equipment. This, in turn,

can affect the successful conduct of national, state, and local elections.

Therefore, it is important that responsibility for these roles be clearly

 16Firmware is software that is embedded in a hardware device that allows reading and

executing the software, but it does not allow modification.

Conclusions
Page 16 GAO-02-52 Voting Equipment Standards

assigned. By doing so, the appropriate federal role in these important

areas can be deliberated, decided, and explicitly defined, thereby avoiding

another situation where the standards are allowed to become out of date.

It is also important that these roles be executed effectively. In the case of

FEC’s ongoing update of the standards, this means that requirements for

equipment usability, and quality assurance should be developed.

As part of the ongoing debate and deliberation over election reform in

general, and the federal role in voting equipment standards in particular,

the Congress may wish to consider assigning explicit federal authority,

responsibility, and accountability for voting equipment standards,

including proactive and continuous update and maintenance of the

standards. Given that no federal or state entity has been assigned explicit

authority or responsibility for testing voting equipment against the FEC

standards, the Congress may wish to consider what, if any, federal role is

appropriate, regarding implementation of the standards, including the

accreditation of ITAs and the qualification of voting equipment.

To improve the quality of FEC’s voting equipment standards, we

recommend that the FEC Commissioners direct the OEA Director to

accelerate the development of requirements for equipment usability,

including considerations for human capabilities and limitations. To

improve the quality of FEC’s current efforts to update the voting

equipment standards, we also recommend that the FEC Commissioners

direct the OEA Director to develop requirements for quality assurance,

including developing a quality assurance plan and conducting quality

assurance process reviews.

In its written comments on a draft of this report (reprinted in appendix II),

the FEC Chairman and Vice Chairman stated that FEC generally agrees

with most of our observations and recommendations, including that

human factors are not being addressed in the revised voting equipment

standards and that FEC needs to accelerate their development in future

iterations of the standards. Additionally, FEC agreed with our matter for

congressional consideration.

Nevertheless, FEC commented that it was concerned with the report’s

portrayal of the Commission as being insufficiently proactive in revising

voting equipment standards, stating that its efforts have been as timely as

Matters for

Congressional

Consideration

Recommendations for

Executive Action

Agency Comments

and Our Evaluation
Page 17 GAO-02-52 Voting Equipment Standards

possible given certain practical constraints, which it described in a

chronology of events and circumstances. FEC also commented that it

disagrees with the draft report’s characterization of the Commission’s

ongoing efforts to update security and quality assurance standards as

incomplete, describing how both areas are being addressed. Subsequent to

providing us with its written comments on a draft of this report, FEC also

provided us with additional draft standards that address security

requirements. Accordingly, we have modified this report, including our

recommendations, to reflect this new information.

We do not agree with either of FEC’s other two points of concern.

Regarding FEC’s concern with the report’s portrayal of the Commission as

being insufficiently proactive in revising voting equipment standards, FEC

states in its comments that 7 years elapsed from the time that the

standards were first issued in 1990 to the time that FEC first began

evaluating them to identify areas that needed to be updated. Further, it

states that another 2 years elapsed between the time FEC began evaluating

the standards and the time it began updating them. We recognize that FEC

is performing, through its own initiative, an important role in developing

and updating the standards, and deserves credit for doing so. However, in

our view, allowing 9 years to pass before beginning to update the

standards, regardless of the practical circumstances that FEC cites, is too

long and does not constitute a proactive maintenance process and is the

primary reason that the current standards are out of date. Regarding FEC’s

disagreement with the report’s characterization of the Commission’s

ongoing efforts to update quality assurance standards as incomplete, we

do not question, and in fact state in this report, that the draft standards

address requirements for quality assurance. However, our main concern is

that important and relevant aspects of quality assurance requirements,

such as quality assurance plans and process reviews, respectively, are not

addressed.

Concerning FEC’s decision to omit quality assurance standards areas from

the revised draft standards, we modified this report to reflect FEC’s

position that its decision resulted from deliberative and collaborative

interaction among NASED and FEC staff and was not, as we were told

during the course of our review by the OEA Director, areas that were

overlooked.

We are sending copies of this report to the Chairmen and Ranking

Minority Members of the Senate Appropriations Subcommittee on

Treasury and General Government and the House Appropriations
Page 18 GAO-02-52 Voting Equipment Standards

Subcommittee on Treasury, Postal Service, and General Government; the

Director of the Office of Management and Budget; and the Chairman and

Vice Chairman of FEC. Copies will also be available at our Web site at

www.gao.gov. If you have any questions, please contact me at (202) 512-

6240 or by email at hiter@gao.gov. Key contributors to this assignment

were Deborah A. Davis, Richard Hung, and Eric Winter.

Randolph C. Hite

Director, Information Technology

 Systems Issues
Appendix I: Objectives, Scope, and

Methodology

Page 19 GAO-02-52 Voting Equipment Standards

The objectives of our review were to (1) identify Federal Election

Commission’s (FEC) role regarding voting equipment and assess how well

FEC is fulfilling its role and (2) identify the National Association of State

Election Director’s (NASED) process for testing and qualifying voting

equipment against FEC’s voluntary voting equipment standards.

To identify FEC’s role regarding voting equipment, we researched FEC’s

statutory and legislative role in developing and maintaining voting

equipment standards. To further identify FEC’s role, we reviewed relevant

documents, including the Plan to Update the Voting Systems Standards,

1

the standards update project contract, project work plans, and legislative

proposals, and interviewed key FEC officials, including the Director, OEA.

To assess FEC’s voting equipment standards, we examined relevant

guidelines and procedures for the development of system requirements.

Specifically, we examined the Department of Defense’s Data Item

Description for System/Subsystem Specifications; the Institute of

Electrical and Electronics Engineers’ Standard 12207 on Software Life

Cycle Processes, and the Software Engineering Institute’s Software

Development Capability Maturity ModelTM 2

 and identified 13 types of

systems requirements and 3 supporting life-cycle processes that are

important in the development of any system. We then compared these

types of requirements and processes against FEC’s 1990 voting equipment

standards to determine if all key elements were addressed. In those areas

where variances were noted, we compared the types of requirements and

processes against relevant sections of volumes I and II of the draft

standards to determine whether FEC had addressed any of these missing

requirements.3

 We only reviewed those portions of the draft standards for

which we identified missing types of requirements and processes in the

1990 standards. In addition, our review of the standards did not include

validating that the requirements are correct and complete beyond

determining whether the standards addressed all of the requirements and

process key elements.

 1

ManTech Advanced Systems International, Inc., February 9, 1999.

2

The Software Engineering Institute is operated by Carnegie Mellon University as a

federally funded research and development center sponsored by the Department of

Defense. Capability Maturity Model is a registered mark of Carnegie Mellon University.

3

We examined all but two sections (hardware standards and software/firmware standards)

in volume I and those sections pertaining to security and certification in volume II.

Appendix I: Objectives, Scope, and

Methodology
Appendix I: Objectives, Scope, and

Methodology

Page 20 GAO-02-52 Voting Equipment Standards

To identify NASED’s process for testing and qualifying voting equipment

against FEC’s voting equipment standards, we interviewed officials from

NASED, the Election Center, and the two independent test authorities

(ITA). We also reviewed documentation describing NASED’s process,

NASED’s Accreditation of Independent Testing Authorities For Voting

System Qualification Testing Handbook,

4

 ITAs’ generic test plans, and

NASED’s policies, procedures, and by-laws. We also provided a copy of

relevant parts of this report to the Chairman of the NASED Voting System

Committee for comment. The Chairman stated that the report accurately

reflected the NASED process.

We also contacted officials in the State Election Director's offices in each

of the 50 states and the District of Columbia to determine which states

required that their voting equipment be in compliance with FEC's

standards. We did not verify the officials' responses.

We performed our work at FEC headquarters in Washington, D.C.,

NASED, the Election Center, and the independent test authorities from

March 2001 through September 2001, in accordance with generally

accepted government auditing standards.

 4

NASED, May 1, 1992.
Appendix II: Comments From the Federal

Election Commission

Page 21 GAO-02-52 Voting Equipment Standards

Appendix II: Comments From the Federal

Election Commission

Note: GAO comments

supplementing those in

the report text appear at

the end of this appendix.

See comment 2.

See comment 1.
Appendix II: Comments From the Federal

Election Commission

Page 22 GAO-02-52 Voting Equipment Standards
Appendix II: Comments From the Federal

Election Commission

Page 23 GAO-02-52 Voting Equipment Standards

See comment 6.

See comment 5.

See comment 4.

See comment 3.
Appendix II: Comments From the Federal

Election Commission

Page 24 GAO-02-52 Voting Equipment Standards
Appendix II: Comments From

the Federal Election

Commission

Page 25 GAO-02-52 Voting Equipment Standards

The following are GAO’s comments on the Federal Election Commission

letter dated July 18, 2001.

1. See comments 2, 5, and 6.

2. We do not dispute either the chronology of events provided in FEC’s

comments or its statement that it does not have explicit statutory

authority to develop and revise the standards. We provide the relevant

elements of this chronology in this report. Additionally, we state in this

report that FEC has assumed and is performing an important role by

developing and revising the standards, despite its lack of explicit

statutory responsibility.

We do not agree with FEC’s comment that it has been proactive in

updating the voting equipment standards. As FEC acknowledges in its

comments, 7 years elapsed from the time the standards were first

issued in 1990 to the time that FEC initiated efforts to assess the

standards to identify areas that needed to be updated. During that

time, considerable experience with the standards was accumulating, as

vendors were developing voting equipment to meet the FEC standards

and ITAs were testing against them. Since then, additional experience

has been gained with the standards as vendors have continued to

develop voting equipment to meet the standards, and ITAs have

continued to test vendors’ equipment against the standards. For

example, we state in this report that ITAs have had to interpret the

1990 standards in the testing process to accommodate vendors’ use of

new technologies and expanded equipment functions that are not

addressed in the 1990 standards. However, FEC does not formally

receive these interpretations, any one of which could be the basis for

prompting an update to the standards. In our view, waiting 9 years to

begin updating the standards is too long, does not constitute proactive

maintenance, and is the primary reason that the current standards are

out of date.

3. FEC is correct in stating that we did not assess all of the revised draft

standards areas. However, we disagree that this assessment approach

ignores the collaborative and dynamic process of NASED’s Voting

Systems Committee and FEC’s staff in overseeing the development of

the standards for two reasons. First, this report recognizes that FEC

worked closely with state and local election officials in revising the

standards. Second, this joint FEC and NASED process has no

relevance to our findings that certain standards areas do not address

GAO Comments
Appendix II: Comments From

the Federal Election

Commission

Page 26 GAO-02-52 Voting Equipment Standards

the full range of items associated with well-defined system

requirements in these areas. As we state in the objectives, scope, and

methodology section of this report, our approach was to assess all of

the 1990 standards because they are the standards against which

voting equipment are currently being developed and independently

tested. In assessing drafts of the updated standards, we assumed that

those areas in the 1990 standards that we found to be satisfactory

would continue to be satisfactory in the updated standards. As long as

our findings are limited to the standards area that we assessed, the

issue of whether we assessed all or some of the draft standards is not

relevant.

4. We acknowledge that FEC’s position, as stated in its comments,

concerning standards areas omitted from the revised draft standards is

that these were based on decisions resulting from deliberative and

collaborative interaction among NASED and FEC staff, and were not,

as we were told during the course of our review by the OEA Director,

areas that were overlooked. Accordingly, we have modified our report

to reflect this position.

5. Subsequent to providing us written comments on a draft of this report,

FEC provided us with a copy of volume II of the standards, which

includes the tests to be conducted to ensure compliance with the

voting equipment standards. Based on our review of the relevant

security sections, the standards satisfy the requirement for security

certification. We have modified our report, including the

recommendations, to reflect this new information.

6. We do not disagree that the draft standards discuss quality assurance

and have been strengthened from the 1990 standards. We acknowledge

these improvements to the standards in our report. However, as we

state, quality assurance includes a number of activities. While FEC’s

draft standards include some of these elements, they do not include all

of them. Specifically, the draft standards do not address requirements

for developing a quality assurance plan and conducting process

reviews to ensure that the product and process standards are followed.
Appendix III: System

Requirements and Supporting

Life-Cycle Processes and Our

Analysis

Page 27 GAO-02-52 Voting Equipment Standards

We identified 13 types of system requirements and 3 supporting life-cycle

processes that are often associated with complete system requirements.

FEC’s 1990 voting equipment standards satisfied 11 of the 13 system

requirements areas and none of the life-cycle processes. We reviewed

FEC’s draft standards for those areas for which we identified variances in

the 1990 standards and found that the draft standards had made

improvements in all five areas. However, the draft standards still do not

satisfy human factors and quality assurance. A detailed description of the

system requirements areas and our complete analysis follow.

1990 Draft

Systems requirements Definition/analysis Standards satisfied?

Functional/Performance Required system capabilities based on the purpose of the system; also

includes parameters for response times, accuracy, capacities,

unexpected/unallowed conditions, error-handling, and continuity of

operations.

1990 analysis: Identified areas include ballot definition,

candidate/measure selection, vote casting, ballot interpretation, voting

reports, accuracy and integrity, processing speed, response times, and

error and status messages.

Yes

System quality Quantitative measures of quality including reliability (perform correctly and

consistently), maintainability (easily serviced/repaired/corrected), and

availability (accessibility to be operated when needed).

1990 analysis: All identified areas included.

Yes

Security/privacy

protection

Requirements for maintaining a secure system and protecting data

privacy, including (1) security/privacy environment in which system must

operate; (2) types of security to be provided (e.g., data confidentiality and

fraud prevention); (3) risks the system must withstand; (4) safeguards

required; (5) security/privacy policies that must be met; (6) accountability

the system must provide (i.e., audit trails); and (7) criteria for security

certification.

1990 analysis: Access control identified as a security safeguard, and

requirements defined for audit records produced by the system to provide

accountability. The other areas, however, are not addressed.

Draft analysis: In addition to access controls and audit records, the

security/privacy environment, the types of security to be provided, the

risks the system must withstand, safeguards necessary, security policies,

and criteria for security certification are identified.

No Yes

Human factors Requirements defining system usability of the system that take into

account human capabilities and limitations, along with use and

accessibility by persons with disabilities.

1990 analysis: System usability and accessibility by persons with

disabilities are not identified.

Draft analysis: Requirements for the use and accessibility by persons

with disabilities are identified. System usability requirements are not.

No No

System external

interface

Characteristics of the interface between the voting system and other

systems, including data types, data formats, and timing.

1990 analysis: Removable storage media, communications devices, and

printers identified as external interfaces.

Yes

Appendix III: System Requirements and

Supporting Life-Cycle Processes and Our

Analysis
Appendix III: System

Requirements and Supporting

Life-Cycle Processes and Our

Analysis

Page 28 GAO-02-52 Voting Equipment Standards

1990 Draft

Systems requirements Definition/analysis Standards satisfied?

Installation-dependent data Requirements for system configuration to meet local operational

requirements.

1990 analysis: Requirements defined for voting systems programming in

accordance with ballot requirements of the election and the jurisdiction in

which the equipment will be used.

Yes

System environmental The natural environment that the system must withstand during

transportation, storage, and operation, including (1) temperature, (2)

humidity, (3) rain, and (4) motion/shock.

1990 analysis: Requirements identified for temperature, humidity, rain,

transit drop, and vibration.

Yes

Design constraints Any commercial standards that must be used in the system’s

development.

1990 analysis: Vendors are instructed to design equipment in

accordance with best commercial and industrial practice; software is to be

designed in a modular fashion, preferably using a high-level programming

language.

Yes

Physical The system’s physical characteristics, including size, weight, color,

nameplates, markings of parts and serial/lot numbers, transportability, and

parts interchangeability.

1990 analysis: All requirements identified.

Yes

Safety Requirements for preventing or minimizing unintended hazards to

personnel, property, and the physical environment.

1990 analysis: All systems shall be designed to meet the requirements of

the Occupational Safety and Health Administration Yes

Personnel Requirements for who will use or support the system, such as number of

workstations and built-in help/training features.

1990 analysis: Vendors instructed to include information on number of

personnel and skill level required to maintain the voting system.

Yes

Training Requirements for training devices and materials to be included with the

system.

1990 analysis: Vendors instructed to document information required for

system use and operator training, and orientation and training of poll

workers, user maintenance technicians, and vendor personnel.

Yes

Logistics Requirements for system maintenance, software support, and system

transportation.

1990 analysis: Vendors instructed to document information required in

these three areas.

Yes
Appendix III: System

Requirements and Supporting

Life-Cycle Processes and Our

Analysis

Page 29 GAO-02-52 Voting Equipment Standards

1990 Draft

Life-cycle process Definition/analysis Standards satisfied?

Documentation The process of recording information produced during the life-cycle

process. Describes and records information about a product, the

processes used to develop the product, and provides a history of what

happened during the development and maintenance of the product.

Includes (1) identification of documents to be produced and delivered to

customer or tester, (2) identification of format, content, and presentation

items for each document, and (3) review and approval process for each

document.

1990 analysis: Requirements identify products to be produced, including

the content and format of the documents. Review and approval process

not specified.

Draft analysis: Products to be produced, including the content and format

of the documents, as well as the review and approval process is identified.

No Yes

Configuration management The process to establish and maintain the integrity of work products

throughout the life-cycle process; it involves establishing product baselines

and systematically controlling changes to them. The process should

include (1) developing a configuration management plan, (2) identifying

work products to be maintained and controlled, (3) establishing a

repository to maintain and control the work products, (4) evaluating and

approving changes to the products, (5) accounting for changes to the work

products, and (6) managing the release and delivery of them.

1990 analysis: Includes requirements for (1) identifying work products to

be maintained and controlled, (2) evaluating and approving changes to the

products, and (3) managing the release and delivery of work products. The

standards do not include requirements for developing a configuration

management plan, establishing a repository to maintain and control the

work products, and accounting for changes to the work products.

Draft analysis: All areas identified.

No Yes

Quality assurance The process that provides adequate assurance of the system development

process. It typically involves independent review of work products and

activities to ensure compliance with applicable development standards and

procedures. The process should include (1) developing a quality

assurance plan, (2) determining system development product and process

standards to be followed, and (3) conducting reviews to ensure that the

product and process standards are followed.

1990 analysis: None of these areas specified.

Draft analysis: The need to document the hardware and software

development process is specified, but a quality assurance plan and quality

assurance reviews are not.

No No

Source: GAO analysis.

(310210)

The General Accounting Office, the investigative arm of Congress, exists to

support Congress in meeting its constitutional responsibilities and to help

improve the performance and accountability of the federal government for the

American people. GAO examines the use of public funds; evaluates federal

programs and policies; and provides analyses, recommendations, and other

assistance to help Congress make informed oversight, policy, and funding

decisions. GAO’s commitment to good government is reflected in its core values

of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents is through the

Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of

current reports and testimony and an expanding archive of older products. The

Web site features a search engine to help you locate documents using key words

and phrases. You can print these documents in their entirety, including charts and

other graphics.

Each day, GAO issues a list of newly released reports, testimony, and

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site

daily. The list contains links to the full-text document files. To have GAO E-mail

this list to you every afternoon, go to our home page and complete the easy-to-use

electronic order form found under “To Order GAO Products.”

The first copy of each printed report is free. Additional copies are $2 each. A

check or money order should be made out to the Superintendent of Documents.

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a

single address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

P.O. Box 37050

Washington, D.C. 20013

To order by Phone: Voice: (202) 512-6000

TDD: (301) 413-0006

Fax: (202) 258-4066

GAO Building

Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)

Washington, D.C. 20013

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm,

E-mail: fraudnet@gao.gov, or

1-800-424-5454 (automated answering system).

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G. Street NW, Room 7149,

Washington, D.C. 20548

GAO’s Mission

Obtaining Copies of

GAO Reports and

Testimony

Order by Mail or Phone

Visit GAO’s Document

Distribution Center

To Report Fraud,

Waste, and Abuse in

Federal Programs

Public Affairs