October 5, 2006

The Honorable Vernon Ehlers, Chairman

The Honorable Juanita Millender-McDonald, Ranking Member

Cornrnittee on House Administration

1309 Longworth Building

Washington, DC 20515

The Honorable Sherwood L.Boehlert, Chair

The Honorable Bart Gordon, Ranking Member

House Conunittee On Science

2320 Rayburn House OfficeBuilding

Washington, DC20515

(See list below letterfor additional recipients, including hearing witnesses.)

pear Members of the House Administration and ScienceCommittees, State Election Officials,

and Hearing Witnesses:

We are writing as practitioners of quality management for information production processes,

to offersome urgent comments and recommendations followingthe July 19, 2006 Joint

Hearing of the House Science and Administration Committees on "VotingMachines: Will New

Standards and Guidelines Prevent Future Problems?" (1)

This hearing effectivelyserved as a status update on efforts to improve elections under the

provisions of the 2002 Help America VoteAct. It sought to address the questions of how and

whether voting equipment standards and testing can help improve accuracy and security and

prevent errors and fraud. Spokespersons for the Election Assistance Commission (EAC),the

National Institute for Standards and Technology (NISI), State election officesfor Minnesota

and Maryland, voting technology vendors and a computer scientist speaking in an individual

capacity commented on and described plans to improve the Voluntary Voting System

Guidelines (VVSG)(2),to establish processes to certify testing labs and voting systems, to

encourage the adoption ofstandards by state election offices and voting technology vendors,

and to assist election officialsin improving voting processes.

These are all highly important tasks in improving elections. Our concerns relate to the need to

manage and understand the present impact ofthe changes that are being introduced into

election processes.

We have two main recommendations:

GThe VVSGincludes many requirements and tests for devices and software, but no

mention of measures ofthe voting process as a whole.A profound change in election

processes such asthe introduction of voting technology demcmds that the entire

voting process be placed under dependable, SOWtd quality control with

scifeguards to prevent voter error cmd vote tampering.

OWe strongly recommend election officialsprepare to assess the accuracy of election

outcomes in a transparent cmd publicly observable process. by comparing

election results against the results of a manual count of a statistical random sample of

ballots selected from the complete pool of cast ballots. Plans should address courses of

action if discrepancies are found. Jurisdictions that do not produce a manually readable

paper ballot that is verified by the voter before leaving the voting booth or that have

regulations restricting their use unfortunately will not be able to perfunn this

measurement, but should prepare to address questions about the basis for confidence in

the reliability of the election process.
1010812006 19:14

I

r

I

I

,

I

l

I

I

......

(212) 543-4266 Seth Johnson PAGE 3/8

Detailed Commentary:

Lack ofRecommendationfor Quality Control

The question of the accuracy of voting processes was repeatedly raised during this hearing on

technology standards. Yet responses to this question addressed it as an issue subject to

ongoing research, and no factual assessments of how election process changes have affected

the accuracy of election results were offered. Witnesses described ongoing research regarding

test procedures for testing labsto measure errors and performance of voting systems,

guidelines for design and usability and election management, independent verification

technologies, and the incorporation of a "paper traiL" But nowhere in the testimony or other

materials available from the EACor the NISTTechnical Guidelines DevelopmentCommittee

(TGDC)do we find a recommendation to establish quality control measures of the election

outputs to observe the impact ofthe changes being introduced by new voting technologies.

Quality measures of process outcomes are critical to enable detection of negative side-effects

that may be introduced by changes in a process, for addressing issues relating to questions of

accountability that often arise between subprocesses, and to provide a picture of how reliable a

process is as a whole. Guidelines by the EAC and NIST that seek to foster improvements in the

quality of elections must call for implementing such quality measures as a means of assuring

the process is in control. Guidelines must also call for designing quality into the voting

process to error-proof it against root causes of failure.

The processes presently established under HAVA are bringing about the replacement of

known-defective technology with new voting technology in elections throughout the country

without providing means to observe and control the potential negative impact that these

changes may cause. This approach (of introducing new technology without assessing,

controlling or improving the voting process) puts election processes further out of control in

many ways.

TIle Voluntary Voting System Guidelines Address Voting Technology-Not the

Process of Producing Election Information Accurately

While they speak ofsystems approaches, performance-based measures, and user-centered

design and human factors in usability research, the 2005 Voluntary Voting System Guidelines

are nevertheless focused on assessing the voting technology, and do not address the fact that

elections are complex information production processes, comprised of many other factors that

work together to produce vote counts (including such functions as determining eligibilityto

vote and capturing votes). Assessing the quality of election results is essential to

understanding the quality of the election process, not just addressing elements of the process in isolation.

In taking up this focus on the technology as such, NIST and the TGDCare addressing their

specific charter under HAVA. However, as stewards of the BaldrigeAward, NIST possesses a

strong capability in process excellencemethodology that would equip it to analyze and

recommend how to apply appropriate forms of quality planning, control and improvement to

processes that produce information in the critical and complex election process. In addition,

there is a broad community of ex'Pertsin the fieldwho apply quality principles to information

processes as a matter of their everyday practice.
1010812006 19:14 (212) 543-4266 Seth Johnson PAGE4f8 '"

lUanaging Election Information Quality

Many businesses have found that finding and correcting errors is unworkable as a way to

achieve reliable improvements in information quality, because information is collected too

rapidly and correcting errors after the fact is costly and difficult. This is particularly true for

elections, which operate under time constraints and for which the cost of -correcting errors

includes recounts or the conduct of new elections. The worst-case scenario is that election

process failure can cause the wrong person to be elected. Instead, the appropriate approach is

to focus on preventing information errors (in voter registration, recorded votes and vote

totals) revealed by factual assessment and by targeting their root caUSes.

The quality of complex information production processes is assured by designing quality in to

them this way. This approach is essential to error-proof and control election processes and

assure their integrity while fundamental changes are being introduced.

Automation and Accuracy

Simply adding automation to a process willnot assure its reliability and accuracy. While

automated devices generally execute programmed functions very consistently, the reliability of

an information production process depends on many more factors than the automation that

may be added to it. Election administration processes are complex information production

processes, with a technical side responsible for systems and applications, and a "business" side

that performs the rest of the process, including the operation of automated systems. Both

sides must work together to produce accurate voting information, and both sides must be held

accountable to the requirements for the product ofthe process - the vote count results. Even

assuming voting devicesfunction properly, the waysin which the introduction of automation

may affectthe election process gowell beyond the precision of the devices' functioning. Many

factors can introduce errors, including mismatching the designed ballot screens with the vote

recording data store, to the usability design of the ballot screens, to poorly written voter

instructions, to the vote counting procedures, just to name a few.

Usability Research and Accuracy: Applying principles of usability to voting devicesin

the vote capture step clearly promises reductions in the incidence of unclear or ambiguous

records of voter intent and thereby may bring about improvements overthe phenomena of

hanging chads, unreadable marks on optical ballots, unclear ballot designs and similar

problems that were on display in the 2000 general election in Florida. However, the promise

of introducing voting technology at any step does not guarantee error-free election processes,

and should not suggest that the reliability of elections is sufficientlyassured by introducing

automation alone. The 2004 elections revealed many different types of failure in automated

technologies. The 2006 elections saw electronic vote counts that were changed several times.

The truth is that the introduction of automation increases the need to manage the reliability

and accuracy of the election process as a whole.

Defining Accuracy: Accuracyis defined as "the degree to which data correctly reflects the

real world object or event being described" (3). Accuracyof the voting process means that the

captured and counted vote agrees with the intended vote ofthe voter. Accuracy is a measure

of the information produced by the process, not ofthe technology employed by the process. It

is not a measure that is performed electronically. Unlike charaderistics that may be

measured electronically, such as completeness of values, acceptable values, non-duplication,

timeliness, or validity according to business rules (4), a measure of accuracy entails comparing

of the electronic representation against the real world entity (or event) being represented.

Measuring Election Accuracy: For elections, which encompass a requirement of

anonymous voting, direct manual comparisons of individual electronic reCDrdsagainst cast

ballots or voter intent are not appropriate. However, accuracy may be assessed by comparing

vote total percentage results of the liveprocess against the percentage results of a manual

count of a representative random sample of human-readable ballots that have been verified by
1010812006 1914 (212) 543-4266 Set!1 Jo!1nson PAGE 5(8

the voter before leaving the voting booth. This is a measure of the accuracy of the election

process if both human-readable ballots and electronically recorded votes and vote totals are

maintained with control of the chain of custody, and a statistically representative sample of

ballots is selected randomly across all districts relevant to a contest.

This measure of accuracy will reveal the effects of defects in the process after the point of

casting the ballot, whether inadvertent or willful, if the electronic vote total percents vary from

the results of the accuracy measure by more than its statistical margin of error. Such a

discrepancy might trigger a manual recount. This accuracy assessment serves as a reliable

check of the integrity of election outcomes with respect to ballots cast. Identifying and

addressing root causes for discovered defects should be performed ultimately for any voting

anomaly causes found.

Security and Election Accur.acy

In response to a question about expanding the model presented by the California Secretary of

State's Voting Systems Technology Assessment Advisory Board, Dr. David Wagner commented

that while he believes that testing for reliability is at a level of readiness such that it may be

applied to the certification of voting systems, the status of security testing is still not adequate.

The hearing's charter acknowledges the deficiencies of the 2005 V{)luntary Voting System

Guidelines with respect to security testing, noting in particular special difficulties associated

with testing software due to its great degree of customization.

However, a properly conducted globalquality control measure assessing the accuracy of the

results produced by the process as a whole would serve very well as a means for assuring

security, since it would detect the effects ofsuccessful attempts, at any step after the capture of

the ballot, to fraudulently affectthe election outcome, ifthe effects are larger than the margin

of error ofthe accuracy measure relative to the election margin difference.

Observations Related to Hearing Participants' Comments

Managing Complexity: Indeed, such global measures of the quality of a product are

generally useful in the management of complex processes. House Science Committee

Chairman Shelwood Boehlert noted that as election processes are computerized, the things

that can go wrong become harder to recognize, fixand prevent, including security issues.

Establishing countermeasures for all potential types of defects that can arise with

computerized voting devices isintrinsically complex. But the key concern in managing the

impact of changes in the election process is the reliability ofthe process as a whole, in which

the devices are used. It is not enough to have "confidence" in computerized voting devicesin

and ofthemselves.

Quality principles provide the framework within which performance information may be used

to improve the process, as suggested in the questions posed by Representative Darlene Hooley,

though it is important to recognize that voting technology is onlyone factor in the overall

process of producing vote counts, and that the proper focus must be on the process and the

quality of its product. Testing of voting technology before, during and after eJections,

according to the recommendation of Britt Williams and others, and as mentioned by

Representative Juanita Millender-McDonald, does not place the process under control and

cannot address the fullrange of potential kinds of defects that may arise following the

introduction of voting technology into elections.

Quality methods enable the protection of the integrity of the entire process, asCommissioner

Donetta Davidson indicated was EAe's mandate under HAVA- and in particular given the way

they address the issue of security, these methods should be incorporated in election

management guidelines. The systems approach as described in the NISI's Human Factors

Report (5) and the comments ofthe ACCURATEgroup on the 20°5 WSG (6), is geared

toward modeling protocols to test the devices, and is not designed to assess how well all

...
1010812006 1914 (212) 543-4266 Seth Johnson PAGE 6/8

factors of the process work together to produce quality election results. It is therefore not

adequate in and of itself for the purpose of understanding and managing the impact of the

changes being introduced in the process. Likewise, the use of performance-based standards

and measures is appropriate only if they measure the full election process, not just the election

technology. The quality of a process as a whole such as elections is assessed through

performance measures that address the requirements that its product must meet. Accuracy of

the election result is the most important characteristic the election process must meet.

Both Minnesota Secretary of State Mary Kiffmeyer and Maryland Administrator of Elections

Linda Lamone expressed the need to incorporate recognition of the role of all factors in the

election process. Established methods for quality management ~ddress such aspects identified

by Ms. Lamone as the processes sun'ounding the technology and the role of people taking part

in administration (as well as systems development), while it also provides a basis for assuring

that elections are being performed well. It also provides safeguards against the increased risks

that Ms. Kiffmeyer noted are brought by the introduction of technology in the process. It

provides a framework for "wrapping the whole system" in the manner she described, enabling

the observation of the effect of defects in complex aspects of the process such as source code

and technology.

Representative Zoe Lofgren echoed Dr. Wagner's concerns about the lack of public reporting

and transparency on the part of the technology testing labs, and Representative Robert Ney

suggested that the relationship of the Election Assistance Commission with the labs might help

facilitate the use of independent assessments. In much the same manner that rigorously

performed quality measures can help manage complexity and address issues of security

(particularly as provided by a measure of accuracy), they can also serve to help obviate the

problems of bias and undue influence of vendors in technology testing and standards

development.

Bearing in mind our comments above generally, Dr. Wagner's recommendations, and his

citing of the recent Brennan Center recommendations (7) and those of the ACCURATE group

of which he is a member, are properly cognizant of the critical issues brought about by the

introduction of technology into election processes and are consistent with quality principles -

including feedback loops for continual improvement, broadening the focus of testing beyond

functional criteria to incorporate evaluations of security, reliability and usability,

strengthening usability and accessibility tests through professional methods, grounding

standards in the best scientific and engineering understanding, employing manually readable

votes to assess the process, and independently reviewing source code. We would add that

feedback and "field data" should address the process and the quality of the output that it

produces, and not solely address features of the process with respect to how they relate to the

quality of the technology in itself.

Voting Technology Certification Is Not Sufficient to Assure Election Integrity

Federal certification ofvoting technology under the 2005 Voluntary Voting SystemGuidelines

should not be regarded as establishing the appropriate controls for assuring the integrity of

elections as voting technology is added to the process. Elections produce informati<m. Voting

technology vendors produce voting technology. Among the criteria to which vendors should

be held accountable is the role ofthe technology in the quality of information that results from

the election as a whole,since they are participants in that process. However, in schematic

terms their role relates to systems development, though in many cases they may also be

engaged for a number of election administration services as well. The factors that contribute

to the quality of election processes differ on the "business"side from those on the development side.

,~
1010812006 19:14

,--I-,

(212) 543-4266 Seth Johnson PAGE 7(8

The bottom line is that the election processes must be managed and controlled from the

beginning ofthe value chain-voter registration-to the end result-dear confidence that the

election outcomes represent the intent of the voters.

We hope that you will address these concerns expeditiously, as currently we see the election

process in the United States at grave risk.

Sincerely,

(Thefollowing list their names in support of the above statement Affiliations are listedfor

identificahon only.)

Larry English, Election Assessment Advisor; Author, "Information Quality Mandate for

Election Reform"

Seth Johnson, Information Quality Improvement Gmsultant

Christy Bryant, Six Sigma Black Belt

Robert Fragola, VP Sales and Marketing, ChoiceMaker Technologies, Inc.

Matthias Groh, Six Sigma Master Black Belt

Raymond C. Hager, Certified Data Management Professional

David C. Hay, President, Essential Strategies, Inc.

Bruce McTavish, Data Architect; Past VP, Seattle Chapter of the Data MaJ1agement

Association

David Rafner, VP of Industry Relations, DAMA International

Dawn M. Wolthuis, President, Tincat Group, Inc.

Hearing Witnesses:

Donetta Davidson, Commissioner, Election Assistance Commission

John S. Groh, Chairman, Election Technology Council, Information Technology

Association of America

Dr. William Jeffrey, Director, National Institute of Standards and Technology

Mary Kiffmeyer, Secretary of State, Minnesota

Linda Lamone, Administrator of Elections, State Board of Elections, Maryland

Dr. David Wagner, Professor of Computer Science, University of California, Berkeley

cc: The Honorable Rush Holt, United States Representative for the State of New Jersey

State Election Officials, Uhited States of America

National Voluntary Laboratory Accreditation Program

Malcolm Baldrige National Quality Award

Brennan Center for Justice at NYU Law School

A Center for Correct, Usable, Reliable, Auditable, and Transparent Elections

Council of State Governments

National Governors Association

National Conference of State Legislatures

National Association of State Election Directors

National Association of Secretaries of State

National Lieutenant Governors Association

National Association of Attorneys General

National Association of Counties

National Association of County Recorders, Election Officials and Clerks

International Association of Clerks, Recorders, Elections Officials and Treasurers

United States Conference of Mayors

United States Commission on CivilRights

Federal Voting Assistance Program

Voting Section, CivilRights Division, United States Department of Justice

Office of Public Integrity, Criminal Division, United States Department of Justice
1010812006 19:14 (212) 543-4266 Seth JtJhnson PAGE 8/8

Please Contact:

Seth Johnson

275 Fort Washington Avenue, Suite 3C

New York, NY 10032

(212) 543-4266

Notes:

(1) Hearing testimony may befound at:

http.//www .house.qov/science/hearinqs/full06/Juh./%2019/index htm

(2) The Voluntary Voting System Guidelines may befound at.

http://Quidelines.kennesaw.edu/vvsQ/intro.asp

(3) See Larry English, "D(}finingand Measuring Accuracy",

http. / /www.infoimpact.com/articles/DMR 7.o8DefininqandM easurinqAccuraclJ .Pdt

(4) See Larry English, "Improving Data Warehouse and Business Information Quality", Wiley &Sons,

1999, pp. 178-179

(5) See National Institute of Standards and Technology Special Publication 500-256, 'Improving the

Usability and ACceSSIbilityof Voting Systems and Products",

http. / /vote. nist. Qov/Final%20H uman %20Factors%20Report%20%20S -04.pdt

(6) See A Centerfor Correct, Usable, Reliable, Auditable, and Transparent Elections, "PublicComment on

the 2005 Voluntary Voting System Guidelines", http://accuratevotinq.orq/accurate/docs/200s vvsq commentpdf

(7) See Brennan Center Task Force on Voting System Security, 'The Machinery qf Democracy: Protecting

Elections in an Electronic World",

l1tJJ2.J-13~lW_lliJ?L§]]ng1l9§}]..tif!Lq[gj1!LQqrc-q1JYj.lg9J!!.IJlQgJ!§.lfg[lJQgQE!12QEL12g[

,-