Elections: Results of GAO's Testing of Voting Systems Used in Sarasota County in Florida's 13th Congressional District
Published: 2008-02-08T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In November 2006, about 18,000 undervotes were reported in Sarasota County in the race for Florida's 13th Congressional District (Florida-13). After the election results were contested in the House of Representatives, the task force unanimously voted to seek GAO's assistance in determining whether the voting systems contributed to the large undervote in Sarasota County. In October 2007, GAO presented its findings on the review of the voting systems and concluded that while prior tests and reviews provided some assurance that the voting systems performed correctly, they were not enough to provide reasonable assurance that the voting systems in Sarasota County did not contribute to the undervote. GAO proposed that a firmware verification test, a ballot test, and a calibration test be conducted. The task force requested that GAO proceed with the proposed additional tests. GAO also verified whether source code escrowed by Florida could be rebuilt into the firmware used in Sarasota County. To conduct its work, GAO conducted tests on a sample of voting systems used in Sarasota County during the 2006 general election. GAO witnessed the rebuild of the firmware from the escrowed source code at the manufacturer's development facility. GAO reviewed test documentation from Florida, Sarasota County, and the voting system manufacturer and met with election officials to prepare the test protocols and detailed test procedures.
section_title: What GAO Found
paragraphs: GAO conducted three tests on the iVotronic Direct Recording Electronic (DRE) voting systems in Sarasota County and these tests did not identify any problems. Based on its testing, GAO obtained increased assurance that the iVotronic DREs used in Sarasota County during the 2006 general election did not contribute to the large undervote in the Florida-13 contest. Although the test results cannot be used to provide absolute assurance, GAO believes that these test results, combined with the other reviews that have been conducted by the State of Florida, GAO, and others, have significantly reduced the possibility that the iVotronic DREs were the cause of the undervote. GAO's firmware verification test showed that the firmware installed in a statistically selected sample of 115 machines used by Sarasota County during the 2006 general election matched the firmware certified by the Florida Division of Elections. The statistical approach used in selecting these machines lets GAO estimate with a 99 percent confidence level that no more than 60 of the 1,499 iVotronic DREs that recorded votes in the 2006 general election were using different firmware. Consequently, GAO is able to place more confidence in the results of other tests conducted on a small number of machines by GAO and by others, which indicated that the iVotronic DREs did not cause the undervote. GAO also confirmed that when the manufacturer rebuilt the iVotronic DRE firmware from the source code that was held in escrow by the Florida Division of Elections and previously reviewed by GAO and others, the resulting firmware matched the version certified by the Florida Division of Elections. For the ballot test, GAO cast predefined test ballots on 10 iVotronic DREs and confirmed that each ballot was displayed and recorded accurately. GAO conducted the calibration test by miscalibrating two iVotronic DREs and casting ballots on them to validate that the machines recorded the information that was displayed on the touch screen. Based on the results of the ballot and calibration tests, GAO found that (1) the machines properly displayed, recorded, and counted the selections for all test ballots cast during ballot testing involving 112 common ways a voter may have interacted with the system, and (2) the deliberately miscalibrated machines, though difficult to use, accurately recorded the ballot selections as displayed on screen. At this point, GAO believes that adequate testing has been performed on the voting machine software and does not recommend further testing in this area. Given the complex interaction of people, processes, and technology that must work effectively together to achieve a successful election, GAO acknowledges the possibility that the large undervote in Florida's 13th Congressional District race could have been caused by factors such as voters who intentionally undervoted, or voters who did not properly cast their ballots on the iVotronic DRE, potentially because of issues relating to interaction between voters and the ballot.

section_title: Letter
paragraphs: I am pleased to appear before the task force today to present the findings  on our testing of the voting equipment used in the 2006 general election in  Florida’s 13th Congressional District (Florida-13). I would like to thank the  task force for its overall support of our efforts and specifically for the  assistance provided in obtaining resources from the House Recording  Studio that were critical to successfully completing our testing efforts.
In November 2006, about 18,000 undervotes were reported in Sarasota  County in the race for Florida’s 13th Congressional District. After the  election results were contested in the House of Representatives, the task  force met and unanimously voted to seek GAO’s assistance in determining  whether the voting systems contributed to the large undervote in Sarasota  County. In our October 2, 2007, statement for the task force, we presented  the findings of our review of the voting systems and stated that while prior  tests and reviews provided some level of assurance that the voting systems  in Sarasota County—iVotronic direct recording electronic (DRE) voting  systems manufactured by Election Systems and Software (ES&S)— functioned correctly, they were not enough to provide reasonable  assurance that the iVotronic DRE voting systems did not contribute to the  undervote. Specifically, we found that assurance was lacking in three  areas and proposed to the task force that additional tests—firmware  verification, ballot, and calibration—be conducted to address these areas.  We stated that successful accomplishment of these tests would provide  increased, but not absolute, assurance that the iVotronic DREs used in  Sarasota County during the 2006 general election did not cause the  undervote. The task force requested that we proceed with the proposed  additional tests. Our objectives were to (1) verify that firmware installed in  a statistical sample of iVotronic DREs was identical to the firmware  certified by the State of Florida, (2) perform ballot testing using 112 ways  to cast a ballot for the Florida-13 contest to ensure that the voting  machines would properly record and count the ballots, and (3)  deliberately miscalibrate voting machines and then cast ballots on those  machines to ensure that the voting machines would properly record the  ballots. As part of the first objective, we also validated that the source  code, which was held in escrow by the Florida Division of Elections,  would produce the firmware used by Sarasota County during the 2006  general election.
To conduct our tests, we developed test protocols and detailed test  procedures. We met with officials from the Sarasota County Supervisor of  Elections, the Florida Department of State and Division of Elections, and  ES&S to obtain the necessary details about the voting systems and prior  tests to document our test procedures. We also reviewed voting system  documentation to develop a testing approach and the test procedures. To  ensure that the certified firmware held in escrow by the Florida Division  of Elections corresponded to the source code that was reviewed by a team  from Florida State University and us, on November 19, 2007, we visited  ES&S’s development facility in Rockford, Illinois, and witnessed the  rebuild of the firmware from the escrowed source code.
Further details on our test methodology are included in the following  sections on each of the three tests. Appendix I outlines the process used to  select machines for testing, and appendix II lists the iVotronic DREs that  we tested. We coordinated with the Florida Division of Elections and the  Sarasota County Supervisor of Elections to obtain access to the iVotronic  DREs and other necessary test equipment to conduct our testing. We  conducted the firmware verification, ballot, and calibration tests at the  Sarasota County Voting Equipment Facility (VEF) in Sarasota, Florida. We  established the test environment on November 26, 2007, and conducted  the tests from November 27, 2007, to December 4, 2007. During this time,  we completed the steps necessary to conduct the tests and collected the  test data. In addition, we video recorded the tests.  One camera was used  to capture a wide angle shot of the test room. Other cameras recorded the  conduct of the firmware verification, ballot, and calibration tests.
We provided a draft of this statement to the Florida Department of State  and ES&S for their review and comments. We briefed the Sarasota County  Supervisor of Elections on the contents of our statement. The Florida  Department of State and ES&S also conducted a sensitivity review to  ensure that business proprietary information is not disclosed in this  statement. We conducted our work from October 2007 to February 2008 in  Washington, D.C.; Tallahassee and Sarasota, Florida; and at ES&S facilities  in Rockford, Illinois, and Omaha, Nebraska.
subsections: section_title: Background
paragraphs: The 13th Congressional District of Florida comprises DeSoto, Hardee,  Sarasota, and parts of Charlotte and Manatee Counties. In the November  2006 general election, there were two candidates in the race to represent  the 13th Congressional District: Vern Buchanan, the Republican candidate,  and Christine Jennings, the Democratic candidate. The State of Florida  certified Vern Buchanan the winner of the election. The margin of victory  was 369 votes out of a total of 238,249 votes counted. Table 1 summarizes  the results of the election and shows that the results from Sarasota County  exhibited a significantly higher undervote rate than in the other counties in  the congressional district.
As seen in table 1, about 18,000 undervotes were reported in Sarasota  County in the race for Florida’s 13th Congressional District. After the  election results were contested in the House of Representatives, the task  force met and unanimously voted to seek GAO’s assistance in determining  whether the voting systems contributed to the large undervote in Sarasota  County. On June 14, 2007, we met with the task force and agreed upon an  engagement plan. We reported on the status of our review at an interim  meeting held by the task force on August 3, 2007.
On October 2, 2007, we reported that our analysis of election data did not  identify any particular voting machines or machine characteristics that  could have caused the large undervote in the Florida-13 race. The  undervotes in Sarasota County were generally distributed across all  machines and precincts. We found that some of the prior tests and reviews  conducted by the State of Florida and Sarasota County provided assurance  that certain components of the voting system in Sarasota County  functioned correctly, but they were not enough to provide reasonable  assurance that the iVotronic DREs did not contribute to the undervote. We  proposed three tests—firmware verification, ballot, and calibration—to  provide increased assurance, but not absolute assurance, that the  iVotronic DREs did not cause the large undervote in Sarasota County. We  stated that the successful conduct of the tests could reduce the possibility  that the voting systems caused the undervote and shift attention to the  possibilities that voters intentionally undervoted or voters did not properly  cast their ballots on the iVotronic DRE, potentially because of issues  relating to interaction between voters and the ballot.
subsections: section_title: Overview of the Voting Systems Used in Sarasota County in the 2006 General Elections
paragraphs: In the 2006 general election, Sarasota County used voting systems  manufactured by ES&S. The State of Florida has certified different  versions of ES&S voting systems. The version used in Sarasota County was  designated ES&S Voting System Release 4.5, Version 2, Revision 2, and  consisted of iVotronic DREs, a Model 650 central count optical scan  tabulator for absentee ballots, and the Unity election management system.  It was certified by the State of Florida on July 17, 2006. The certified  system includes different configurations and optional elements, several of  which were not used in Sarasota County.
The election management part of the voting system is called Unity; the  version that was used was 2.4.4.2. Figure 1 shows the overall election  operation using the Unity election management system and the iVotronic  DRE.
Sarasota County used iVotronic DREs for early and election day voting.  Specifically, Sarasota County used the 12-inch iVotronic DRE, hardware  version 1.1 with firmware version 8.0.1.2. Some of the iVotronic DREs are  configured to use audio ballots, which are often referred to as Americans  with Disabilities Act (ADA) machines. The iVotronic DRE uses a touch  screen—a pressure-sensitive graphics display panel—to display and  record votes (see fig. 2).
The machine has a storage case that also serves as the voting booth. The  operation of the iVotronic DRE requires the use of a personalized  electronic ballot (PEB), which is a storage device with an infrared window  used for transmission of ballot data to and from the iVotronic DRE. The  iVotronic DRE has four independent flash memory modules, one of which  contains the program code—firmware—that runs the machine; the  remaining three flash memory modules store redundant copies of ballot  definitions, machine configuration information, ballots cast by voters, and  event logs (see fig. 3). The iVotronic DRE includes a VOTE button that the  voter has to press to cast a ballot and record the information in the flash  memory. The iVotronic DRE also includes a compact flash card that can  be used to load sound files onto iVotronic DREs with ADA functionality.  The iVotronic DRE’s firmware can be updated through the compact flash  card. Additionally, at the end of polling, the ballots and audit information  are to be copied from the internal flash memory module to the compact  flash card.
To use the iVotronic DRE for voting, a poll worker activates the iVotronic  DRE by inserting a PEB into the PEB slot after the voter has signed in at  the polling place. After the poll worker makes selections so that the  appropriate ballot will appear, the PEB is removed and the voter is ready  to begin using the system. The ballot is presented to the voter in a series of  display screens, with candidate information on the left side of the screen  and selection boxes on the right side (see fig. 4).
The voter can make a selection by touching anywhere on the line, and the  iVotronic DRE responds by highlighting the entire line and displaying an X  in the box next to the candidate’s name. The voter can also change his or  her selection by touching the line corresponding to another candidate or  by deselecting his or her choice. “Previous Page” and “Next Page” buttons  are used to navigate the multipage ballot. After completing all selections,  the voter is presented with a summary screen with all of his or her  selections (see fig. 5). From the summary screen, the voter can change any  selection by selecting the race. The race will be displayed to the voter on  its own ballot page. When the voter is satisfied with the selections and has  reached the final summary screen, the red VOTE button is illuminated,  indicating the voter can now cast his or her ballot. When the VOTE button  is pressed, the voting session is complete and the ballot is recorded on the  iVotronic DRE. In Sarasota County’s 2006 general election, there were nine  different ballot styles with between 28 and 40 races, which required  between 15 and 21 electronic ballot pages to display, and 3 to 4 summary  pages for review purposes.
section_title: Election Systems Involve People, Processes, and Technology
paragraphs: An election system is based upon a complex interaction of people (voters,  election officials, and poll workers), processes (controls), and technology  that must work effectively together to achieve a successful election. The  particular technology used to cast and count votes is a critical part of how  elections are conducted, but it is only one facet of a multifaceted election  process that involves the interplay of people, processes, and technology.
As we have previously reported, every stage of the election process— registration, absentee and early voting, preparing for and conducting  Election Day activities, provisional voting, and vote counting—is affected  by the interaction of people, processes, and technology. Breakdowns in  the interaction of people, processes, and technology may, at any stage of  an election, impair an accurate vote count. For example, if the voter  registration process is flawed, ineligible voters may be allowed to cast  votes. Poll worker training deficiencies may contribute to discrepancies in  the number of votes credited and cast, if voter information was not  entered properly into poll books. Mistakes in using the DRE systems could  result from inadequate understanding of the equipment on the part of  those using it.
As noted in our October statement, we recognize that human interaction  with the ballot layout could be a potential cause of the undervote, and we  noted that several suggestions have been offered as possible ways to  establish that voters are intentionally undervoting and to provide some  assurance that the voting systems did not cause the undervote. For  instance,    A voter-verified paper trail could provide an independent confirmation  that the touch screen voting systems did not malfunction in recording  and counting the votes from the election. The paper trail would reflect  the voter’s selections and, if necessary, could be used in the counting  or recounting of votes. This issue was also recognized in the source  code review performed by the Security and Assurance in Information  Technology (SAIT) laboratory at Florida State University as well as the  2005 and draft 2007 Voluntary Voting Systems Guidelines prepared for  the Election Assistance Commission. We have previously reported on  the need to implement such a function properly.
Explicit feedback to voters that a race has been undervoted and a  prompt for voters to affirm their intent to undervote might help prevent  many voters from unintentionally not casting a vote in a race. On the  iVotronic DREs, such feedback and prompts are provided only when  the voter attempts to cast a completely blank ballot, but not when a  voter fails to vote in individual races.
Offering a “none of the above” option in a race would provide voters  with the opportunity to indicate that they are intentionally undervoting.  For example, the State of Nevada provides this option in certain races  in its elections.
We reported that decisions about these or other suggestions about ballot  layout or voting system functions should be informed by human factors  studies that assess such measures’ effectiveness in accurately recording  voters’ preferences, making voting systems easier to use, and preventing  unintentional undervotes.
section_title: Tests Confirm Sarasota County iVotronic DREs Used Same Firmware Certified by Florida
paragraphs: We previously reported that having reasonable assurance that all iVotronic  DREs that recorded votes in the 2006 general election were running the  same certified firmware would allow us to have more confidence that the  iVotronic DREs will behave similarly when tested. Consequently, if we  are reasonably confident that the same firmware was running in all 1,499  machines, then we are more confident that the results of other tests,  conducted both by GAO and by others, on a small number of machines can  be used to obtain increased assurance that the iVotronic DREs did not  cause the undervote. We also reported that there was a lack of assurance  that the source code that was held in escrow by the Florida Division of  Elections and that was previously reviewed by Florida State University  and by us, if rebuilt, would corresponded to the firmware that was  certified and held in escrow by the Florida Division of Elections. We found  that the firmware on a statistically selected sample of 115 iVotronic DREs  was the same as that certified by the Florida Division of Elections. We also  found that the escrowed source code, when rebuilt into executable  firmware, corresponded to the 8.0.1.2 firmware that was certified by the  Florida Division of Elections.
subsections: section_title: Methodology for Firmware Verification Testing
paragraphs: Our methodology to obtain reasonable assurance that the firmware used  on Sarasota County’s iVotronic DREs during the 2006 general election was  the same as that certified by the State of Florida was broken down into  two basic steps: (1) selecting a representative sample of machines, and (2)  verifying that the firmware extracted from the voting machines was the  same as the escrowed firmware that had been certified by the Florida  Division of Elections. Appendix I details the methodology for selecting the  representative sample of machines. Appendix II contains a list of the serial  numbers of the tested iVotronic DREs.
To ensure that we would be testing with the iVotronic firmware certified  by the Florida Division of Elections, on October 18, 2007, we and officials  from the Florida Division of Elections made two copies of the escrowed   iVotronic 8.0.1.2 firmware on compact discs (CD) and placed them in two  tamper-evident bags with serial numbers. The bags were subsequently  hand-delivered by a Florida Division of Elections official for our use in the  firmware verification test and for the rebuilding of the firmware from the  source code.
In order to extract the firmware from an iVotronic DRE, the machine was  placed on an anti-static mat and the case was opened using a special  screwdriver. After lifting the case, a special extraction tool was used to  remove the flash memory module that contains the firmware. The flash  memory module was then inserted in the socket of a Needham  Electronics’ EMP-300 device that was connected to the universal serial bus  (USB) port of a personal computer (PC). The EMPWin application running  on that PC was used to read the firmware from the flash memory module  and save the extracted firmware on the PC. The Florida Division of  Elections loaned us the EMP-300 and EMPWin application for use in  extracting firmware from the flash memory module.
To compare the extracted firmware with the escrowed version, we relied  on two commercially available software programs. First, we acquired a  license for PrestoSoft’s ExamDiff Pro software that enables comparison of  files. The ExamDiff Pro software is a commercially available program  designed to highlight the differences between two files. For each selected  iVotronic DRE, the extracted firmware was compared with the escrowed  version with any differences highlighted by the program.
Second, to further ensure that the extracted firmware matched the  escrowed firmware, we compared the SHA-1 hash value of the extracted  firmware to the hash value of the comparable certified firmware. We  computed the SHA-1 hash by using the Maresware hash software that was  provided by the Florida Division of Elections. In order to ensure that the  commercial Maresware hash software properly calculated the SHA-1 hash  value, we (1) created four files and obtained a fifth file that contained  executable code, (2) obtained hash values for each file by either using an  external program that generated the hash values using the same hashing  algorithm as the commercial product or using known hash values, and (3)  used the commercial program acquired for testing the firmware to ensure  that the hash values it generated for these five files were identical to the  expected hash values for those files. In each case, the hash values  generated by the commercial program were identical to the expected  values. Accordingly, reasonable assurance for the purposes of our review  was obtained that the commercial program produced its hash values in  accordance with the NIST algorithm.
At the end of each day, we (1) used the commercial Maresware software to  compute hash values for each of the firmware programs that had been  unloaded during that day and all previous days, and (2) compared each  hash created by this program to the expected value that was calculated  from the firmware that had been escrowed by the Florida Division of  Elections. This comparison provided further assurance that the extracted  firmware was (1) identical to the version escrowed by the Florida Division  of Elections when the hashes agreed, or (2) different if the hashes did not  agree.
We also verified that sequestered machines were not used since the 2006  general election. For each of these sequestered machines, we used an  audit PEB to copy the audit logs onto a compact flash card and then used  the Unity election reporting manager to generate event log reports. We  examined the event logs for the date and time of occurrence of activities  that would indicate whether the machine had been used. Lack of such  activities since the 2006 general election provided reasonable assurance  that the machines had not been used since they were sequestered.
In addition, to verify that the source code for iVotronic DRE firmware  version 8.0.1.2 previously examined by the Florida State University SAIT  source code review team and by GAO corresponded with the version  certified by the Florida Division of Elections, ES&S officials stated that it  still had the development environment that could be used to compile, or  rebuild, the certified firmware from the source code retained in escrow by  the Florida Division of Elections. As we previously noted, a software  review and security analysis of the iVotronic DRE firmware was  conducted by a team led by Florida State University’s SAIT laboratory.  The software review team attempted to confirm or refute many different  hypotheses that, if true, might explain the undervote in the race for the  13th Congressional District. In doing so, they made several observations  about the source code, which we were able to independently verify.
The rebuilding of the firmware was conducted by ES&S at its Rockford,  Illinois, facility on November 19, 2007, and witnessed by us. Prior to the  rebuild, the Florida Division of Elections provided an unofficial copy of  the source code to ES&S so that ES&S could prepare the development  environment and test the rebuild steps. Using the official sealed copy of  the source code CD, ES&S rebuilt the firmware in front of GAO  representatives. ES&S described the development environment and we  inspected it to satisfy ourselves that the firmware was faithfully rebuilt  using the escrowed source code. After the rebuilding of the firmware, the  certified version of 8.0.1.2 firmware was compared with the rebuilt version  using PrestoSoft’s ExamDiff Pro.
section_title: Results of Firmware Verification Testing
paragraphs: While the Florida audit team had previously confirmed that the firmware  running on six iVotronic DREs matched the certified version held in  escrow by the Florida Division of Elections, we found that the sample size  was too small to support generalization to all 1,499 iVotronic DREs that  recorded votes during the 2006 general election. Accordingly, we  conducted a firmware verification test on a statistically valid sample of 115  iVotronic DRE machines used by Sarasota County during the 2006 general  election. The selected machines fell into two groups—machines that had  not been used since the 2006 general election (referred to as sequestered  machines) and machines that had been used in subsequent elections. For  each machine, we extracted the firmware from a flash memory module in  that machine and then compared the extracted firmware with the  escrowed version using commercially available file comparison tools to  determine whether they agreed. We found that the firmware installed in  the flash memory module of each machine matched the escrowed  firmware that had been certified by Florida. The statistical approach used  to select these machines lets us estimate with a 99 percent confidence  level that at least 1,439, or 96 percent, of the 1,499 machines used in the  2006 general election used the firmware that was certified by the State of  Florida.
We witnessed the rebuild of the iVotronic DRE’s firmware from the source  code that was held in escrow by the Florida Division of Elections and that  was previously reviewed by Florida State University and by us. At ES&S’s  software development facility, we observed that rebuilding the firmware  from the escrowed source code resulted in the same firmware that was  certified and held in escrow by the Florida Division of Elections. The  comparison of the escrowed firmware to the version that was rebuilt by  the vendor identified no differences and provides us reasonable assurance  that the escrowed firmware corresponded to the escrowed source code.  The successful rebuilding of the firmware from the escrowed source code  enables us to have greater confidence in the conclusions derived from  prior source code reviews by Florida State University and us.
section_title: Ballot Testing Showed That Machines Accurately Recorded and Counted Ballots
paragraphs: In our October 2007 statement, we noted that there were 112 common  ways a voter may interact with the system to select a candidate in the  Florida-13 race and cast the ballot, and that prior testing of the iVotronic  DREs covered only 13 of these 112 possible ways. We developed 224 test  ballots to verify that the iVotronic DRE could accurately capture ballots  using each of these 112 common ways a voter may interact with the  system; 112 test ballots were cast on one machine configured for early  voting, and another 112 ballots were cast on nine machines configured for  election day voting. Our tests showed that for each of the 224 test ballots,  the iVotronic DRE correctly captured each vote as cast for the Florida-13  race. We also conducted firmware verification tests on these machines and  verified that they were running the certified firmware.
subsections: section_title: Methodology for Ballot Testing
paragraphs: The methodology for ballot testing can be broken into two major areas— development of the test ballots and execution of the test using those  ballots. The following sections discuss these areas.
subsections: section_title: Development of Test Ballots
paragraphs: In examining how the system allowed voters to make a selection in the  Florida-13 race, we found at least 112 different ways a voter could make  his or her selection and cast the ballot in the Florida-13 race, assuming  that it was the only race on the ballot. Specifically, a voter could (1)  initially select either candidate or neither candidate (i.e., undervote), (2)  change the vote on the initial screen, and (3) use a combination of features  to change or verify his or her selection by using the page back and review  screen options. Accordingly, we tested these 112 ways to select a  candidate on the early voting machine and on the election day machines  (224 test ballots in total).
The 112 standard test ballots cover all combinations of the following types  of voter behavior:    Voter makes selection on the initial ballot screen and makes no  changes or takes any other action to return to the contest to review or  change selection.
Voter makes selection on the initial ballot screen and decides before  leaving that screen to change the selection because of an error in  selecting the candidate or for some other reason.
Voter makes selection on the initial ballot screen and then decides to  use the page back option to review or change selection.
Voter makes selection on the initial ballot screen and continues to the  review screen and then decides to use the review screen option to  review or change selection.
Voter makes selection on the initial ballot screen and uses a  combination of page back and review screen options to review or  change selection.
In each instance where a selection could be made, three choices were  possible for the Florida-13 race: a selection for one of the two candidates,  or no selection (i.e., an undervote).
In developing the standard test ballots, we did not consider all  combinations of some other types of voter behavior that would have  significantly increased the number of test cases without providing  significant benefits. In most cases, such behavior are variants of the  primary voter behavior that we examined. The following are examples of  voter behavior that were not included in the standard test set in order to  reduce the number of test cases to practicable levels:    Using a one-touch or two-touch method to make changes on a ballot  page.
Varying the number of pages a voter may go back (“page backs”) to  return to the page containing the Florida-13 race to change or review a  selection.
Casting a ballot from the review screen selection. The VOTE button is  not activated until the voter reaches the last review screen. However,  once the VOTE button has been activated, a ballot may be cast from  any screen. For example, a voter may activate the VOTE button and  then return to a contest to review or change the selection using the  review screen option. Once the voter goes to the contest from the  review screen and makes any desired changes, the voter can then cast  the ballot from that screen rather than going back to the last page of  the review screen or even the review screen that was used to return to  the selection.
Although we did not consider all combinations of these types of voter  behavior when developing the standard test ballots, we included some of  these user interactions in the execution of applicable test ballots to  provide increased assurance that the system would handle these voter  behaviors. For each applicable test ballot, we randomly determined the  test procedure that should be used for the following attributes:    Initial change method – The standard test ballots address voters  making changes on the initial ballot screen. Where possible, the  method used to change (one-touch or two-touch) the selection was  randomly selected.
Number of page backs – The ballots used by Sarasota County  included the page back function. After reviewing the ballots, it  appeared reasonable to expect that voters who may have used the page  back option would probably decide that they had missed the race by  the time they went one or two pages beyond the page with the  Florida-13 race. Therefore, when a standard test ballot contained a  page back requirement, the number of page backs was randomly  selected to determine whether one or two page backs should be used.
Page back change method – Some test ballots required a change after  the page back option was selected. As with the initial change method,  where possible, the method of changing (one-touch or two-touch) the  selection was randomly assigned.
Review screen change method – The system displays a review  screen that shows the voter’s selections (or lack of selections) after the  voter has progressed through all contests. On the review screen, the  voter can select a race to go directly to that contest and (1) review the  selection made, and (2) make any desired corrections. The standard  test ballots were designed to cover this type of event. Where possible,  the method used to make the change (one-touch or two-touch) was  randomly selected.
Activate VOTE button and cast ballots from the review screen –  In order to test casting ballots from locations other than the last review  screen, the VOTE button must be activated prior to going to a screen  where the ballot is cast. In order to determine which test ballots  should be used for this test, a two-step approach was adopted. First, a  random selection of the ballots that use the review screen option was  made to determine which test ballots should have the VOTE button  activated. Then a random selection of these test ballots was made to  determine whether the ballot should be cast from the review screen  selection.
Besides those attributes that directly affect the selection in the Florida-13  race, we varied the other attributes on the ballot in order to complete the  ballot test. For each of the 224 test ballots, we used random values for  other attributes, including the following:    Ballot style – Each ballot was randomly assigned one of the nine  ballot styles used in the election.
Write-in candidate – All ballot styles includes write-in options in at  least 2 races —United States Senate and State Governor/Lieutenant  Governor. To verify that the iVotronic DRE accurately recorded the  selection in the Florida-13 race for each test ballot, we needed a way to  identify each test ballot in the ballot image log. To accomplish this, we  randomly selected one of these two races, selected the write-in  candidate for the race, and entered a unique value (i.e., the test ballot  number) in the write-in field.
Candidates and selections in other races on the ballot – Each  ballot style had between 28 and 40 contests on the ballot. The values  for the contests besides the Florida-13 race and the write-in field were  also randomly selected. For example, most items had three possible  choices—candidate 1 (or Yes), candidate 2 (or No), and undervote.  Which of these three values was used for a given contest was randomly  determined.
The values used for these attributes were independently determined for  the election day and early voting test ballots. For example, Test Ballot 2  (election day) and Test Ballot 202 (early voting) were designed to test the  same standard condition described by one of the 112 standard test  ballots. Table 2 illustrates some of the similarities and differences  between the two test ballots that result from the random selection process  used to determine the other aspects of the ballot.
Finally, we selected 10 random machines to be used for the ballot testing.  One machine was selected from those that were used in early voting in the  2006 general election. The other nine were selected from those that used  each of the ballot styles on election day in the 2006 general election. For  each election day machine, the assigned precinct was the same as the  precinct where the machine was used during the 2006 general election.  For the early voting machine, we needed to assign precincts for each  ballot style. We used the precinct associated with the back-up machine  used for election day testing as the precinct for that ballot style. If the  first back-up machine was assigned the same precinct number as the  primary election day machine, then we used the precinct associated with  the second back-up machine. This approach was taken to maximize the  number of precincts used in the testing efforts.
section_title: Process Used in Executing the Ballot Test
paragraphs: A two-person test team conducted the ballot testing. One tester read out  aloud the steps called for in the test ballot while the other tester  performed those actions. In order to ensure that all of the actions relating  to the Florida-13 congressional race were performed as laid out in the test  ballots, a two-person review team observed a video display of the test and  compared the actions taken by the tester to those called for in the test  ballot. Furthermore, after the testing was completed, another team  reviewed the video recording of these tests to validate that the actions  relating to the Florida-13 contest taken by the tester were consistent with  those called for by the test ballots.
The criteria used to determine whether the test produced the expected  result was derived from the Florida Voting System Standards.  Specifically, among other things, these standards require the system to  allow the voter to (1) determine whether the inputs given to the system  have selected the candidates that he or she intended to select, (2) review  the candidate selections made by the voter, and (3) change any selection  previously made and confirm the new selection prior to the act of casting  the ballot. Furthermore, the system must communicate to the voter the  fact that the voter has failed to vote in a race (undervote) and require the  voter to confirm his or her intent to undervote before casting the ballot.  During the ballot test, the actual system response was compared to the  expected results by a review team and after the testing was completed  another review team compared the video records to the test ballots to  validate that the tests had been performed in accordance with test scripts  for the Florida-13 contest.
At the beginning of testing on each iVotronic DRE, the machine was  opened for voting and a zero tape was printed. After the casting of all test  ballots on the machine, the machine was closed and a results tape was  printed. The closing of the machine also writes the audit data to the  compact flash card, including event data and ballot images. We examined  the results tapes and compared the total votes cast for the Florida-13  contest against what was expected from the test ballots. We also kept  track of the total number of ballots handled by the machine, called the  “protective count” of an iVotronic DRE, before and after the test and  confirmed that the increase in protective count matched the number of  test ballots cast on that machine.
Using the Unity election reporting manager, we read the compact flash  cards and processed the audit data on each ballot test machine. We  generated the ballot image log and examined the individual test ballots in  the ballot image log. We looked for the unique identifier that was used for  each test ballot and then confirmed that the ballot image reflected the  correct selection for the Florida-13 race as called for by the test ballot. For  example, the test script for Test Ballot 1 required the tester to (1) select a  write-in candidate for U.S. Senate and (2) enter the value of “TB1” in the  write-in field. Because only this test ballot used this value, we could  review the ballot image log to determine what selection the voting  machine recorded for the Florida-13 contest for the ballot showing “TB1”  as the write-in candidate for U.S. Senate.
Finally, using the process discussed previously for firmware testing, the  firmware on all machines used for ballot testing was validated to ensure  these machines used the same firmware that had been certified by the  Florida Division of Elections.
section_title: Results of Ballot Testing
paragraphs: After executing the ballot tests on the election day and early voting  machines, we found that all 10 iVotronic DREs captured the votes for the  Florida-13 race on the test ballots accurately. We used a unique identifier  in a write-in field in each test ballot and verified that the iVotronic DRE  accurately captured the tester’s final selections in the Florida-13 race for  each test ballot.
Testing 112 ways to select a candidate on a single machine also provided  us some additional assurance that the volume of ballots cast on election  day did not contribute to the undervote. We noted that casting 112 ballots  on a single machine was more than the number of ballots cast on over 99  percent of the 1,415 machines used on election day.
section_title: Deliberately Miscalibrated iVotronic DREs Accurately Recorded Displayed Ballots
paragraphs: Because little was known about the effect of a miscalibrated machine on  the behavior of an iVotronic DRE, we deliberately miscalibrated two  iVotronic DREs using 10 different miscalibration methods to verify the  functioning of the machine. Although the miscalibration made the machine  more difficult to use, the 39 ballots used in this test confirmed that the  system correctly recorded the displayed vote for the Florida-13 contest  and did not appear to contribute to the undervote.
subsections: section_title: Methodology for Calibration Testing
paragraphs: For the calibration testing, we judgmentally selected five different  miscalibration patterns and repeated each pattern twice—once with a  small amount of miscalibration and the second time with a large amount of  miscalibration. The amount of miscalibration was also subjective— roughly 0.25 to 0.5 inch for a small amount and about 0.7 to 1 inch for a  large miscalibration.
The miscalibration patterns are shown in the following figures.
We conducted calibration testing on two different machines that were  used for ballot testing. As with ballot testing, at the beginning of testing  of each machine, we opened the machine for voting and printed a zero  tape. During the opening process, we calibrated the machine with one of  the miscalibration patterns. After the machine was miscalibrated, we then  executed at least three of the test ballots that were used during ballot  testing on that machine for each test. The test ballots were rotated among  the miscalibration patterns. For example, one of the machines had eight  different ballot test scripts. The first three were used on one  miscalibration pattern, the next three on another miscalibration pattern,  and the final two plus the first one would be used on another  miscalibration pattern. After the ballots were cast for one miscalibration  pattern, the machine would be miscalibrated with another pattern. After  the needed miscalibration patterns were tested on a machine, the  iVotronic DRE was closed and a results tape was printed. The closing of  the iVotronic DRE also wrote the audit data to the compact flash card.
During the testing, the tester was instructed to take whatever actions were  necessary to achieve the desired result. For example, if the script called  for the selection of Candidate A, then the tester would keep touching the  screen until Candidate A was selected. A review team monitored the  testing to ensure that (1) the proper candidate for the Florida-13  congressional race was ultimately selected and (2) the review screen  showed this candidate selection when it was first presented.
As with the ballot test, we used the Unity election reporting manager to  read the compact flash cards and processed the audit data or each ballot  test machine. We generated the ballot image log and examined the  individual test ballots in the ballot image log. We looked for the unique  identifier that was used for each test ballot and then confirmed that the  ballot image reflected the correct selection for the Florida-13 race as  called for by the test ballot. After the testing had been completed, the  expected results shown in the test ballot scripts were compared to the  actual results contained in the ballot image log and the results tape using  the same process discussed in the ballot testing methodology.
section_title: Results of Calibration Testing
paragraphs: The 39 ballots used in this test confirmed that the system correctly  recorded the displayed vote for the Florida-13 contest. We also noted that  the miscalibration clearly made the machines harder to use and during an  actual election these machines would have probably been either  recalibrated or removed from service once the voter brought the problem  to the precinct’s attention, according to a Sarasota County official who  observed the tests.
Figure 11 shows an example of effects of our miscalibration efforts on the  screen that is used to confirm the calibration results. Specifically, the  stylus points to where the tester is touching the screen while the “X” on  the screen shows where the machine indicated the stylus was touching the  screen. In a properly calibrated machine, the stylus and the “X” are  basically at the same point.
Figure 12 shows an example of where the tester is touching the screen to  make a selection and how this “touch” is translated into a selection. As can  be seen, the finger making the selection is touching a position that in a  properly calibrated machine would not result in the selection shown.  However, the machine clearly shows the candidate selected and our tests  confirmed that for the 39 ballots tested, the candidate actually shown by  the system as selected (in this example, the shaded line) was the candidate  shown on the review screen, as well as the candidate that received the  vote when the ballot was cast.
section_title: Conclusions
paragraphs: Our tests showed that (1) the firmware installed in a statistically selected  sample of machines used by Sarasota County during the 2006 general  election matched the firmware certified by the Florida Division of  Elections, and we confirmed that when the manufacturer rebuilt the  iVotronic 8.0.1.2 firmware from the escrowed source code, the resulting  firmware matched the certified version of firmware held in escrow, (2) the  machines properly displayed, recorded, and counted the selections for all  test ballots cast during the ballot testing involving the 112 common ways a  voter may interact with the system to cast a ballot for the Florida-13 race,  and (3) the machines accurately recorded the test ballots displayed on  deliberately miscalibrated machines.  The results of these tests did not  identify any problems that would indicate that the iVotronic DREs were  responsible for the undervote in the Florida-13 race in the 2006 general  election.
As we noted when we proposed these tests, even after completing these  tests, we do not have absolute assurance that the iVotronic DREs did not  play any role in the large undervote. Absolute assurance is impossible to  achieve because we are unable to recreate the conditions of the election in  which the undervote occurred. Although the test results cannot be used to  provide absolute assurance, we believe that these test results, combined  with the other reviews that have been conducted by Florida, GAO, and  others, have significantly reduced the possibility that the iVotronic DREs  were the cause of the undervote. At this point, we believe that adequate  testing has been performed on the voting machine software to reach this  conclusion and do not recommend further testing in this area. Given the  complex interaction of people, processes, and technology that must work  effectively together to achieve a successful election, we acknowledge the  possibility that the large undervote in Florida’s 13th Congressional District  race could have been caused by factors such as voters who intentionally  undervoted, or voters who did not properly cast their ballots on the  iVotronic DRE, potentially because of issues relating to interaction  between voters and the ballot.
section_title: Comments
paragraphs: We provided draft copies of this statement to the Secretary of State of  Florida and ES&S for their review and comment. We briefed the Sarasota  County Supervisor of Elections on the contents of this statement and  asked for their comments. The Florida Department of State provided  technical comments, which we incorporated. ES&S and the Sarasota  County Supervisor of Elections provided no comments.
Mr. Chairman, this completes my prepared statement. I would be happy to  respond to any questions you or other Members of the Task Force may  have at this time.
section_title: Contact and Acknowledgments
paragraphs: For further information about this statement, please contact Naba  Barkakati at (202) 512-6412 or barkakatin@gao.gov. Contact points for our  Office of Congressional Relations and Public Affairs may be found on the  last page of this statement. Other key contributors to this statement  include James Ashley, Stephen Brown, Francine Delvecchio, Cynthia  Grant, Geoffrey Hamilton, Richard Hung, Douglas Manor, John C. Martin,  Jan Montgomery, Daniel Novillo, Deborah Ortega, Keith Rhodes, Sidney  Schwartz, Patrick Tobo, George Warnock, and Elizabeth Wood. We also  appreciate the assistance of the House Recording Studio in the video  recording of the tests.
section_title: Appendix I: Methodology for Selecting IVotronic DREs for GAO Testing
paragraphs: Each of the three tests—firmware verification, ballot, and calibration— was conducted on a sample of the 1,499 iVotronic DREs that recorded  votes during the 2006 general election in Sarasota County, Florida. We  selected 115 iVotronic DREs for the firmware test, 10 for the ballot test,  and 2 for the calibration test. Appendix II contains the serial numbers of  the iVotronic DREs that were tested.
subsections: section_title: Firmware Test Sample
paragraphs: We selected a stratified random probability sample of iVotronic DREs  from the population of 1,499. The sample was designed to allow us to  generalize the results of the firmware sample to the population of  iVotronic DREs used in this election. We stratified the population into two  strata based on whether the machines had been sequestered since the 2006  general election. There were a total of 818 machines that were sequestered  and 681 machines that had been used in subsequent elections. The  population and sample are described in table 3.
We calculated the sample size in each stratum using the hypergeometric  distribution to account for the relatively small populations in each  stratum. We determined each sample size to be the minimum number of  machines necessary to yield an upper bound of 7.5 percent, at the 99  percent confidence level, if we observed zero failures in the firmware test.  Assuming that we found no machines using an uncertified firmware  version, these sample sizes allowed us to conclude with 99 percent  confidence that no more than 7.5 percent of the machines in each stratum  were using uncertified firmware. Further, this sample allowed us to  conclude that no more than 4 percent of the 1,499 iVotronic DREs were  using uncertified firmware, at the 99 percent confidence level.
An additional five sequestered machines and five non-sequestered  machines were selected as back-up machines should there be problems in  locating the selected machines or some other problem that prevented  testing them.
section_title: Ballot Test Sample
paragraphs: We randomly selected a total of 10 machines from the population of 1,384  machines that were not selected in the firmware test sample. This sample  size is not sufficient to allow us to make direct generalizations to the  population. However, if we are reasonably confident that the same  software is used in all 1,499 machines, then we are more confident that the  results of the other tests on a small number of machines can be used to  obtain increased assurance that the iVotronic DREs did not cause the  undervote. We randomly selected one machine from each of the nine  ballot styles used during the general election and one machine from the  machines used for early voting. In case of problems in operating or  locating the machines, we also selected randomly selected two additional  machines for each ballot style and for early voting.
section_title: Calibration Test Sample
paragraphs: The two iVotronic DREs selected for calibration testing were selected  from those tested in the ballot test. Because the machines used for the  ballot tests included an ADA machine and “standard” machines, we  selected one of each for calibration testing. Although we did not test the  ADA capabilities of the ADA machine (e.g., the audio ballots), we found  that the on-screen appearance of selections on the ADA machine differed  slightly from that on non-ADA machines. For example, the standard non- ADA machine displayed a blue bar across the screen and an X in the box  next to the candidate’s name when a selection was made, while an ADA  machine only showed an X in the box next to the candidate’s name.
section_title: Appendix II: List of Machines Tested by GAO
paragraphs: Table 4 table lists the iVotronic DREs that were tested by GAO. For each  machine, the table shows whether the machine was sequestered and what  type of testing was conducted on the machine.