Elections: Results of GAO's Testing of Voting Systems Used in Sarasota County in Florida's 13th Congressional District
Released: 2008-02-08T00:00:00

Letter

Background

The 13th Congressional District of Florida comprises DeSoto, Hardee, Sarasota, and parts of Charlotte and Manatee Counties. In the November 2006 general election, there were two candidates in the race to represent the 13th Congressional District: Vern Buchanan, the Republican candidate, and Christine Jennings, the Democratic candidate. The State of Florida certified Vern Buchanan the winner of the election. The margin of victory was 369 votes out of a total of 238,249 votes counted. Table 1 summarizes the results of the election and shows that the results from Sarasota County exhibited a significantly higher undervote rate than in the other counties in the congressional district.

As seen in table 1, about 18,000 undervotes were reported in Sarasota County in the race for Florida’s 13th Congressional District. After the election results were contested in the House of Representatives, the task force met and unanimously voted to seek GAO’s assistance in determining whether the voting systems contributed to the large undervote in Sarasota County. On June 14, 2007, we met with the task force and agreed upon an engagement plan. We reported on the status of our review at an interim meeting held by the task force on August 3, 2007.

On October 2, 2007, we reported that our analysis of election data did not identify any particular voting machines or machine characteristics that could have caused the large undervote in the Florida-13 race. The undervotes in Sarasota County were generally distributed across all machines and precincts. We found that some of the prior tests and reviews conducted by the State of Florida and Sarasota County provided assurance that certain components of the voting system in Sarasota County functioned correctly, but they were not enough to provide reasonable assurance that the iVotronic DREs did not contribute to the undervote. We proposed three tests—firmware verification, ballot, and calibration—to provide increased assurance, but not absolute assurance, that the iVotronic DREs did not cause the large undervote in Sarasota County. We stated that the successful conduct of the tests could reduce the possibility that the voting systems caused the undervote and shift attention to the possibilities that voters intentionally undervoted or voters did not properly cast their ballots on the iVotronic DRE, potentially because of issues relating to interaction between voters and the ballot.

Overview of the Voting Systems Used in Sarasota County in the 2006 General Elections

In the 2006 general election, Sarasota County used voting systems manufactured by ES&S. The State of Florida has certified different versions of ES&S voting systems. The version used in Sarasota County was designated ES&S Voting System Release 4.5, Version 2, Revision 2, and consisted of iVotronic DREs, a Model 650 central count optical scan tabulator for absentee ballots, and the Unity election management system. It was certified by the State of Florida on July 17, 2006. The certified system includes different configurations and optional elements, several of which were not used in Sarasota County.

The election management part of the voting system is called Unity; the version that was used was 2.4.4.2. Figure 1 shows the overall election operation using the Unity election management system and the iVotronic DRE.

Sarasota County used iVotronic DREs for early and election day voting. Specifically, Sarasota County used the 12-inch iVotronic DRE, hardware version 1.1 with firmware version 8.0.1.2. Some of the iVotronic DREs are configured to use audio ballots, which are often referred to as Americans with Disabilities Act (ADA) machines. The iVotronic DRE uses a touch screen—a pressure-sensitive graphics display panel—to display and record votes (see fig. 2).

The machine has a storage case that also serves as the voting booth. The operation of the iVotronic DRE requires the use of a personalized electronic ballot (PEB), which is a storage device with an infrared window used for transmission of ballot data to and from the iVotronic DRE. The iVotronic DRE has four independent flash memory modules, one of which contains the program code—firmware—that runs the machine; the remaining three flash memory modules store redundant copies of ballot definitions, machine configuration information, ballots cast by voters, and event logs (see fig. 3). The iVotronic DRE includes a VOTE button that the voter has to press to cast a ballot and record the information in the flash memory. The iVotronic DRE also includes a compact flash card that can be used to load sound files onto iVotronic DREs with ADA functionality. The iVotronic DRE’s firmware can be updated through the compact flash card. Additionally, at the end of polling, the ballots and audit information are to be copied from the internal flash memory module to the compact flash card.

To use the iVotronic DRE for voting, a poll worker activates the iVotronic DRE by inserting a PEB into the PEB slot after the voter has signed in at the polling place. After the poll worker makes selections so that the appropriate ballot will appear, the PEB is removed and the voter is ready to begin using the system. The ballot is presented to the voter in a series of display screens, with candidate information on the left side of the screen and selection boxes on the right side (see fig. 4).

The voter can make a selection by touching anywhere on the line, and the iVotronic DRE responds by highlighting the entire line and displaying an X in the box next to the candidate’s name. The voter can also change his or her selection by touching the line corresponding to another candidate or by deselecting his or her choice. “Previous Page” and “Next Page” buttons are used to navigate the multipage ballot. After completing all selections, the voter is presented with a summary screen with all of his or her selections (see fig. 5). From the summary screen, the voter can change any selection by selecting the race. The race will be displayed to the voter on its own ballot page. When the voter is satisfied with the selections and has reached the final summary screen, the red VOTE button is illuminated, indicating the voter can now cast his or her ballot. When the VOTE button is pressed, the voting session is complete and the ballot is recorded on the iVotronic DRE. In Sarasota County’s 2006 general election, there were nine different ballot styles with between 28 and 40 races, which required between 15 and 21 electronic ballot pages to display, and 3 to 4 summary pages for review purposes.

Election Systems Involve People, Processes, and Technology

An election system is based upon a complex interaction of people (voters, election officials, and poll workers), processes (controls), and technology that must work effectively together to achieve a successful election. The particular technology used to cast and count votes is a critical part of how elections are conducted, but it is only one facet of a multifaceted election process that involves the interplay of people, processes, and technology.

As we have previously reported, every stage of the election process— registration, absentee and early voting, preparing for and conducting Election Day activities, provisional voting, and vote counting—is affected by the interaction of people, processes, and technology. Breakdowns in the interaction of people, processes, and technology may, at any stage of an election, impair an accurate vote count. For example, if the voter registration process is flawed, ineligible voters may be allowed to cast votes. Poll worker training deficiencies may contribute to discrepancies in the number of votes credited and cast, if voter information was not entered properly into poll books. Mistakes in using the DRE systems could result from inadequate understanding of the equipment on the part of those using it.

As noted in our October statement, we recognize that human interaction with the ballot layout could be a potential cause of the undervote, and we noted that several suggestions have been offered as possible ways to establish that voters are intentionally undervoting and to provide some assurance that the voting systems did not cause the undervote. For instance, A voter-verified paper trail could provide an independent confirmation that the touch screen voting systems did not malfunction in recording and counting the votes from the election. The paper trail would reflect the voter’s selections and, if necessary, could be used in the counting or recounting of votes. This issue was also recognized in the source code review performed by the Security and Assurance in Information Technology (SAIT) laboratory at Florida State University as well as the 2005 and draft 2007 Voluntary Voting Systems Guidelines prepared for the Election Assistance Commission. We have previously reported on the need to implement such a function properly.

Explicit feedback to voters that a race has been undervoted and a prompt for voters to affirm their intent to undervote might help prevent many voters from unintentionally not casting a vote in a race. On the iVotronic DREs, such feedback and prompts are provided only when the voter attempts to cast a completely blank ballot, but not when a voter fails to vote in individual races.

Offering a “none of the above” option in a race would provide voters with the opportunity to indicate that they are intentionally undervoting. For example, the State of Nevada provides this option in certain races in its elections.

We reported that decisions about these or other suggestions about ballot layout or voting system functions should be informed by human factors studies that assess such measures’ effectiveness in accurately recording voters’ preferences, making voting systems easier to use, and preventing unintentional undervotes.

Tests Confirm Sarasota County iVotronic DREs Used Same Firmware Certified by Florida

We previously reported that having reasonable assurance that all iVotronic DREs that recorded votes in the 2006 general election were running the same certified firmware would allow us to have more confidence that the iVotronic DREs will behave similarly when tested. Consequently, if we are reasonably confident that the same firmware was running in all 1,499 machines, then we are more confident that the results of other tests, conducted both by GAO and by others, on a small number of machines can be used to obtain increased assurance that the iVotronic DREs did not cause the undervote. We also reported that there was a lack of assurance that the source code that was held in escrow by the Florida Division of Elections and that was previously reviewed by Florida State University and by us, if rebuilt, would corresponded to the firmware that was certified and held in escrow by the Florida Division of Elections. We found that the firmware on a statistically selected sample of 115 iVotronic DREs was the same as that certified by the Florida Division of Elections. We also found that the escrowed source code, when rebuilt into executable firmware, corresponded to the 8.0.1.2 firmware that was certified by the Florida Division of Elections.

Methodology for Firmware Verification Testing

Our methodology to obtain reasonable assurance that the firmware used on Sarasota County’s iVotronic DREs during the 2006 general election was the same as that certified by the State of Florida was broken down into two basic steps: (1) selecting a representative sample of machines, and (2) verifying that the firmware extracted from the voting machines was the same as the escrowed firmware that had been certified by the Florida Division of Elections. Appendix I details the methodology for selecting the representative sample of machines. Appendix II contains a list of the serial numbers of the tested iVotronic DREs.

To ensure that we would be testing with the iVotronic firmware certified by the Florida Division of Elections, on October 18, 2007, we and officials from the Florida Division of Elections made two copies of the escrowed iVotronic 8.0.1.2 firmware on compact discs (CD) and placed them in two tamper-evident bags with serial numbers. The bags were subsequently hand-delivered by a Florida Division of Elections official for our use in the firmware verification test and for the rebuilding of the firmware from the source code.

In order to extract the firmware from an iVotronic DRE, the machine was placed on an anti-static mat and the case was opened using a special screwdriver. After lifting the case, a special extraction tool was used to remove the flash memory module that contains the firmware. The flash memory module was then inserted in the socket of a Needham Electronics’ EMP-300 device that was connected to the universal serial bus (USB) port of a personal computer (PC). The EMPWin application running on that PC was used to read the firmware from the flash memory module and save the extracted firmware on the PC. The Florida Division of Elections loaned us the EMP-300 and EMPWin application for use in extracting firmware from the flash memory module.

To compare the extracted firmware with the escrowed version, we relied on two commercially available software programs. First, we acquired a license for PrestoSoft’s ExamDiff Pro software that enables comparison of files. The ExamDiff Pro software is a commercially available program designed to highlight the differences between two files. For each selected iVotronic DRE, the extracted firmware was compared with the escrowed version with any differences highlighted by the program.

Second, to further ensure that the extracted firmware matched the escrowed firmware, we compared the SHA-1 hash value of the extracted firmware to the hash value of the comparable certified firmware. We computed the SHA-1 hash by using the Maresware hash software that was provided by the Florida Division of Elections. In order to ensure that the commercial Maresware hash software properly calculated the SHA-1 hash value, we (1) created four files and obtained a fifth file that contained executable code, (2) obtained hash values for each file by either using an external program that generated the hash values using the same hashing algorithm as the commercial product or using known hash values, and (3) used the commercial program acquired for testing the firmware to ensure that the hash values it generated for these five files were identical to the expected hash values for those files. In each case, the hash values generated by the commercial program were identical to the expected values. Accordingly, reasonable assurance for the purposes of our review was obtained that the commercial program produced its hash values in accordance with the NIST algorithm.

At the end of each day, we (1) used the commercial Maresware software to compute hash values for each of the firmware programs that had been unloaded during that day and all previous days, and (2) compared each hash created by this program to the expected value that was calculated from the firmware that had been escrowed by the Florida Division of Elections. This comparison provided further assurance that the extracted firmware was (1) identical to the version escrowed by the Florida Division of Elections when the hashes agreed, or (2) different if the hashes did not agree.

We also verified that sequestered machines were not used since the 2006 general election. For each of these sequestered machines, we used an audit PEB to copy the audit logs onto a compact flash card and then used the Unity election reporting manager to generate event log reports. We examined the event logs for the date and time of occurrence of activities that would indicate whether the machine had been used. Lack of such activities since the 2006 general election provided reasonable assurance that the machines had not been used since they were sequestered.

In addition, to verify that the source code for iVotronic DRE firmware version 8.0.1.2 previously examined by the Florida State University SAIT source code review team and by GAO corresponded with the version certified by the Florida Division of Elections, ES&S officials stated that it still had the development environment that could be used to compile, or rebuild, the certified firmware from the source code retained in escrow by the Florida Division of Elections. As we previously noted, a software review and security analysis of the iVotronic DRE firmware was conducted by a team led by Florida State University’s SAIT laboratory. The software review team attempted to confirm or refute many different hypotheses that, if true, might explain the undervote in the race for the 13th Congressional District. In doing so, they made several observations about the source code, which we were able to independently verify.

The rebuilding of the firmware was conducted by ES&S at its Rockford, Illinois, facility on November 19, 2007, and witnessed by us. Prior to the rebuild, the Florida Division of Elections provided an unofficial copy of the source code to ES&S so that ES&S could prepare the development environment and test the rebuild steps. Using the official sealed copy of the source code CD, ES&S rebuilt the firmware in front of GAO representatives. ES&S described the development environment and we inspected it to satisfy ourselves that the firmware was faithfully rebuilt using the escrowed source code. After the rebuilding of the firmware, the certified version of 8.0.1.2 firmware was compared with the rebuilt version using PrestoSoft’s ExamDiff Pro.

Results of Firmware Verification Testing

While the Florida audit team had previously confirmed that the firmware running on six iVotronic DREs matched the certified version held in escrow by the Florida Division of Elections, we found that the sample size was too small to support generalization to all 1,499 iVotronic DREs that recorded votes during the 2006 general election. Accordingly, we conducted a firmware verification test on a statistically valid sample of 115 iVotronic DRE machines used by Sarasota County during the 2006 general election. The selected machines fell into two groups—machines that had not been used since the 2006 general election (referred to as sequestered machines) and machines that had been used in subsequent elections. For each machine, we extracted the firmware from a flash memory module in that machine and then compared the extracted firmware with the escrowed version using commercially available file comparison tools to determine whether they agreed. We found that the firmware installed in the flash memory module of each machine matched the escrowed firmware that had been certified by Florida. The statistical approach used to select these machines lets us estimate with a 99 percent confidence level that at least 1,439, or 96 percent, of the 1,499 machines used in the 2006 general election used the firmware that was certified by the State of Florida.

We witnessed the rebuild of the iVotronic DRE’s firmware from the source code that was held in escrow by the Florida Division of Elections and that was previously reviewed by Florida State University and by us. At ES&S’s software development facility, we observed that rebuilding the firmware from the escrowed source code resulted in the same firmware that was certified and held in escrow by the Florida Division of Elections. The comparison of the escrowed firmware to the version that was rebuilt by the vendor identified no differences and provides us reasonable assurance that the escrowed firmware corresponded to the escrowed source code. The successful rebuilding of the firmware from the escrowed source code enables us to have greater confidence in the conclusions derived from prior source code reviews by Florida State University and us.

Ballot Testing Showed That Machines Accurately Recorded and Counted Ballots

In our October 2007 statement, we noted that there were 112 common ways a voter may interact with the system to select a candidate in the Florida-13 race and cast the ballot, and that prior testing of the iVotronic DREs covered only 13 of these 112 possible ways. We developed 224 test ballots to verify that the iVotronic DRE could accurately capture ballots using each of these 112 common ways a voter may interact with the system; 112 test ballots were cast on one machine configured for early voting, and another 112 ballots were cast on nine machines configured for election day voting. Our tests showed that for each of the 224 test ballots, the iVotronic DRE correctly captured each vote as cast for the Florida-13 race. We also conducted firmware verification tests on these machines and verified that they were running the certified firmware.

Methodology for Ballot Testing

The methodology for ballot testing can be broken into two major areas— development of the test ballots and execution of the test using those ballots. The following sections discuss these areas.

Development of Test Ballots

In examining how the system allowed voters to make a selection in the Florida-13 race, we found at least 112 different ways a voter could make his or her selection and cast the ballot in the Florida-13 race, assuming that it was the only race on the ballot. Specifically, a voter could (1) initially select either candidate or neither candidate (i.e., undervote), (2) change the vote on the initial screen, and (3) use a combination of features to change or verify his or her selection by using the page back and review screen options. Accordingly, we tested these 112 ways to select a candidate on the early voting machine and on the election day machines (224 test ballots in total).

The 112 standard test ballots cover all combinations of the following types of voter behavior: Voter makes selection on the initial ballot screen and makes no changes or takes any other action to return to the contest to review or change selection.

Voter makes selection on the initial ballot screen and decides before leaving that screen to change the selection because of an error in selecting the candidate or for some other reason.

Voter makes selection on the initial ballot screen and then decides to use the page back option to review or change selection.

Voter makes selection on the initial ballot screen and continues to the review screen and then decides to use the review screen option to review or change selection.

Voter makes selection on the initial ballot screen and uses a combination of page back and review screen options to review or change selection.

In each instance where a selection could be made, three choices were possible for the Florida-13 race: a selection for one of the two candidates, or no selection (i.e., an undervote).

In developing the standard test ballots, we did not consider all combinations of some other types of voter behavior that would have significantly increased the number of test cases without providing significant benefits. In most cases, such behavior are variants of the primary voter behavior that we examined. The following are examples of voter behavior that were not included in the standard test set in order to reduce the number of test cases to practicable levels: Using a one-touch or two-touch method to make changes on a ballot page.

Varying the number of pages a voter may go back (“page backs”) to return to the page containing the Florida-13 race to change or review a selection.

Casting a ballot from the review screen selection. The VOTE button is not activated until the voter reaches the last review screen. However, once the VOTE button has been activated, a ballot may be cast from any screen. For example, a voter may activate the VOTE button and then return to a contest to review or change the selection using the review screen option. Once the voter goes to the contest from the review screen and makes any desired changes, the voter can then cast the ballot from that screen rather than going back to the last page of the review screen or even the review screen that was used to return to the selection.

Although we did not consider all combinations of these types of voter behavior when developing the standard test ballots, we included some of these user interactions in the execution of applicable test ballots to provide increased assurance that the system would handle these voter behaviors. For each applicable test ballot, we randomly determined the test procedure that should be used for the following attributes: Initial change method – The standard test ballots address voters making changes on the initial ballot screen. Where possible, the method used to change (one-touch or two-touch) the selection was randomly selected.

Number of page backs – The ballots used by Sarasota County included the page back function. After reviewing the ballots, it appeared reasonable to expect that voters who may have used the page back option would probably decide that they had missed the race by the time they went one or two pages beyond the page with the Florida-13 race. Therefore, when a standard test ballot contained a page back requirement, the number of page backs was randomly selected to determine whether one or two page backs should be used.

Page back change method – Some test ballots required a change after the page back option was selected. As with the initial change method, where possible, the method of changing (one-touch or two-touch) the selection was randomly assigned.

Review screen change method – The system displays a review screen that shows the voter’s selections (or lack of selections) after the voter has progressed through all contests. On the review screen, the voter can select a race to go directly to that contest and (1) review the selection made, and (2) make any desired corrections. The standard test ballots were designed to cover this type of event. Where possible, the method used to make the change (one-touch or two-touch) was randomly selected.

Activate VOTE button and cast ballots from the review screen – In order to test casting ballots from locations other than the last review screen, the VOTE button must be activated prior to going to a screen where the ballot is cast. In order to determine which test ballots should be used for this test, a two-step approach was adopted. First, a random selection of the ballots that use the review screen option was made to determine which test ballots should have the VOTE button activated. Then a random selection of these test ballots was made to determine whether the ballot should be cast from the review screen selection.

Besides those attributes that directly affect the selection in the Florida-13 race, we varied the other attributes on the ballot in order to complete the ballot test. For each of the 224 test ballots, we used random values for other attributes, including the following: Ballot style – Each ballot was randomly assigned one of the nine ballot styles used in the election.

Write-in candidate – All ballot styles includes write-in options in at least 2 races —United States Senate and State Governor/Lieutenant Governor. To verify that the iVotronic DRE accurately recorded the selection in the Florida-13 race for each test ballot, we needed a way to identify each test ballot in the ballot image log. To accomplish this, we randomly selected one of these two races, selected the write-in candidate for the race, and entered a unique value (i.e., the test ballot number) in the write-in field.

Candidates and selections in other races on the ballot – Each ballot style had between 28 and 40 contests on the ballot. The values for the contests besides the Florida-13 race and the write-in field were also randomly selected. For example, most items had three possible choices—candidate 1 (or Yes), candidate 2 (or No), and undervote. Which of these three values was used for a given contest was randomly determined.

The values used for these attributes were independently determined for the election day and early voting test ballots. For example, Test Ballot 2 (election day) and Test Ballot 202 (early voting) were designed to test the same standard condition described by one of the 112 standard test ballots. Table 2 illustrates some of the similarities and differences between the two test ballots that result from the random selection process used to determine the other aspects of the ballot.

Finally, we selected 10 random machines to be used for the ballot testing. One machine was selected from those that were used in early voting in the 2006 general election. The other nine were selected from those that used each of the ballot styles on election day in the 2006 general election. For each election day machine, the assigned precinct was the same as the precinct where the machine was used during the 2006 general election. For the early voting machine, we needed to assign precincts for each ballot style. We used the precinct associated with the back-up machine used for election day testing as the precinct for that ballot style. If the first back-up machine was assigned the same precinct number as the primary election day machine, then we used the precinct associated with the second back-up machine. This approach was taken to maximize the number of precincts used in the testing efforts.

Process Used in Executing the Ballot Test

A two-person test team conducted the ballot testing. One tester read out aloud the steps called for in the test ballot while the other tester performed those actions. In order to ensure that all of the actions relating to the Florida-13 congressional race were performed as laid out in the test ballots, a two-person review team observed a video display of the test and compared the actions taken by the tester to those called for in the test ballot. Furthermore, after the testing was completed, another team reviewed the video recording of these tests to validate that the actions relating to the Florida-13 contest taken by the tester were consistent with those called for by the test ballots.

The criteria used to determine whether the test produced the expected result was derived from the Florida Voting System Standards. Specifically, among other things, these standards require the system to allow the voter to (1) determine whether the inputs given to the system have selected the candidates that he or she intended to select, (2) review the candidate selections made by the voter, and (3) change any selection previously made and confirm the new selection prior to the act of casting the ballot. Furthermore, the system must communicate to the voter the fact that the voter has failed to vote in a race (undervote) and require the voter to confirm his or her intent to undervote before casting the ballot. During the ballot test, the actual system response was compared to the expected results by a review team and after the testing was completed another review team compared the video records to the test ballots to validate that the tests had been performed in accordance with test scripts for the Florida-13 contest.

At the beginning of testing on each iVotronic DRE, the machine was opened for voting and a zero tape was printed. After the casting of all test ballots on the machine, the machine was closed and a results tape was printed. The closing of the machine also writes the audit data to the compact flash card, including event data and ballot images. We examined the results tapes and compared the total votes cast for the Florida-13 contest against what was expected from the test ballots. We also kept track of the total number of ballots handled by the machine, called the “protective count” of an iVotronic DRE, before and after the test and confirmed that the increase in protective count matched the number of test ballots cast on that machine.

Using the Unity election reporting manager, we read the compact flash cards and processed the audit data on each ballot test machine. We generated the ballot image log and examined the individual test ballots in the ballot image log. We looked for the unique identifier that was used for each test ballot and then confirmed that the ballot image reflected the correct selection for the Florida-13 race as called for by the test ballot. For example, the test script for Test Ballot 1 required the tester to (1) select a write-in candidate for U.S. Senate and (2) enter the value of “TB1” in the write-in field. Because only this test ballot used this value, we could review the ballot image log to determine what selection the voting machine recorded for the Florida-13 contest for the ballot showing “TB1” as the write-in candidate for U.S. Senate.

Finally, using the process discussed previously for firmware testing, the firmware on all machines used for ballot testing was validated to ensure these machines used the same firmware that had been certified by the Florida Division of Elections.

Results of Ballot Testing

After executing the ballot tests on the election day and early voting machines, we found that all 10 iVotronic DREs captured the votes for the Florida-13 race on the test ballots accurately. We used a unique identifier in a write-in field in each test ballot and verified that the iVotronic DRE accurately captured the tester’s final selections in the Florida-13 race for each test ballot.

Testing 112 ways to select a candidate on a single machine also provided us some additional assurance that the volume of ballots cast on election day did not contribute to the undervote. We noted that casting 112 ballots on a single machine was more than the number of ballots cast on over 99 percent of the 1,415 machines used on election day.

Deliberately Miscalibrated iVotronic DREs Accurately Recorded Displayed Ballots

Because little was known about the effect of a miscalibrated machine on the behavior of an iVotronic DRE, we deliberately miscalibrated two iVotronic DREs using 10 different miscalibration methods to verify the functioning of the machine. Although the miscalibration made the machine more difficult to use, the 39 ballots used in this test confirmed that the system correctly recorded the displayed vote for the Florida-13 contest and did not appear to contribute to the undervote.

Methodology for Calibration Testing

For the calibration testing, we judgmentally selected five different miscalibration patterns and repeated each pattern twice—once with a small amount of miscalibration and the second time with a large amount of miscalibration. The amount of miscalibration was also subjective— roughly 0.25 to 0.5 inch for a small amount and about 0.7 to 1 inch for a large miscalibration.

The miscalibration patterns are shown in the following figures.

We conducted calibration testing on two different machines that were used for ballot testing. As with ballot testing, at the beginning of testing of each machine, we opened the machine for voting and printed a zero tape. During the opening process, we calibrated the machine with one of the miscalibration patterns. After the machine was miscalibrated, we then executed at least three of the test ballots that were used during ballot testing on that machine for each test. The test ballots were rotated among the miscalibration patterns. For example, one of the machines had eight different ballot test scripts. The first three were used on one miscalibration pattern, the next three on another miscalibration pattern, and the final two plus the first one would be used on another miscalibration pattern. After the ballots were cast for one miscalibration pattern, the machine would be miscalibrated with another pattern. After the needed miscalibration patterns were tested on a machine, the iVotronic DRE was closed and a results tape was printed. The closing of the iVotronic DRE also wrote the audit data to the compact flash card.

During the testing, the tester was instructed to take whatever actions were necessary to achieve the desired result. For example, if the script called for the selection of Candidate A, then the tester would keep touching the screen until Candidate A was selected. A review team monitored the testing to ensure that (1) the proper candidate for the Florida-13 congressional race was ultimately selected and (2) the review screen showed this candidate selection when it was first presented.

As with the ballot test, we used the Unity election reporting manager to read the compact flash cards and processed the audit data or each ballot test machine. We generated the ballot image log and examined the individual test ballots in the ballot image log. We looked for the unique identifier that was used for each test ballot and then confirmed that the ballot image reflected the correct selection for the Florida-13 race as called for by the test ballot. After the testing had been completed, the expected results shown in the test ballot scripts were compared to the actual results contained in the ballot image log and the results tape using the same process discussed in the ballot testing methodology.

Results of Calibration Testing

The 39 ballots used in this test confirmed that the system correctly recorded the displayed vote for the Florida-13 contest. We also noted that the miscalibration clearly made the machines harder to use and during an actual election these machines would have probably been either recalibrated or removed from service once the voter brought the problem to the precinct’s attention, according to a Sarasota County official who observed the tests.

Figure 11 shows an example of effects of our miscalibration efforts on the screen that is used to confirm the calibration results. Specifically, the stylus points to where the tester is touching the screen while the “X” on the screen shows where the machine indicated the stylus was touching the screen. In a properly calibrated machine, the stylus and the “X” are basically at the same point.

Figure 12 shows an example of where the tester is touching the screen to make a selection and how this “touch” is translated into a selection. As can be seen, the finger making the selection is touching a position that in a properly calibrated machine would not result in the selection shown. However, the machine clearly shows the candidate selected and our tests confirmed that for the 39 ballots tested, the candidate actually shown by the system as selected (in this example, the shaded line) was the candidate shown on the review screen, as well as the candidate that received the vote when the ballot was cast.

Conclusions

Our tests showed that (1) the firmware installed in a statistically selected sample of machines used by Sarasota County during the 2006 general election matched the firmware certified by the Florida Division of Elections, and we confirmed that when the manufacturer rebuilt the iVotronic 8.0.1.2 firmware from the escrowed source code, the resulting firmware matched the certified version of firmware held in escrow, (2) the machines properly displayed, recorded, and counted the selections for all test ballots cast during the ballot testing involving the 112 common ways a voter may interact with the system to cast a ballot for the Florida-13 race, and (3) the machines accurately recorded the test ballots displayed on deliberately miscalibrated machines. The results of these tests did not identify any problems that would indicate that the iVotronic DREs were responsible for the undervote in the Florida-13 race in the 2006 general election.

As we noted when we proposed these tests, even after completing these tests, we do not have absolute assurance that the iVotronic DREs did not play any role in the large undervote. Absolute assurance is impossible to achieve because we are unable to recreate the conditions of the election in which the undervote occurred. Although the test results cannot be used to provide absolute assurance, we believe that these test results, combined with the other reviews that have been conducted by Florida, GAO, and others, have significantly reduced the possibility that the iVotronic DREs were the cause of the undervote. At this point, we believe that adequate testing has been performed on the voting machine software to reach this conclusion and do not recommend further testing in this area. Given the complex interaction of people, processes, and technology that must work effectively together to achieve a successful election, we acknowledge the possibility that the large undervote in Florida’s 13th Congressional District race could have been caused by factors such as voters who intentionally undervoted, or voters who did not properly cast their ballots on the iVotronic DRE, potentially because of issues relating to interaction between voters and the ballot.

Comments

We provided draft copies of this statement to the Secretary of State of Florida and ES&S for their review and comment. We briefed the Sarasota County Supervisor of Elections on the contents of this statement and asked for their comments. The Florida Department of State provided technical comments, which we incorporated. ES&S and the Sarasota County Supervisor of Elections provided no comments.

Mr. Chairman, this completes my prepared statement. I would be happy to respond to any questions you or other Members of the Task Force may have at this time.

Contact and Acknowledgments

For further information about this statement, please contact Naba Barkakati at (202) 512-6412 or barkakatin@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this statement. Other key contributors to this statement include James Ashley, Stephen Brown, Francine Delvecchio, Cynthia Grant, Geoffrey Hamilton, Richard Hung, Douglas Manor, John C. Martin, Jan Montgomery, Daniel Novillo, Deborah Ortega, Keith Rhodes, Sidney Schwartz, Patrick Tobo, George Warnock, and Elizabeth Wood. We also appreciate the assistance of the House Recording Studio in the video recording of the tests.

Appendix I: Methodology for Selecting IVotronic DREs for GAO Testing

Each of the three tests—firmware verification, ballot, and calibration— was conducted on a sample of the 1,499 iVotronic DREs that recorded votes during the 2006 general election in Sarasota County, Florida. We selected 115 iVotronic DREs for the firmware test, 10 for the ballot test, and 2 for the calibration test. Appendix II contains the serial numbers of the iVotronic DREs that were tested.

Firmware Test Sample

We selected a stratified random probability sample of iVotronic DREs from the population of 1,499. The sample was designed to allow us to generalize the results of the firmware sample to the population of iVotronic DREs used in this election. We stratified the population into two strata based on whether the machines had been sequestered since the 2006 general election. There were a total of 818 machines that were sequestered and 681 machines that had been used in subsequent elections. The population and sample are described in table 3.

We calculated the sample size in each stratum using the hypergeometric distribution to account for the relatively small populations in each stratum. We determined each sample size to be the minimum number of machines necessary to yield an upper bound of 7.5 percent, at the 99 percent confidence level, if we observed zero failures in the firmware test. Assuming that we found no machines using an uncertified firmware version, these sample sizes allowed us to conclude with 99 percent confidence that no more than 7.5 percent of the machines in each stratum were using uncertified firmware. Further, this sample allowed us to conclude that no more than 4 percent of the 1,499 iVotronic DREs were using uncertified firmware, at the 99 percent confidence level.

An additional five sequestered machines and five non-sequestered machines were selected as back-up machines should there be problems in locating the selected machines or some other problem that prevented testing them.

Ballot Test Sample

We randomly selected a total of 10 machines from the population of 1,384 machines that were not selected in the firmware test sample. This sample size is not sufficient to allow us to make direct generalizations to the population. However, if we are reasonably confident that the same software is used in all 1,499 machines, then we are more confident that the results of the other tests on a small number of machines can be used to obtain increased assurance that the iVotronic DREs did not cause the undervote. We randomly selected one machine from each of the nine ballot styles used during the general election and one machine from the machines used for early voting. In case of problems in operating or locating the machines, we also selected randomly selected two additional machines for each ballot style and for early voting.

Calibration Test Sample

The two iVotronic DREs selected for calibration testing were selected from those tested in the ballot test. Because the machines used for the ballot tests included an ADA machine and “standard” machines, we selected one of each for calibration testing. Although we did not test the ADA capabilities of the ADA machine (e.g., the audio ballots), we found that the on-screen appearance of selections on the ADA machine differed slightly from that on non-ADA machines. For example, the standard non- ADA machine displayed a blue bar across the screen and an X in the box next to the candidate’s name when a selection was made, while an ADA machine only showed an X in the box next to the candidate’s name.

Appendix II: List of Machines Tested by GAO

Table 4 table lists the iVotronic DREs that were tested by GAO. For each machine, the table shows whether the machine was sequestered and what type of testing was conducted on the machine.