Elections: Further Testing Could Provide Increased but Not Absolute Assurance That Voting Systems Did Not Cause Undervotes in Florida's 13th Congressional District
Published: 2007-10-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In November 2006, about 18,000 undervotes were reported in Sarasota County in the race for Florida's 13th Congressional District (FL-13). After the contesting of the election results in the House of Representatives, the task force unanimously voted to seek GAO's assistance in determining whether the voting systems contributed to the large undervote in Sarasota County. GAO agreed with the task force on an engagement plan, including the following review objectives: (1) What voting systems were used in Sarasota County and what processes governed their use? (2) What was the scope of the undervote in Sarasota County in the general election? (3) What tests were conducted on the voting systems in Sarasota County prior to the general election and what were the results of those tests? (4) Considering the voting systems tests conducted after the general election, are additional tests needed to determine whether the voting systems contributed to the undervote? To conduct its work, GAO met with officials from the State of Florida, Sarasota County, and Election Systems and Software (ES&S)--the voting systems manufacturer--and reviewed voting systems test documentation. GAO analyzed election data to characterize the undervote. On the basis of its assessments of prior testing and other activities, GAO identified potential additional tests for the Sarasota County voting systems.
section_title: What GAO Found
paragraphs: In the 2006 general election, Sarasota County used voting systems manufactured by ES&S, specifically iVotronic direct recording electronic (DRE) voting systems during early and election day voting and the Unity election management system, which handles the election administration functions, such as ballot design and election reporting. GAO's analysis of the 2006 general election data from Sarasota County did not identify any particular voting machines or machine characteristics that could have caused the large undervote in the FL-13 race. The undervotes in Sarasota County were generally distributed across all machines and precincts. GAO's analysis found that some of the prior tests and reviews conducted by the State of Florida and Sarasota County provide assurance that certain components of the voting systems in Sarasota County functioned correctly, but they are not enough to provide reasonable assurance that the iVotronic DREs did not contribute to the undervote. Specifically, GAO found that assurance is lacking in three areas, and proposes that tests be conducted to address those areas. First, because there is insufficient assurance that the firmware in all the iVotronic DREs used in the election matched the certified version held by the Florida Division of Elections, GAO proposes that a firmware verification test be conducted on a representative sample of 115 (of the 1,499) machines that were used in the general election. Second, because an insufficient number of ways to select a candidate in the FL-13 race were tested, GAO proposes that a test be conducted to verify all 112 ways that GAO identified to select a candidate. Third, because no prior tests were identified that address the effect of a miscalibrated iVotronic DRE on the undervote, GAO proposes that an iVotronic DRE be deliberately miscalibrated to verify the accurate recording of ballots under these conditions. GAO expects these three tests would take 2 weeks, once the necessary arrangements are made. Should the task force ask GAO to conduct the proposed tests, several matters would need to be addressed before testing could begin, including obtaining access to the iVotronic DREs that have been subject to a sequestration order, arranging for a test site, obtaining some commercially available test tools, developing test protocols and detailed test procedures, and arranging for the video recording of the tests. Sarasota County election officials have indicated that they can help GAO access the machines and provide a test site between November 26 and December 7, 2007. Although the proposed tests could help provide increased assurance, they would not provide absolute assurance that the iVotronic DREs did not cause the large undervote in Sarasota County. The successful conduct of the proposed tests could reduce the possibility that the voting systems caused the undervote and shift attention to the possibilities that the undervote was the result of intentional actions by voters or voters that did not properly cast their votes on the voting system.

section_title: Letter
paragraphs: I am pleased to appear before the task force today to present the findings  on our review of voting equipment used in Florida’s 13th Congressional  District (Florida-13), which we are conducting in response to your request  of May 25, 2007.
In November 2006, about 18,000 undervotes were reported in Sarasota  County in the race for Florida’s 13th Congressional District. Following the  contesting of the election results in the House of Representatives, the task  force met and unanimously voted to seek GAO’s assistance in determining  whether the voting systems contributed to the large undervote in Sarasota  County. On June 14, 2007, we met with the task force and agreed upon an  engagement plan, which included the following review objectives:  (1) What voting systems and equipment were used in Sarasota County and  what processes governed their use? (2) What was the scope of the  undervote in Sarasota County in the general election? (3) To what extent  were tests conducted on the voting systems in Sarasota County prior to  the general election and what were the results of those tests?  (4) Considering the tests that were conducted on the voting systems from  Sarasota County after the general election, are additional tests needed to  determine whether the voting systems contributed to the undervote?
To conduct our work, we met with officials from the Sarasota County  Supervisor of Elections, the Florida Department of State and Division of  Elections, and Election Systems and Software (ES&S), the manufacturer  of the voting systems used in Sarasota County. We reviewed voting system  documentation, including standards documents, audit and testing  documentation, submissions from the contestant and contestee, and  selected Florida election laws and rules. In Sarasota County, election  officials demonstrated how the ES&S voting system was used to support  the 2006 general election. To determine the scope of the undervote in  Sarasota County, we collected election data from the Supervisor of  Elections and analyzed it to determine whether the undervote could be  attributed to particular voting machines or machine characteristics.  Specifically, we examined ballot image logs and event logs from the voting  systems and technician and incident reports generated by elections staff  from Sarasota County on election day. We also conducted various  statistical analyses to characterize the undervote and to identify whether a  subset of machines or precincts may have caused the large undervote.
We reviewed test documentation and interviewed officials involved with  testing from ES&S, the Florida Division of Elections, and the Sarasota  County Supervisor of Elections. To determine the need for additional  tests, we also reviewed the tests conducted following the election,  including those conducted or sponsored by the Florida Division of  Elections, including the parallel testing, the examination of Sarasota  County’s election procedures and practices, and the source code review  conducted at Florida State University’s Security and Assurance in  Information Technology (SAIT) laboratory. We reviewed the final reports  of these tests and also met with the leader of the source code review team.  Following the agreement to and execution of a non-disclosure agreement  with the Florida Department of State and ES&S, we obtained access to the  iVotronic source code and reviewed it to further our understanding of the  system and to verify some of the source code review’s findings. We  analyzed the available information and identified a key set of voting  system objectives that, if implemented properly, would provide reasonable  assurance that the voting systems did not malfunction and cause the large  undervote in Sarasota County. Using these objectives, we used the results  of testing previously conducted and assessed the extent to which these  key voting system objectives could be met. For those objectives that could  not be adequately assured, we assessed the significance of those  objectives and identified tests that could be conducted to help try to  assure those key voting system objectives were met. For each test, we  identified resources that would be required, including time and manpower.
We provided a draft of this report to the Florida Department of State,  ES&S, and the Sarasota County Supervisor of Elections for their review  and comments. The Florida Department of State and ES&S also conducted  a sensitivity review to ensure that business proprietary information is not  disclosed in this statement.
We conducted our work from June to September 2007 in Washington, D.C.;  Tallahassee and Sarasota, Florida; and Omaha, Nebraska.
subsections: section_title: Background
paragraphs: The 13th Congressional District of Florida comprises DeSoto, Hardee,  Sarasota, and parts of Charlotte and Manatee Counties. In the November  2006 general election, there were two candidates in the race to represent  the 13th Congressional District: Vern Buchanan, the Republican candidate,  and Christine Jennings, the Democratic candidate. The State of Florida  certified Vern Buchanan the winner of the election. The margin of victory  was 369 votes out of a total of 238,249 votes counted. Table 1 summarizes  the results of the election and shows that the results from Sarasota County  exhibited a significantly higher undervote rate than in the other counties in  the congressional district.
In Florida, the Division of Elections in the Secretary of State’s office helps  the Secretary carry out his or her responsibilities as the chief election  officer. The Division of Elections is responsible for establishing rules  governing the use of voting systems in Florida. Voting systems cannot be  used in any county in Florida until the Florida Division of Elections has  issued a certification of the voting system’s compliance with the Florida  Voting System Standards. The Florida Voting Systems Certification  program is administered by the Bureau of Voting Systems Certification in  the Division of Elections.
An elected supervisor of elections is responsible for implementing  elections in each county in Florida in accordance with Florida election  laws and rules. The supervisor of elections is responsible for the purchase  and maintenance of the voting systems as well the preparation and use of  the voting systems to conduct each election.
section_title: Sarasota County Used ES&S Voting Systems in 2006 General Elections
paragraphs: In the 2006 general election, Sarasota County used voting systems  manufactured by ES&S. The State of Florida has certified different  versions of ES&S voting systems. The version used in Sarasota County was  designated ES&S Voting System Release 4.5, Version 2, Revision 2, and  consisted of iVotronic DREs, a Model 650 central count optical scan  tabulator for absentee ballots, and the Unity election management system.  It was certified by the State of Florida on July 17, 2006. The certified  system includes different configurations and optional elements, several of  which were not used in Sarasota County.
The election management part of the voting system is called Unity; the  version that was used was 2.4.4.2. Figure 1 shows the overall election  operation using the Unity election management system and the iVotronic  DRE.
Sarasota County used iVotronic DREs for early and election day voting.  Specifically, Sarasota County used the 12-inch iVotronic DRE, hardware  version 1.1 with firmware version 8.0.1.2. Some of the iVotronic DREs are  configured with Americans with Disabilities Act (ADA) functionality,  which includes the use of audio ballots. The iVotronic DRE uses a touch  screen—a pressure-sensitive graphics display panel—to display and  record votes (see fig. 2).
The machine has a storage case that also serves as the voting booth. The  operation of the iVotronic DRE requires using a personalized electronic  ballot (PEB), which is a storage device with an infrared window used for  transmission of ballot data to and from the iVotronic DRE. The iVotronic  DRE has four independent flash memory modules, one of which contains  the program code—firmware—that runs the machine and the remaining  three flash memory modules store redundant copies of ballot definitions,  machine configuration information, ballots cast by voters, and event logs.  The iVotronic DRE includes a VOTE button that the voter has to press to  cast a ballot and record the information in the flash memory. The  iVotronic DRE also includes a compact flash card that can be used to load  sound files onto iVotronic DREs with ADA functionality. The iVotronic  DRE’s firmware can be updated through the compact flash card.  Additionally, at the end of polling, the ballots and audit information are to  be copied from the internal flash memory module to the compact flash  card.
To use the iVotronic DRE for voting, a poll worker activates the iVotronic  DRE by inserting a PEB into the PEB slot after the voter has signed in at  the polling place. After the poll worker makes selections so that the  appropriate ballot will appear, the PEB is removed and the voter is ready  to begin using the system. The ballot is presented to the voter in a series of  display screens, with candidate information on the left side of the screen  and selection boxes on the right side (see fig. 3).
The voter can make a selection by touching anywhere on the line, and the  iVotronic DRE responds by highlighting the entire line and displaying an X  in the box next to the candidate’s name. The voter can also change his or  her selection by touching the line corresponding to another candidate or  by deselecting his or her choice. “Previous Page” and “Next Page” buttons  are used to navigate the multipage ballot. After completing all selections,  the voter is presented with a summary screen with all of his or her  selections (see fig. 4). From the summary screen, the voter can change any  selection by selecting the race. The race will be displayed to the voter on  its own ballot page. When the voter is satisfied with the selections and has  reached the final summary screen, the red VOTE button is illuminated,  indicating the voter can now cast his or her ballot. When the VOTE button  is pressed, the voting session is complete and the ballot is recorded on the  iVotronic DRE. In Sarasota County’s 2006 general election, there were nine  different ballot styles with between 28 and 40 races, which required  between 15 and 21 electronic ballot pages to display, and 3 to 4 summary  pages for review purposes.
section_title: Analysis of Election Data Shows that Undervote Was Distributed across All Machines and Precincts
paragraphs: Our analysis of the 2006 general election data from Sarasota County does  not identify any particular voting machines or machine characteristics that  could have caused the large undervote in Florida’s 13th Congressional  District race. The undervotes in Sarasota County for the congressional  race were generally distributed across all machines and precincts. Using  voting system data that we obtained from Sarasota County, we found that  1,499 iVotronic DREs recorded votes in the 2006 general election;  84 iVotronic DREs recorded votes during early voting, and 1,415 iVotronic  DREs recorded votes on election day. Using these data, we verified that  the vote counts for the contestant, contestee, and undervotes match the  reported vote totals for Sarasota County in Florida’s 13th Congressional  District race. As can be seen in table 2, the undervote rate in early voting  was significantly higher than in election day voting.
The range of the undervote rate for all machines was between 0 and  49 percent, with an average undervote rate of 14.3 percent. When just the  early voting machines are considered, the undervote rate ranged between  5 and 28 percent. The largest number of undervotes cast on any one  machine on election day was 39. While the range of ballots cast on any one  machine on election day was between 1 and 121, the median number of  ballots cast on any one machine was 66. The range of undervote rate by  precinct was between 0 and 41 percent, and the average undervote by  precinct was about 14.8 percent.
section_title: Prior Tests and Reviews Provide Some Assurance, but Do Not Provide Reasonable Assurance That the iVotronic DREs Did Not Contribute to the Undervote
paragraphs: Prior to the elections, Sarasota County’s voting systems were subjected to  several different tests that included testing by the manufacturer,  certification testing by the Florida Division of Elections, testing by  independent testing authorities, and logic and accuracy testing by Sarasota  County’s Supervisor of Elections. After the 2006 general election, an audit  of Sarasota County’s election was conducted by the State of Florida that  included a review of the iVotronic source code, parallel tests, and an  examination of Sarasota County’s election procedures. Although these  tests and reviews provide some assurance, as do certain controls that were  in place during the election, that the voting systems in Sarasota County  functioned correctly, they do not provide reasonable assurance that the  iVotronic DREs did not contribute to the undervote.
subsections: section_title: Prior Tests and Reviews of Sarasota County’s Voting Systems Provide Useful Information, but Have Some Shortcomings
paragraphs: According to ES&S officials, ES&S tested the version of the iVotronic DRE  that was used in Sarasota County in 2001-2002, but they could not provide  us documentation for those tests because the documentation had not been  retained.
The Florida Division of Elections conducted certification testing of the  iVotronic DRE and the Unity election management system before Sarasota  County acquired the system from the manufacturer. The certification  process included tests of the election management system and the conduct  of mock primary and general elections on the entire voting system. ES&S  Voting System, Release 4.5, Version 2, Revision 2, was certified by the  Florida Division of Elections on July 17, 2006. According to Florida  Division of Elections officials, testing of each version focuses on the new  components, and components that were included in prior versions are not  as vigorously tested. The 8.0.1.2 version of the iVotronic firmware was first  tested as a part of ES&S Release 4.5, Version 1, which was certified in  2005. Version 2 introduced version 2.4.4.2 of the Unity Election  Management System, which was certified in August 2005. Certification  testing was conducted on software that was received from an independent  test authority, who witnessed the building of the firmware from the source  code. An independent test authority also conducted environmental testing  of the iVotronic DRE in 2001 that was relied upon by the Florida Division  of Elections for certification.
A logic and accuracy test was conducted by Sarasota County on   October 20, 2006, on 32 iVotronic DREs, and it successfully verified that all  ballot positions on all nine ballot styles could be properly recorded. In  addition, the use of a provisional ballot and audio ballot were tested, as  well as machines configured for early voting with all nine ballot styles.
After the 2006 general election, the Florida Division of Elections  conducted an audit of Sarasota County’s 2006 general election that  included two parallel tests, an examination of the certified voting system  and conduct of election by Sarasota County’s elections office, and an  independent review of the iVotronic DRE firmware’s source code. After  the conduct of this audit, the audit team concluded that there was no  evidence that suggested the official election results were in error or that  the voting systems contributed to the undervote in Sarasota County. The  parallel tests were performed using 10 iVotronic DREs—5 used in the 2006  general election and 5 that were not used. Four of the machines in each  test replicated the votes cast on four election day iVotronic DREs. The  fifth machine in each test used an ad hoc test script that involved picking a  random vote pattern along with a specific vote selection pattern picked  from 10 predetermined vote patterns for the 13th Congressional District  for each ballot cast. The audit report asserts that testing a total of  10 machines is more than adequate to identify any machine problems or  irregularities that could have contributed to undervotes in the Florida-13  race. However, we concluded that the results from the testing of  10 machines cannot be applied to all 1,499 iVotronic DREs used during the  2006 general election because the sample was not random and the sample  size was too small.
In examining whether voting systems that were used in Sarasota County  matched the systems that were certified by the Florida Division of  Elections, the Florida audit team examined the Unity election management  system and the firmware installed on six iVotronic DREs. The audit team  confirmed that the software running on the Unity election management  system and the firmware in the six iVotronic DREs matched the certified  versions held in escrow by the Florida Division of Elections. On the basis  of its review, the audit team concluded that there is no evidence to  indicate that the iVotronic DREs had been compromised or changed. We  agree that the test verifies that those six machines were not changed, but  any extrapolation beyond this cannot be statistically justified because the  size of the sample is too small. Therefore, these tests cannot be used to  obtain reasonable assurance that the 1,499 machines used in the general  election used the certified firmware.
A software review and security analysis of the iVotronic firmware version  8.0.1.2 was conducted by a team led by Florida State University’s SAIT  Laboratory. The eight experts in the software review team attempted to  confirm or refute many different hypotheses that, if true, might explain the  undervote in the race for the 13th Congressional District. In doing so, they  made several observations about the code, which we were able to  independently verify. The software review and our verification of the  observations were helpful, but a key shortcoming was the lack of  assurance whether the source code reviewed by the SAIT team or by us, if  compiled, would correspond to the iVotronic firmware that was used in  Sarasota County for the 2006 election. According to ES&S and Florida  Division of Elections officials, in May 2005 an independent testing  authority witnessed the process of compiling the source code and building  the version of firmware that was eventually certified by the Florida  Division of Elections. According to ES&S officials, if necessary, ES&S can  recreate the firmware from the source code, but the firmware would not  be exactly identical to the firmware certified by the Florida Division of  Elections because the embedded date and time stamp in the firmware  would be different.
The software review team also looked for security vulnerabilities in  software that could have been exploited to cause the undervote. Although  the team found several software vulnerabilities, the team concluded that  none of them were exploited in Sarasota in a way that would have  contributed to the undervote. We did not independently verify the team’s  conclusion.
section_title: Reasonable Assurance of Some Voting System Objectives Has Been Achieved
paragraphs: The Unity election management system and the iVotronic DREs are the  major voting system components that may require testing to determine  whether they contributed to the large undervote in Sarasota County. Our  review of tests already conducted and documentation from the election  provide us reasonable assurance that the key functions of the Unity  election management system—election definition and vote tabulation— did not contribute to the undervote. The election definitions created using  the Unity election management system are tested during logic and  accuracy testing to demonstrate that they include all races, candidates,  and issues and that each of the items can be selected by a voter. The votes  tabulated on the iVotronic DRE at each precinct matched the data  uploaded to the Unity election management system, and the totals from  the precinct results tapes agree with that obtained by Unity. Further, the  state audit confirmed that the Unity election management system software  running in Sarasota County matched the escrowed version certified by the  Florida Division of Elections.
We have reasonable assurance that the number of ballots recorded by the  iVotronic DREs is correct because this number is very close to the number  of people recorded on the precinct registers as showing up at the polling  places to vote either during early voting or on election day. This assurance  also allows us to conclude that issues, such as votes cast by “fleeing  voters”—votes that are cast by poll workers for voters who leave the  polling place before pressing the button to cast the vote—and the potential  loss of votes during a system shutdown, did not affect the undervote in  this election. If these issues had occurred, they would have caused a  discrepancy between the number of voters who sign in at the polling place  to vote and the public counts recorded on the iVotronic DREs.
We have reasonable assurance that provisional ballots were appropriately  handled by the iVotronic DREs and the Unity election management  system. We also verified that during the Florida certification test process,  the Division of Elections relied on successful environmental and shock  testing conducted by an independent test authority.
section_title: Reasonable Assurance That All iVotronic DREs Used in the 2006 General Election Used Software Certified by the Florida Division of Elections Is Lacking
paragraphs: We found that prior testing and activities do not provide reasonable  assurance that all iVotronic DREs used in Sarasota County on election day  were using the hardware and firmware certified for use by the Florida  Division of Elections. Sarasota County has records indicating that only  certified versions were procured from ES&S, and the firmware version is  checked in an election on the zero and results tapes. However, because  there was no independent validation of the system versions, we cannot  conclude that no modifications were made to the systems that would have  likely made them inconsistent with the certified version. As we previously  mentioned, the firmware comparison of only 6 iVotronic DREs in the state  audit is insufficient to support generalization to all 1,499 iVotronic DREs  that recorded votes during the election. Without reasonable assurance that  all iVotronic DREs are running the same certified firmware, it is difficult  for us to rely on the results of other testing that has been conducted, such  as the parallel tests or the logic and accuracy tests.
section_title: The Ability of Voters to Make Selections in Different Ways and Have Their Votes Properly Recorded Has Not Been Fully Tested
paragraphs: Prior testing of the iVotronic DREs only verified 13 of the 112 ways that we  identified that a voter may use to select a candidate in Florida’s 13th  Congressional District race. Specifically, on an iVotronic DRE, a voter  could (1) initially select either candidate or neither candidate (i.e.  undervote), (2) change the vote on the initial screen, and (3) use a  combination of page back and review screen options to change or verify  his or her selection before casting the ballot. By taking into account these  variations, our analysis has found at least 112 different ways a voter could  make his or her selection in Florida’s 13th Congressional District race,  assuming that it was the only race on the ballot. Out of 112 different ways  to select a candidate in the congressional race, Florida certification tests  and the Sarasota County logic and accuracy tests verified 3 ways to select  a candidate; and the Florida parallel tests verified 10 ways to select a  candidate—meaning that of the 112 ways, 13 have been tested. By not  verifying these different ways to select a candidate, we do not have  reasonable assurance that the system will properly handle expected forms  of voter behavior.
section_title: The Effect of Miscalibrated iVotronic DREs Is Unclear
paragraphs: During the setup of the iVotronic DRE, sometimes referred to as the clear  and test process, the touch screens are calibrated by using a stylus to  touch the screen at 20 different locations. The calibration process is  designed to align the display screen with the touch screen input. It has  been reported that a miscalibrated machine could affect the selection  process by highlighting a candidate that is not aligned with what the voter  selected. We identified two reported cases on election day where the  miscalibration of the iVotronic DRE led to its closure and discontinued use  for the rest of the day. While a miscalibrated machine could certainly  make an iVotronic DRE harder to use, it is not clear it would have helped  to contribute to the undervote. We did not identify any prior testing or  activities that would help us understand the effect of a miscalibrated  iVotronic DRE on the undervote.
section_title: Further Tests Could Provide Increased but Not Absolute Assurance That the iVotronic DREs Used in the Election Did Not Cause the Undervote
paragraphs: On the basis of our analysis of all prior test and audit activities, we  propose that a firmware verification test, a ballot test, and a calibration  test be conducted to try to obtain increased assurance that the iVotronic  DREs used in Sarasota County during the 2006 general election did not  cause the undervote.
We propose that the firmware verification testing be started first, once the  necessary arrangements have been made, such as access to the needed  machines and the development of test protocols and detailed test  procedures. Once we have reasonable assurance that the iVotronic DREs  are running the same certified firmware, we could conduct the ballot test  and calibration test on a small number of machines to determine whether  it is likely the machines accurately recorded and counted the ballots. If the  firmware verification tests are successfully conducted, we would have  much more confidence that the iVotronic DREs will behave similarly when  tested. If there are differences in the firmware running on the iVotronic  DREs, we would need to reassess the number of machines that need to be  tested for ballot testing and calibration testing in order for us to have  confidence that the test results would be true for all 1,499 iVotronic DREs  used during the election. In other words, if we are reasonably confident  that the same software is used in all 1,499 machines, then we are more  confident that the results of the other tests on a small number of machines  can be used to obtain increased assurance that the iVotronic DREs did not  cause the undervote. Although the proposed tests would provide increased  assurance, they would not conclusively eliminate the machines as a cause  of the undervote.
subsections: section_title: Conduct Firmware Testing to Verify That the Firmware in the iVotronic DREs Used in Sarasota County Matches the Certified Version
paragraphs: We propose to conduct a firmware verification test using a statistical  sampling approach that can provide reasonable assurance that all  1,499 iVotronic DREs are running the certified version of firmware. The  exact number of machines that would be tested depends on the  confidence level desired and how much error can be tolerated. We  propose drawing a representative sample from all the iVotronic DREs that  recorded votes in the general election. With a sample size of 115 iVotronic  DREs, which would be divided between sequestered and nonsequestered  machines, and assuming that there are no test failures, we would be able  to conclude with a 99 percent confidence level that no more than 4 percent  of the 1,499 iVotronic DREs used in the election were using uncertified  firmware.
We suggest a test approach similar to what was used by the Florida  Division of Elections when it verified the firmware for 6 iVotronic DREs.
We estimate that the firmware testing for 115 machines could be  conducted in about 5 to 7 days and would require about 5 or 6 people,  once the necessary arrangements have been made. The machines would  be transported to a test facility specified by Sarasota County election  officials where we could perform the test. The activities involved in  conducting a firmware validation test would include locating and  retrieving the selected iVotronic DRE from the storage facility,  transporting it to the test facility, opening the DRE, extracting the chip  with the firmware, reading the contents of the chip using a specialized  chip reader, and conducting a comparison between the contents and the  certified firmware to determine if any differences exist. To conduct this  test, we would need commercially available specialized hardware and  software similar to that used by the Florida Division of Elections in its  firmware comparison test.
section_title: Conduct Ballot Testing of iVotronic DREs to Confirm Correct Operation
paragraphs: We propose conducting ballot testing on 10 iVotronic DREs, each  configured with one of the nine different ballot styles, with the 10th  machine configured as an early voting machine with all nine ballot styles.  We would test 112 ways to select a candidate on the early voting machine.  On the election day machines, we would test the 112 different ways  distributed across the 9 machines in a random manner, meaning each  machine would on average record 12-13 ballots. Assuming that  (1) reasonable assurance is obtained that all iVotronic DREs used during  the election were using the same certified firmware, and (2) we found no  failures during the ballot testing, this testing would provide increased  assurance that the iVotronic DREs used during the election, both in early  voting and in election day voting, were able to accurately record and count  ballots when using any of the 112 ways to select a candidate in the  Florida-13 race.
We would plan to code each ballot by including an identifier in the write-in  candidate field for either the U.S. senator or governor’s race. Using this  write-in coding, we could examine the ballot image and confirm that each  ballot was accurately recorded and counted by the iVotronic DRE. Any  encountered failures would also be more rapidly attributed to a specific  test case, and we would be able to more readily repeat the test case to  determine if we have a repeatable condition. Testing 112 ways to select a  candidate on a single machine would also provide us some additional  assurance that the volume of ballots cast on election day did not cause a  problem. We note that casting 112 ballots on a single machine is more than  that cast on over 99 percent of the 1,415 machines used on election day.
We estimate the ballot testing would take about 2 to 3 days and require the  equivalent of 2 people, once the necessary arrangements have been made.
section_title: Deliberately Miscalibrate an iVotronic DRE to Understand the Effect on the Undervote
paragraphs: Because little is known about the effect of a miscalibrated machine on the  behavior of an iVotronic DRE, we propose to deliberately miscalibrate an  iVotronic DREs and verify the functioning of the machine. We propose to  identify different ways to miscalibrate a ballot and to test ballots on the  miscalibrated iVotronic DRE to verify that it still properly records votes.  With this test we would confirm whether (1) the review screen displays  the same selection in the Florida-13 race as was highlighted in the  selection screen, and (2) that the vote is recorded as it was displayed on  the review screen. Again, we would plan to use the write-in candidate  option to verify the proper recording of the ballot. This test would  demonstrate whether the system correctly records a vote for the race and  hence whether it contributed to the undervote. We estimate that the  calibration test could be completed in about 1 day by 2 people, once the  necessary arrangements have been made.
section_title: Several Matters Remain to Be Addressed to Conduct Further Testing
paragraphs: Should the task force ask us to conduct the proposed testing, we want to  make the task force aware of several other matters that would need to be  addressed before we could begin testing. These activities would require  some time and resources to complete before testing could commence.
First, we would need to gain access to iVotronic DREs that have been  subject to a sequestration order in the state court system of Florida. If we  do not have access to the needed machines, we would be unable to obtain  reasonable assurance that the machines used on election day were using  certified software, and without this assurance, the results from prior tests  and any results of our ballot and calibration tests would be less  meaningful because we would be unable to apply the results to all  1,499 iVotronic DREs used during the election. Second, we would need to  agree upon an appropriate facility for the tests. Sarasota County  Supervisor of Elections has indicated that we can use its warehouse space,  but because of upcoming elections in November and January, the only  time the election officials would be able to provide us this space and the  necessary support is between November 26 and December 7, 2007. If  testing cannot be completed during this time period, Sarasota County  officials stated that they would not be able to assist us until February 2008.  Third, some tests may require commercially available specialized software,  hardware, or other tools to conduct the tests. We would need to make  arrangements to either borrow or to purchase such testing tools before  commencing testing. Fourth, in order to conduct any tests, we would need  to develop test protocols and detailed test procedures and steps. We also  anticipate that we would need to conduct a dry run, or dress rehearsal, of  our test procedures to ensure that our test tools function properly and that  our time estimates are reasonable. Finally, we would need to make  arrangements for video recording of our testing. It would be our  preference to have a visual record of the tests to document the actual test  conduct and to facilitate certain types of test analysis.
section_title: Other Observations on Touch Screen Voting Systems
paragraphs: We recognize that human interaction with the ballot layout could be a  potential cause of the undervote. Although we have not explored this issue  in our review, we note that there is an ongoing academic study that is  exploring this issue using voting machines obtained from ES&S. We  believe that such experiments could be useful and could provide insight  into the ballot layout issue.
During our review, we noted that several suggestions have been offered as  possible ways to establish that voters are intentionally undervoting and to  provide some assurance that the voting systems did not cause the  undervote. First, a voter-verified paper trail could provide an independent  confirmation that the touch screen voting systems did not malfunction in  recording and counting the votes from the election. The paper trail would  reflect the voter’s selections and, if necessary, could be used in the  counting or recounting of votes. This issue is recognized in the Florida  State University SAIT source code review as well as the 2005 and draft  2007 Voluntary Voting Systems Guidelines prepared for the Election  Assistance Commission. We have previously reported on the need to  implement such a function properly. Second, explicit feedback to voters  that a race has been undervoted and a prompt for voters to affirm their  intent to undervote might help prevent many voters from unintentionally  undervoting a race. On the iVotronic DREs, such feedback and prompts  are provided only when the voter attempts to cast a completely blank  ballot, but not when a voter undervotes in individual races. Third, offering  a “none of the above” option in a race would provide voters with the  opportunity to indicate that they are intentionally undervoting. The State  of Nevada provides this option in certain races in its elections. Decisions  about these or other suggestions about ballot layout or voting system  functions should be informed by human factors studies that assess their  effectiveness in accurately recording voters’ preferences, making voting  systems easier to use, and preventing unintentional undervotes.
section_title: Conclusions
paragraphs: The high undervote encountered in Sarasota County in the 2006 election  for Florida’s 13th Congressional District has raised questions about  whether the voting systems accurately recorded and counted the votes  cast by eligible voters. Other possible reasons for the undervote could be  that voters intentionally undervoted or voters did not properly cast their  ballots on the voting systems, potentially because of issues relating to the  interaction between voters and the ballot. The focus of our review has  been to determine whether the voting systems—the iVotronic DREs, in  particular—contributed to the undervote. We found that the prior reviews  of Sarasota County’s 2006 general election have provided valuable  information about the voting systems. Our review found that in some cases  we were able to rely on this information to eliminate areas of concern.  This allowed us to identify the areas where increased assurances were  needed to answer the questions being raised. Accordingly, the primary  focus of the tests we are proposing is to obtain increased assurance that  the results of the prior reviews and our proposed testing can be applied to  all the iVotronic DREs used in the election. Our proposed tests involving  the firmware comparison, ballot testing, and calibration testing could help  reduce the possibility that the undervote was caused by the iVotronic  DREs. However, even after completing the tests, we would not have  absolute assurance that the iVotronic DREs did not play any role in the  large undervote. Absolute assurance is impossible to achieve because we  are unable to recreate the conditions of the election in which the  undervote occurred. By successfully conducting the proposed tests, we  could reduce the possibility that the iVotronic DREs were the cause of the  undervote and shift attention to the possibilities that the undervote was  the result of intentional actions by the voter or voters that did not properly  cast their votes on the voting system.
section_title: Comments and Our Evaluation
paragraphs: We provided draft copies of this statement to the Secretary of State of  Florida, the Supervisor of Elections of Sarasota County, and ES&S for  review and comment. The Florida Department of State provided technical  comments, which we incorporated. The Sarasota County Supervisor of  Elections appreciated the opportunity to review the draft, but provided us  no comments.
In its comments, ES&S stated that it believes that the collective results of  testing already conducted on the Sarasota County voting systems have  demonstrated that they performed properly and as they were designed to  function and that all votes were accurately captured and counted as cast  in Florida’s 13th Congressional District race. Further, ES&S asserts that  tests and analyses should be conducted to examine the effect of the ballot  display on the undervote, which it believes is the most probable cause of  the undervote.
We disagree that the collective results of testing already conducted on the  Sarasota County voting systems adequately demonstrate that the voting  systems could not have contributed to the undervote in the Florida-13  race. First, as we have cited, we do not have adequate assurance that all  the iVotronic DREs used in Sarasota County used the firmware certified by  the Florida Division of Elections. Without this assurance, it is difficult for  us to apply the results from the other tests to all 1,499 machines that  recorded votes during the election because we are uncertain that all  machines would have behaved in a similar manner. Further, we believe  that expected forms of voter behavior to select a candidate in the  Florida-13 race were not thoroughly tested. While ES&S asserts that such  processes would have no effect on the iVotronic DRE’s ability to capture  and record a voter’s selection, we did not identify testing that verified this.  Further, while ES&S states that the testing of a deliberately miscalibrated  iVotronic DRE would result in a clearly visible indication of which  candidate was selected, we could not identify any testing that  demonstrated this.
We acknowledge that the large undervote in Florida’s 13th Congressional  District race could have been caused by voters who intentionally  undervoted or voters who did not properly cast their ballots, potentially  because of issues related to the human interaction with the ballot.  However, the focus of our review, as agreed with the task force, was to  review whether the voting systems could have contributed to the large  undervote. ES&S also provided technical comments, which we  incorporated as appropriate.
Mr. Chairman, this completes my prepared statement. I would be happy to  respond to any questions you or other members of the task force may have  at this time.
section_title: Contacts and Acknowledgments
paragraphs: For further information about this statement, please contact Keith Rhodes,  Chief Technologist, at (202) 512-6412 or rhodesk@gao.gov, or Naba  Barkakati at (202) 512-4499 or barkakatin@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this statement. Other key contributors to this statement  include James Ashley, James Fields, Jason Fong, Cynthia Grant, Geoffrey  Hamilton, Richard Hung, John C. Martin, Jan Montgomery, Jennifer  Popovic, Sidney Schwartz, and Daniel Wexler.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.