Preliminary Report on the Development of a User-Based Conformance Test
for the Usability of Voting Equipment

Prepared by:

**User-Centered Design, Inc.**

**Submitted to:**

**The National Institute of Standards and Technology**

**Date:**

**March 10, 2006**

Table of Contents

[BACKGROUND 4](#background)

[ENSURING THE VALIDITY OF THE TEST
5](#ensuring-the-validity-of-the-test)

[DETERMINING BALLOT CHARACTERISTICS
7](#determining-ballot-characteristics)

[DEFINING THE TEST BALLOT 11](#defining-the-test-ballot)

[Ballot Style 11](#ballot-style)

[Write-ins 12](#write-ins)

[Nature of the Contest Data on the Ballot
12](#nature-of-the-contest-data-on-the-ballot)

[Nature of the Referenda 12](#nature-of-the-referenda)

[VOTING TASK ANALYSIS 12](#voting-task-analysis)

[CREATING A TEST BALLOT TO REPRESENT THE VOTING TASKS
16](#creating-a-test-ballot-to-represent-the-voting-tasks)

[PARTICIPANT DEFINITION 17](#participant-definition)

[CONDUCTING THE TEST 18](#conducting-the-test)

[ANALYSIS OF DATA 19](#analysis-of-data)

[How will we measure voting accuracy?
19](#how-will-we-measure-voting-accuracy)

[How will we handle data from participants who fail to cast their
ballot?
20](#how-will-we-handle-data-from-participants-who-fail-to-cast-their-ballot)

[How will we measure efficiency? 20](#how-will-we-measure-efficiency)

[How will we measure satisfaction?
20](#how-will-we-measure-satisfaction)

[CURRENT RESEARCH ISSUES 20](#current-research-issues)

[Non-English Ballots 20](#non-english-ballots)

[Usability by Users with Disabilities
21](#usability-by-users-with-disabilities)

# BACKGROUND

A resolution of the TGDC (#05-05) calls for the development of "human
performance based standards and usability testing" for voting systems.
The Voluntary Voting System Guidelines (VVSG 2005) confirm the intention
to add requirements for conformance tests for usability based on human
performance testing in the future.

These conformance tests would establish whether a system met performance
requirements, as measured through "human performance tests, conducted
with human voters as the test participants" (TGDC #05-05). This is a new
type of test for certifying voting systems, and the TGDC directed NIST
to develop this testing method. NIST contracted with User-Centered
Design, Inc., (UCD) a human factors engineering consulting firm, to
develop this test.

There are many different kinds of usability tests serving various
purposes. Examples include:

-   Exploratory testing in the early design stage of product development
    (usually called formative testing in the usability field),

-   Debugging (discovering and resolving defects),

-   Comparative testing of competing products,

-   Summative testing of a completed product to determine how well it
    meets its usability objectives

Conformance testing is distinct from all these. Conformance tests are
used to determine whether or not a particular product meets the
requirements of a standard. Although the test procedure may involve
measurement of certain quantities (e.g. the number of errors or the time
required to complete a task), the final result of a conformance test is
simply a "pass" (the product meets the requirement in question) or a
"fail" (the requirement is not met). Of course, conformance testing may
often have some incidental diagnostic value, but this is not its main
purpose.

Conformance testing with human subjects presents certain special
difficulties:

-   There are no \"natural\" underlying metrics on which to build a
    requirement. Therefore, to some extent, the design of the test
    becomes the design of the requirement. In the case of voting, we
    must somewhat artificially constrain the choice of human subjects
    and the tasks they are to perform in order to make tests feasible
    and repeatable. But the testing must be realistic enough that the
    results are credibly relevant to the quality of the systems\' actual
    performance.

-   There are no natural pass/fail cutoff points. For instance, there is
    no way to tell beforehand exactly how much time subjects \"should\"
    take to complete a given task. Rather, the benchmarks must be based
    on the results of actual experimentation. The goal is to set
    benchmarks that are strong enough to weed out truly inadequate
    systems, while still being reasonably attainable for good systems.

To define a conformance test for the usability of a voting machine using
human test subjects, a significant amount of research is required. The
work on this project is being conducted in two phases:

1.  **The pilot phase**: develop a test methodology and procedures and
    validate by running the test under controlled conditions.

2.  **The development phase**: develop the exact test procedure,
    including the reference ballot and instructions, and ensure that the
    test can be conducted by a Voting System Testing Lab (VSTL).

This document reports on the work to date in the **pilot phase**,
outlining the preliminary decisions that have been made regarding the
test, and the current plans to define the validation tests.

# ENSURING THE VALIDITY OF THE TEST

Summary: For the test to be suitable for conformance testing it must be
a valid test that produces consistent results.

The goal of the test will be to demonstrate, as part of the
certification process, that a machine meets or exceeds usability
requirements that will be defined in the VVSG. To do this, we need to
establish a baseline for satisfactory performance on usability measures.
Each new machine will then be tested to see if it meets or exceeds this
baseline. If it does, the machine will pass the usability portion of the
certification process.

The conformance test must be valid and reliable, producing consistent
results even when the test procedure is administered by different Voting
System Testing Labs (VSTLs). One of the goals of the pilot phase is to
demonstrate that a test can be defined for the usability of voting
machines using human subjects as participants that meets the conditions
for validity and reliability.

Before we describe the test, it is important to understand what we mean
by "validity". **Validity** is the degree to which the results of a
research study provide trustworthy information about the truth or
falsity of the hypothesis (From Cherulnik, P.D. (2001). *Methods for
Behavioral Research: A Systematic Approach*. Thousand Oaks, CA: Sage
Press, p. 466)). Validity has two major categories: external and
internal validity.

**External validity** asks the question of "generalizability" -- how
well the results from a test can be used to determine how a larger
population will perform (Ibid). However, this test will need to be
conducted using a sub-population of voters that is small enough to be
cost effective but yet is large enough to produce sufficient data to
determine differences between the usability of different machines. To
meet all of these conditions, this group cannot be a representative
sample of the voting population. Though it is acknowledged that external
validity of the test may be important for general acceptance of the test
(see face validity), there is no need to demonstrate external validity
for a conformance test since results from a machine being considered for
certification will be compared to baseline data derived from results of
an equivalent population that performed the same test as part of the
baseline development.

**Internal validity** refers to the situation where "experimental
treatments make a difference in this specific experimental instance"
(from Cambell, D.T. & Stanley, J.C. (1963) *Experimental and
Quasi-experimental Designs for Research*, Boston, Houghton Mifflin Co,
p. 5). Though experts have differing opinions on the specific number and
types of internal validity, this project is particularly interested in
construct validity, concurrent validity, content validity, and face
validity.

-   **Construct validity** is a demonstration that the test is measuring
    what it claims to measure. It refers to the degree to which
    inferences can legitimately be made from the operationalizations in
    study. In this case, this means that (1) the test does, in fact
    measure usability and (2) that the measurement is based on the
    voting machine itself and not on other factors (e.g., the selection
    of particular subjects). We will be addressing construct validity in
    the pilot test by showing that the measures we use for testing
    usability correlate with other measures of usability. Though there
    appears to be little objective data from other research to use for
    this purpose, we may use existing data from other work performed to
    demonstrate construct validity of our test. For example, in
    preliminary research on two DREs, one of which uses a touch screen
    design and one which does not, the time on task for the non-touch
    screen device was shown to be higher than for the touch screen
    device. If we can show that our test results in the same finding,
    this suggests that we have construct validity for our method of
    testing and calculating time on task. We may also be able to use
    subjective data including expert evaluations. . For example, if
    multiple expert reviews of two devices indicate that a performance
    difference between these devices is expected, and the test provides
    the same finding, this suggests that we have construct validity for
    our method of testing and calculating performance.

-   **Concurrent validity** refers to the relationship between this test
    and other tests that purport to measure the same data. There are (to
    our knowledge), no other tests of this type to compare to, but we
    will use this principle in the development of the test itself. For
    example, we could use concurrent validity to optimize the test for
    costs by demonstrating that a small, more stringently defined
    audience shows concurrent validity with a test that consists of a
    larger number of less stringently defined participants.

-   **Content validity** and **face validity** are closely related.
    Content validity is an assessment that the test is valid as assessed
    by experts. Face validity is an assessment that the test is valid as
    assessed by the casual observer (a non expert). We have developed a
    preliminary task list and a preliminary test ballot for use in the
    pilot test based on a review of some existing ballots, discussions
    with political scientists, and our own research. This material has
    been reviewed for content validity by political science researchers,
    but additional work on a fuller representation of ballots is needed
    to more accurately determine ballot and task characteristics across
    the United States followed by a review by additional subject matter
    experts. Face validity of the procedures will also need to be
    addressed, though the means of accomplishing this is unknown at this
    time.

As mentioned previously, the ballot and tasks have been reviewed for
content validity. For the pilot test, we are planning to use two
machines that differ significantly (e.g., a DRE and a paper based
system). We will use the same voting tests, ballot, and participants for
both systems, and will attempt to show that differences in performance
against the usability measures are based on differences between the two
machines. Comparison of the data from the two machines will be used to
attempt to demonstrate construct validity. The pilot test will be
repeated to attempt to demonstrate test **reliability**.

Once a test has been shown to be valid and reliable (and the pass/fail
criteria have been agreed to), the VTSL must be able to conduct the test
without compromising the test validity and reliability. Several steps
are necessary to ensure this is possible including complete definition
of the recruitment criteria and procedures, test protocol, data
collection, and data analysis. This work will be completed in the
*development phase* of this project.

In any test of this type there exists a possibility that a particular
run of the test is invalidated by a breakdown somewhere in the
preparation, administration, or results analysis. This can be caused by
a \"sampling error\" (e.g., improper recruitment of the participants for
the test), procedural errors somewhere in the test (e.g., accidental
interference by the test facilitator during data collection), or
analysis errors. The test procedure must protect against such errors,
because the result might affect the outcome of the certification. The
test will be designed to ensure that each test run is a true recreation
of the original test.

Our proposed solution is to test two machines at the same time.

-   The first machine is the one being considered for certification.

-   The second machine is a machine whose results on the test have been
    previously collected.

The two machines are be tested together, with participants who are
randomly assigned to each of two test groups (one for each machine). The
test participants are not informed which machine is being considered for
certification. The results of the test from both machines are presented
for analysis. The people conducting the test are not informed which
results are from the machine being considered for certification. If the
analysis of the results from the second machine matches the expected
values, the test is considered valid. The results from the machine being
considered for certification can then be compared to the baseline values
knowing the test was conducted and the results analyzed correctly. This
procedure will be fully defined as part of the overall test protocol for
VTSL testing.

# DETERMINING BALLOT CHARACTERISTICS 

Summary: All tests must use the same reference ballot, so there is no
variation in the test results because of differences in the ballot. We
started by analyzing existing ballots to define a test ballot
specification.

An appropriate ballot specification must be defined as part of this
project, and used for all tests, so that the ballot itself will not be
an independent variable in the test. However, ballots vary widely across
the United States so there is no existing standard ballot specification
that can be used for these tests.

In the development of the ballot specification, it is important that the
specification include dimensions that test usability and are realistic
(i.e., that have content and face validity). For example, if one of the
tasks in the test involves selecting a set of candidates from a long
list of candidates, it is important that the length of the list on the
test ballot be representative of lists on real ballots. Therefore, we
analyzed the characteristics of real ballots. The following dimensions
were defined as dimensions of interest for this project:

1.  **Hierarchical Depth\
    **Where in the government hierarchy each office was located. For
    example, presidential contests are at the top of the hierarchy.
    Contests at the state and local levels are below this top level of
    the hierarchy. We have assumed three levels of hierarchy --
    national, state, and local.

2.  **Number of Contests**\
    The total number of contests on a ballot regardless of the contest
    type or size.

3.  **Partisanship\
    **Whether the party affiliations are indicated on the ballot. Each
    contest was coded as partisan if the candidate's party affiliation
    was indicated on the ballot or as nonpartisan if party affiliation
    was not indicated on the ballot.

4.  **Contest Type\
    **Whether the contest is incumbent-challenger, open seat, or
    retention. In an incumbent-challenger contest, the seat is currently
    occupied by one of the candidates on the ballot, though this may or
    may not be indicated on the ballot. In an open seat contest, none of
    the candidates currently occupies the seat. In a retention contest,
    the voter is asked if this person is (or this group of people are)
    to be retained in office. Though incumbents are sometimes indicated
    on ballots, the total number of incumbent-challenger contests may be
    greater than indicated if states do not require incumbents to be
    indicated on the ballots. Therefore, the ratio of open seat and
    incumbent-challenger contests is unknown. Only retention contents
    are consistently indicated on ballots.

5.  **Number of Seats\
    **The total number of open seats to be filled in a contest. The
    contest is considered single-member if the number of seats to be
    filled is one and multi-member if more than one seat is to be
    filled.

6.  **Field Size\
    **The total number of candidates running in a contest.

7.  **Number of Referenda\
    **The total number of referenda of any type (i.e., constitutional
    amendments, initiatives, etc.) on a ballot.

8.  **Referendum Size**\
    The total number of words in a referendum on the ballot.

9.  **Referendum Type**\
    Whether the full referendum text is included or if only a summary is
    provided. It is coded as "Summary" or "Full Text."

10. **Referendum Reading Level\
    **The level of difficulty to which the text is written. This can be
    measured using a standardized scale such as the Flesh-Kincaid, Flesh
    Reading Ease, or Fog index. These are industry standard for
    determining reading level using formulas based on sentence length
    and number of syllables in sampled or full text of the referenda.

To determine ballot characteristics, we performed a preliminary analysis
of ballots looking at these dimensions. Eighty-two ballots from 16
different states and DC were made available to the project. We discarded
any ballots which were incomplete, used for primary elections, or were
unofficial copies. An analysis of the 14 remaining ballots was conducted
for the pilot test, reviewing each dimension of ballot complexity. This
is a small number, but we believe there was sufficient information to
construct a test ballot for the purpose of pilot testing. A more
complete analysis will be needed before determining a final test ballot
design for VSTL testing, This should include analysis of the
distribution curves for each dimension, using a larger sample of ballots
to fully understand the nature of each dimension as it occurs in the
population of real ballots.

Some of the results of our analysis are shown in the graphs below.

![](media/image1.wmf){width="5.127083333333333in"
height="2.6416666666666666in"}

Figure 1: Number of Contests by Hierarchical Depth

![](media/image2.wmf){width="5.127083333333333in"
height="2.535416666666667in"}

Figure 2: Number of Contests (including referenda) per Ballot

![](media/image3.wmf){width="4.870833333333334in" height="2.4125in"}

Figure 3: Total Number of Contests (not including referenda) by Type

![](media/image4.wmf){width="4.870833333333334in"
height="2.513888888888889in"}

Figure 4: Total Number of Contests by Number of Open Seats

![](media/image5.wmf){width="4.870833333333334in"
height="2.6152777777777776in"}

Figure 5: Total Number of Contests by Field Size

![](media/image6.wmf){width="4.999305555555556in"
height="2.6840277777777777in"}

Figure 6: Total Number of Words per Referenda

# DEFINING THE TEST BALLOT

Summary: For the test ballot, we will use a ballot that includes a
presidential contest, a contest for the Senate and House of
Representatives, a contest for Governor, several local contests, and a
number of referenda. All of the candidates, parties, and referenda text
have been included in the specification. The test machine vendors will
be responsible for creating the ballot layout based on this test ballot
specification. They will be asked to create an optimal layout for their
system including style, formatting, and instructions. Only one test
ballot will be allowed for each voting system.

The following section describes the ballot defined for pilot testing.
Additional work will be needed to ensure that the ballot used for VSTL
testing is realistic but provides the data needed for this test.

## Ballot Style 

There are multiple differences in the features and presentation styles
that exist on ballots throughout the country. Many states impose rules
or introduce features that are not found on other ballots. Some examples
of the ballot characteristics in this category include:

-   Icon identification (e.g., icons, when they are used to identify
    parties, are not consistent across the country.)

-   Number of columns in the ballot presentation

-   Font style and size

-   Party row versus party column versus office block designs

The VVSG has established requirements for some of these elements (e.g.,
font size). The vendor will be instructed to follow the VVSG guidelines
for all design elements covered in the VVSG and will be allowed to make
their own determination for the remainder of these characteristics.

## Write-ins

Write-ins are allowed in most contests. In some contests, it is unlikely
that a write-in campaign would be viable -- for example, in a
presidential election. However, successful write-in campaigns occur in
smaller districts and for local offices.

On the pilot test ballot:

-   Write-ins will be available for all contests except referenda and
    retention races

-   Testing of tasks that include write-ins will be restricted to local
    offices

## Nature of the Contest Data on the Ballot

Previous tests and vendor demos often use famous or humorous data on
their ballots (e.g., George Washington or Donald Duck for President).
Referenda are often obvious items such as ratifying the Constitution of
the United States or voting for a four day work week. The contest data
on the test ballot will need to be as realistic as possible with no
obvious inconsistencies from the real world. However, it is also
important that it not distract participants from their voting tasks with
inappropriate, unrealistic or humorous names or other material.

The pilot test ballot uses:

-   Realistic but not real names for candidates and no nicknames or
    middle initials

-   Fictitious and politically neutral party names (e.g., yellow,
    purple, gold, etc.)

-   Fictitious but realistic referenda that are non-partisan and
    non-controversial.

## Nature of the Referenda

We concluded that whether the referenda are summary or full text, and
the reading level of the referendum are not usability issues for the
voting machine itself.

The referenda on the pilot test ballot use:

-   Full text

-   A consistent reading level.

# VOTING TASK ANALYSIS

Summary: The test ballot must be constructed to incorporate a wide range
of voting tasks, such as voting in different types of races, casting a
write-in vote, etc. Those tasks must be chosen to ensure that different
functions of the voting system are used during the test. We started by
analyzing the voting tasks represented in typical ballots and voting
systems, and considering which were important for this usability test.

Usability is defined as the ability of a specific group of people to
perform a specific set of tasks in a specific environment. Therefore, it
is important to define the tasks that are involved if we are to develop
a usability test for the act of voting. One possibility is to define the
creating and casting of a ballot that contains the voter's intent on
each contest as a single complex task. In this case, failure to cast the
ballot as required for any reason is considered a task failure. However,
it is also possible to define the act of voting as a number of discrete
tasks such as voting by selecting from a list of names or voting by
write in. In this case, the ability of the participants to complete each
discrete task is analyzed.

For the pilot test, we have decided to develop a test based on the
discrete subtasks of voting. This will allow us to perform the analysis
at either level to help determine the most appropriate level of the VSTL
tests. Therefore, we started by analyzing the ballots to define discrete
tasks. As we analyzed the range of voting tasks, we considered the
criteria for including a specific task in the test or not. We wanted the
set of tasks to be realistic, but also to represent a broad enough set
of activities and enough variations on the tasks. This gives the test
content and face validity while providing the range of data needed to
properly analyze the test results. For example, it is reasonable to test
for the ability to vote in a multimember contest by selecting all names
from a list, by typing in a write-in name, or by a combination of both.
However, testing all combinations of these task variables is neither
necessary nor feasible.

In addition to these task variations, we wanted to ensure that usability
issues associated with the ballot layout were also included in the test.
For example, there might be differences in performance when voting for a
candidate whose name appears first in a candidate list than when it
appears further down the list -- particularly if the list continues onto
multiple columns, screens, or pages. Contests and tasks were defined to
address these types of potential presentation issues.

In the task analysis results, we defined two types of tasks: core tasks
and secondary tasks.

-   **Core tasks** are the primary tasks performed by voters in
    elections. In general, the core tasks can be mapped to a single
    contest on a ballot, though one ballot level task (casting the
    ballot) was considered a core task.

-   **Secondary tasks** include those that are used only in specific
    situations or that may or may not occur as a natural part of voting.
    These include those tasks used by a sub population of voters (e.g.,
    casting a ballot containing votes for only one contest), tasks that
    may occur in any ballot based on the user interaction (e.g.,
    changing a vote from one candidate to another), compound tasks
    (e.g., skipping a contest, then returning to this contest to enter a
    vote prior to casting the ballot), and specific system functions
    that need to be tested for potential usability problems (e.g.,
    detecting the presence of all contests on a ballot).

The following tasks have been defined as *core* tasks:

1.  Vote for a person running in a single-member contest with only one
    candidate.

2.  Cast a write-in vote for a person running in a winner-take-all
    contest with only one candidate.

3.  Vote for the first candidate in a winner-take-all contest with two
    candidates.

4.  Vote for the second candidate in a winner-take-all contest with two
    candidates.

5.  Vote for one candidate in a winner-take-all contest where the number
    of candidates is greater than two but the list is "short."

6.  Vote for the last person on a medium length list of names (e.g.,
    seven candidates).

7.  Vote for the candidate in a position near the end of a "long" list
    of candidates.

8.  Vote for and against a set of "short" referenda.

9.  Skip a "short" referendum.

10. Vote for (or against) a "very long" referendum.

11. Skip a contest.

12. Vote for a complete slate of candidates in a multi-member contest by
    name selection.

13. Vote for a complete set of candidates in a multi-member contest by a
    combination of name selection and write-in.

14. Vote for a partial slate of candidates in a multi-member contest by
    name selection.

15. Vote for a partial slate candidates in a multi-member contest by a
    combination of name selection and write-in.

16. Vote in a contest where multiple candidates appear on the ticket as
    a single vote (e.g., a presidential contest).

17. Vote to retain a candidate in a retention contest.

Many of the core tasks relate to the same type of contest (e.g., a
single member contest). The ballot specification includes a separate
contest for each of them to support task-based data collection. The core
tasks include variations but are not completely exhaustive. This lack of
exhaustive testing is necessary to prevent fatigue from participants, to
maintain face validity of the ballot and test, and since exhaustive
task-based testing is not required to determine high level usability
(e.g., only representative task-based testing is needed).

The following tasks have been defined as *secondary* tasks.

1.  Detecting and correcting over votes.

2.  Entering a vote and changing it before casting the ballot.

3.  Voting in a contest and then clearing the vote before casting the
    ballot.

4.  Initially skipping a contest but then entering a vote in this
    contest before casting the ballot.

5.  Returning to a contest in which the participant has already voted
    and changing a vote prior to casting the ballot.

6.  Return to a contest or referendum to a change the vote.

7.  Enter a write-in candidate name; then change to a listed candidate.

8.  Select a candidate; then change it to a write-in candidate.

9.  Return to a blank contest or referendum, and add the vote.

10. Return to a contest or referendum, and clear the vote.

In addition to these task level variations, secondary tasks at the
ballot level have been defined. These include:

11. Detecting the presence of all contests on a ballot

12. Vote a null ballot.

13. Vote for only one contest on a ballot.

14. Vote a straight ticket ballot (without exception).

15. Enter a partial straight ticket (e.g., straight ticket for only some
    contests on a ballot).

16. Vote a straight ticket...

    a.  When there are multiple candidates from a selected party in a
        multi-winner contest

    b.  When a straight ticket is desired with an exception for a
        specific contest (either skip a contest or vote for a different
        party)

    c.  When a straight ticket is desired with an exception that only a
        partial slate is to be voted in a multi-member contest

This list of tasks is a combination of realistic variation, potential
presentation issues, and realistically representative. We believe this
list is sufficient to represent the tasks performed by voters without
being overly repetitious and without including exception cases. We do
not consider this list of tasks final. Full validation of these tasks
will require a more thorough task analysis. However, a preliminary
content validity assessment of the tasks outlined in this section has
been made. Full validation might include performing a review of a more
complete set of ballots, a database review of voting results from real
elections, and interviews with subject matter experts. Content experts
will conduct a review of our conclusions. There is a direct relationship
between the number of tasks and the complexity of administering and
scoring the test. Once the task analysis is complete, we will reduce the
number of tasks to the minimal set required for testing usability.

To test all of the tasks defined above, it is anticipated that multiple
tests are required. For example, one test might be conducted using the
ballot where participants will be instructed who or what to vote for but
not instructed on any procedures required or strategies for using the
voting equipment (i.e., not how to vote). This might test all core
tasks. Additional tests would then be required to test secondary tasks.
Some of these additional tasks are at the contest level such as clearing
a vote prior to casting the ballot or changing votes from one name to
another. Other tasks are at the ballot level such as casting a straight
ticket ballot or casting a ballot after voting for only one office. In
these tests, participants would be told who or what to vote for and also
told to perform some specific tasks before casting the ballot (e.g.,
return to a contest and clear a vote before casting the ballot).

To minimize costs, tasks need to be combined where possible and some
ballot tasks may include multiple context level tasks. For example, a
single task to return to a contest and change a value from a selected
candidate to a write-in may be sufficient to test the voters' ability
both to return to a contest and to change a candidate selected by name
to a write-in, depending on the details of the acceptance criteria
(i.e., whether acceptance criteria are defined at the ballot or task
level). In addition, some tasks are very short (e.g., casting a null
ballot), so it is anticipated that some participants will be asked to
perform multiple ballot level tasks. This issue needs more research and
investigation.

We are currently investigating if each participant will be asked to
complete their tasks in the same order, especially on ballot-level
tasks. Typically when participants perform multiple tasks, the tasks
would be presented in varying order to account for order effects. For
example, if three ballot level tasks are provided to participants in one
of the tests, the ballot level tasks might be provided in six different
orders or randomized. On the other hand, it might be more feasible, if
it is determined to be acceptable, to conduct the test in a single order
since any learning effects would be present in both the machine test and
in the test used to determine baseline acceptance values. This issue
needs more research and investigation.

# CREATING A TEST BALLOT TO REPRESENT THE VOTING TASKS

Summary: A ballot specification has been defined for use in testing.
This ballot is sufficient to test all of the core and secondary level
tasks, though only the core level tasks will be included in the pilot
test. This ballot is specified as a logical specification, to allow for
the variation in physical layout in the different systems that might be
tested.

Based on the task analysis and list of tasks, we created a single ballot
definition that can be used for all of the tasks defined above including
both core and secondary tasks and including all contest level and ballot
level tasks. The pilot test will only include the core level tasks,
since this is sufficient to demonstrate the validity and reliability of
the test. Ultimately, all of the tests must be tested, but this effort
should be conducted after full validation of the tasks and a
determination if testing will be at the ballot level or the discrete
task level.

We have specified a logical and not physical definition since each
system has its own style of presentation. As a result, the specific
number of pages or screens that these logical specifications will
represent is unknown, and therefore it is not clear at this time whether
the ballots created from these specifications will allow participants to
complete all the desired tasks. In particular, tasks such as "vote for
someone on the second page or screen" may not be possible on all
systems.

The ballot specification also includes some non-partisan contests,
because they are required in order to complete the tasks such as voting
a straight ticket with modifications and to determine what, if any,
interaction results when a voter votes straight ticket and non-partisan
contests, See Appendix A of this report for the ballot specification.

# PARTICIPANT DEFINITION

Summary: The pilot test will be conducted with a narrowly-defined
population, using approximately 30 to 50 participants. For the pilot
test, we plan to use young adults (18 and 21 years old) who have not yet
voted in an election. Additional work will be needed in the development
phase to define the population to be used for VSTL testing.

For each test, we will need enough participants to provide a reasonable
percentage of errors (anticipated to be \<5%), enough participants to
provide reasonable reliability, but not so many participants that
conducting the test becomes logistically unreasonable. For face and
content validity purposes, we need citizens who are qualified to vote in
US elections. To meet these goals, we believe that the target goal of
the test is with a participant population of between 30 and 50
individuals.

For the pilot test, we will recruit 30 to 50 participants that:

-   are US citizens who are eligible to vote

-   are in the 18-21age range

-   have not yet voted in a election similar to the one being used for
    testing

We will eliminate those who might have a knowledge bias about the
intended test:

-   anyone who (or whose family) is involved in voting as a politician
    or voting official

-   anyone who (or whose family) works for a voting machine manufacturer

-   anyone with a background in political science or computer science

This audience was selected because it represents a large pool of
potential participants who can be easily recruited. We anticipate that
these young citizens will be reasonably similar to one another so that
their data is consistent enough for our use to demonstrate test validity
and reliability.

There are three other factors in recruiting test participants that are
still under investigation.

-   We will need to determine whether our target number of participants
    is a large enough pool to demonstrate the validity and reliability
    of this test. If the initial pilot test does not provide appropriate
    data, we may need to recruit higher numbers or use more stringent
    recruitment criteria

-   We might have to re-define the recruitment criteria if the error
    rates are not high enough to allow for proper data analysis. Low
    error rates in the test would require more participants. So to the
    extent possible, the population selected must also represent a
    population that exhibits a high magnitude of errors for the data
    being investigated. This may not be an issue in the pilot testing,
    but will need to be researched before the VTSL population is
    determined.

-   Since the VSTLs will be located in various geographical locations,
    we will need to ensure that the participants can be recruited in
    each of these locations and that the geographical differences do not
    result in different results. These differences could be a result of
    regional differences not present in the pilot test including
    differences in voting laws.

These three factors will need to be investigated before the final
definition of test participants can be made. These factors have an
impact on the cost of the test since they determine the numbers of
people who must be recruited, the difficulty in recruiting them, and the
reliability of the test itself.

We will also need to investigate whether individuals recruited in
different areas perform differently on the test. Such differences may
occurs based on geography, the different voting laws that they
represent, and even by type of areas where people live (e.g., rural,
suburban, or urban). This issue will be investigated during the
development phase.

# CONDUCTING THE TEST

**Summary: Participants in the test will all be asked to perform tasks
associated with voting. Each participant will work independently from
identical instructions. The test facilitator will be responsible for the
administration of the test but will not be allowed to interact with the
participant while they are attempting the complete the tasks.**

The usability test will be conducted by a test facilitator. The
facilitator will be responsible for:

-   Greeting the participants

-   Instructing the participants on how the test will be conducted

-   Collecting timing data during testing

-   Administering any post test questionnaires

In some usability tests, the facilitator may engage the participant in
conversation, asking them questions, or providing instructions
throughout the test session. In this test, however, the facilitators
cannot interact with the participant during the actual test. Any
interaction between the participant and the facilitator once testing has
started would invalidate the result from this participant. It is
acknowledged that this is not the same as in an election where poll
workers are available to assist voters if they have problems As a
result, it is likely that some usability problems encountered in the
test would be more significant than the same usability problem
encountered in an election. This is one reason why the results of the
test cannot be generalized to the product's use in real elections.
However, this limitation is necessary to ensure valid and reliable data
and does not affect the test outcome since comparison will be made
between the results from baseline data using the same restriction.

To ensure that each participant is given the exact same instructions,
written detailed instructions and a script will be provided for the
facilitators and the participants. This script helps ensure that the
facilitators complete their responsibilities while minimizing any chance
that they could influence the participants. An initial draft of this
script is included in Appendix B of this report.

The participants will be provided instructions on how to vote in each
contest. These instructions will include who or what to vote for but not
instructions on any procedures required or strategies for using the
voting equipment. Each participant will be asked to cast the same
specific set of votes on the ballots. This will ensure that there is a
set of expected values to compare to tabulated results. These
instructions are being developed and will be validated as part of this
project.

# ANALYSIS OF DATA

Summary: The data from this usability test will be analyzed to provide
scores for accuracy, efficiency, and satisfaction. Accuracy will be
measured as the number of errors made in each of the tasks. The system
must have a passing score for all of the tasks to pass certification.
Efficiency will be measured by timing each participant's session from a
pre-determined start and stop points. Satisfaction will be measured by
scoring the questionnaires filled out by the participants at the
beginning and end of the sessions.

The results of testing will be analyzed to make pass/fail determinations
of a machine against baseline data for all three dimensions of voting
system usability identified by the EAC -- accuracy (effectiveness),
efficiency, and satisfaction. The data required and the analysis
approach is described below. During the pilot phase, we will attempt to
demonstrate validity and reliability of the data collected for analysis.
During the development phase, the analysis data will be fully defined
and baseline values for each pass/fail criterion will be determined.
Procedures will be established in the development phase to ensure proper
counting, analysis, and reporting of this data.

## How will we measure voting accuracy?

Whether the analysis is conducted at the ballot level or at the contest
level, proper determination of error rates by participants requires
analysis of data by reviewing each ballot cast. However, since each
participant will be asked to cast the same vote, there is no need to
match the ballot data to the participant who attempted to cast it.
Maintenance of individual "ballot images" is a requirement of the VVSG
so this data is available for any machine going through certification
testing, though the procedures for obtaining this data vary by machine.
In addition, since the data is not summarized by the machine in the form
needed for the usability calculations, manual analysis of results are
required.

We have identified three different types of errors that affect voting
accuracy and that can be used to report on effectiveness:

1.  Inability to complete a task

2.  Ability to complete a task but with unintended missing results
    (e.g., under votes) and

3.  Ability to complete a task but with incorrect results (e.g., over
    votes or incorrect votes)

The test results will be used to indicate if a machine passes or fails
the certification based on the scores it receives in these three
categories for each task.

We have initially decided to perform the analysis for each task
separately. This provides the opportunity to establish different
acceptance levels for different tasks. It is possible that more complex
tasks may have a higher acceptable error rate than simpler tasks, or
more frequently performed tasks might have a lower acceptable error rate
than less frequently performed tasks.

In the analysis, we will determine the number of each type of error made
for each task to determine an effectiveness measure. We will report
effectiveness findings for each task. For acceptance, the system would
be expected to score better than the pass/fail criteria on each task.

## How will we handle data from participants who fail to cast their ballot?

We anticipate that some participants will not be able to complete the
evaluation. We also anticipate that some participants who complete the
ballot will fail to cast it. When analyzing data for effectiveness,
efficiency and satisfaction, these groups will need to be analyzed
separately. The one exception that may be considered is satisfaction
ratings, since this is not (necessarily) compromised by including all
types of groups and may even be more representative of the intended
measure. Failure to complete the evaluation will likely be a pass/fail
criterion in actual testing. In our pilot, these rates will be used as
part of our test of validity and reliability.

## How will we measure efficiency?

Our measure of efficiency data is based on the "time on task" data for
all participants who completed the entire test. During the test, the
facilitator will record start and end times for each participant. Since
we will not be able to monitor each task, these times will be for the
entire session, not for specific tasks within the test. We have not yet
determined the start and stop indicators, which will need to be
unambiguous event so that we can have accurate and repeatable timings.
Since monitoring the participant is presumed to be unlikely, these
events must be externally discernable. This area is particularly
difficult as it introduces a potential measurement precision error and
an increased probability of data loss. Some automated means of data
collection should be considered for VSTL testing. This issue is still
under investigation.

## How will we measure satisfaction?

We will attempt to will attempt to determine satisfaction levels from
data collected on questionnaires.

# CURRENT RESEARCH ISSUES

## Non-English Ballots

Though language selection will be tested, for the pilot research only
English ballots will be used. At this point, it is uncertain how
multi-language ballots will be included in the VSTL tests. This issue is
under investigation.

## Usability by Users with Disabilities

It is unclear at this time how best to test usability issues associated
with users who are blind, have partial visual disabilities or have
mobility/dexterity disabilities.[^1]. Many users with disabilities use
system functions that are not required by other users. From a Human
Factors Engineering perspective, any change in the system input or
output. constitutes a different system and would ideally need to be
tested separately. Based on this assumption, the following variations in
each device is anticipated

-   variations in visual display fonts, contrast, color, and possible
    dual visual and audio output for users with partial visual
    disabilities

-   audio output combined with tactile controls for users who are blind

-   tactile controls for users with mobility and dexterity disabilities

Though research is needed to evaluate this idea, we anticipate that a
separate test for each of these configurations will be required.

A corollary to this position is that some accessibility features,
usually available to all users such as large font and high contract
displays on DRE machines, represents a confounding variable for testing
on a machine. As such, intermittent use of such features by participants
might provide enough variation to preclude obtaining reliable results.
It may be desirable to disable or preclude the use of these features in
testing unless the test is designed to specifically require their use.
In other words, the use of large fonts and high contract displays may be
restricted to a test where everyone uses these features. This issue is
under investigation.

Appendix A: Logical Ballot Specification

# Test Ballot Logical Specification

**Overview**

NIST proposes to design summative usability tests for voting equipment,
as part of a conformance test suite for the HAVA/EAC voting standards.
An important part of this effort will be the design of a test ballot.
This ballot will become part of the measurement apparatus by which
voting equipment is evaluated. This page contains the logical
specification for that ballot.

-   The logical ballot specification is to be implemented by each
    vendor, using the style most appropriate for the specific equipment.
    The vendor designs layout, chooses font sizes and styles, use of
    color, etc.

-   Vendor designs instructions for voting (e.g. \"Vote for no more than
    three\", \"You must de-select before changing your vote.\") and any
    other equipment-specific information.

-   Vendor incorporates, without any changes, all the ballot content
    information in the specification, including:

    -   candidate names

    -   party names

    -   names of offices

    -   designation of districts

    -   wording of referenda and ballot questions

    -   order of contests

    -   order of candidates within a contest

Note: In this document, the format for partisan candidates is: \"name /
party\". Colors are used as party names.

Pilot Test Ballot Specification

Information applicable to whole ballot

  ------------------------------ ----------------------------------------
  Date and Time                  2004-nov-02, 7:00 AM to 8:00 PM

  State                          Maryland

  County                         Madison

  Party Line Voting Method       Enabled for partisan races
  ------------------------------ ----------------------------------------

Information applicable to every race

  -------------------------- --------------------------------------------
  Full-term or partial-term  Full-term
  election                   

  Voting Method              Simple vote for N candidate(s) - (i.e. no
                             ranked voting)
  -------------------------- --------------------------------------------

-   **Race #0:**

  -------------------------------------------- --------------------------
  Title of Contest                             Straight Party Vote

  Partisanship                                 Partisan

  Minimum Votes Allowed                        0

  Maximum Votes Allowed                        1

  Maximum Write-in Votes Allowed               0
  -------------------------------------------- --------------------------

-   **Option #1.1:** Blue

-   **Option #1.2:** Yellow

-   **Option #1.3:** Purple

-   **Option #1.4:** Orange

-   **Option #1.5:** Pink

-   **Option #1.6:** Gold

-   **Option #1.7:** Gray

-   **Option #1.8:** Aqua

-   **Option #1.9:** Brown

```{=html}
<!-- -->
```
-   **Race #1:**

  ---------------------------- ------------------------------------------
  Title of Office              President and Vice-President of the United
                               States

  District of Office           United States

  Partisanship                 Partisan

  Minimum Votes Allowed        0

  Maximum Votes Allowed        1

  Maximum Write-in Votes       0
  Allowed                      
  ---------------------------- ------------------------------------------

-   **Candidate #1.1:** Joseph Barchi and Joseph Hallaren / Blue

-   **Candidate #1.2:** Adam Cramer and Greg Vuocolo / Yellow

-   **Candidate #1.3:** Daniel Court and Amy Blumhardt / Purple

-   **Candidate #1.4:** Alvin Boone and James Lian / Orange

-   **Candidate #1.5:** Austin Hildebrand-MacDougall and James Garritty
    / Pink

-   **Candidate #1.6:** Martin Patterson and Clay Lariviere / Gold

-   **Candidate #1.7:** Elizabeth Harp and Antoine Jefferson / Gray

-   **Candidate #1.8:** Charles Layne and Andrew Kowalski / Aqua

-   **Candidate #1.9:** Marzena Pazgier and Welton Phelps / Brown

```{=html}
<!-- -->
```
-   **Race #2:**

  --------------------------------------------------- -------------------
  Title of Office                                     US Senate

  District of Office                                  Statewide

  Partisanship                                        Partisan

  Minimum Votes Allowed                               0

  Maximum Votes Allowed                               1

  Maximum Write-in Votes Allowed                      1
  --------------------------------------------------- -------------------

-   **Candidate #2.1:** Dennis Weiford / Blue

-   **Candidate #2.2:** Lloyd Garriss / Yellow

-   **Candidate #2.3:** Sylvia Wentworth-Farthington / Purple

-   **Candidate #2.4:** John Hewetson / Orange

-   **Candidate #2.5:** Victor Martinez / Pink

-   **Candidate #2.6:** Heather Portier / Gold

-   **Candidate #2.7:** David Platt / Gray

```{=html}
<!-- -->
```
-   **Race #3:**

  --------------------------------------- -------------------------------
  Title of Office                         US Representative

  District of Office                      6th Congressional District

  Partisanship                            Partisan

  Minimum Votes Allowed                   0

  Maximum Votes Allowed                   1

  Maximum Write-in Votes Allowed          1
  --------------------------------------- -------------------------------

-   **Candidate #3.1:** Brad Plunkard / Blue

-   **Candidate #3.2:** Bruce Reeder / Yellow

-   **Candidate #3.3:** Brad Schott / Purple

-   **Candidate #3.4:** Glen Tawney / Orange

-   **Candidate #3.5:** Carroll Forrest / Pink

```{=html}
<!-- -->
```
-   **Race #4:**

  ---------------------------------------------------- ------------------
  Title of Office                                      Governor

  District of Office                                   Statewide

  Partisanship                                         Partisan

  Minimum Votes Allowed                                0

  Maximum Votes Allowed                                1

  Maximum Write-in Votes Allowed                       1
  ---------------------------------------------------- ------------------

-   **Candidate #4.1:** Charlene Franz / Blue

-   **Candidate #4.2:** Gerard Harris / Yellow

-   **Candidate #4.3:** Linda Bargmann / Purple

-   **Candidate #4.4:** Barbara Adcock / Orange

-   **Candidate #4.5:** Carrie Steel-Loy / Pink

-   **Candidate #4.6:** Frederick Sharp / Gold

-   **Candidate #4.7:** Alex Wallace /Gray

-   **Candidate #4.8:** Barbara Williams / Aqua

-   **Candidate #4.9:** Althea Sharp / Brown

-   **Candidate #4.10:** Douglas Alpern / Independent

-   **Candidate #4.11:** Ann Windbeck / Independent

-   **Candidate #4.12:** Mike Greher / Independent

-   **Candidate #4.13:** Patricia Alexander / Independent

-   **Candidate #4.14:** Kenneth Mitchell / Independent

-   **Candidate #4.15:** Stan Lee / Independent

-   **Candidate #4.16:** Henry Ash / Independent

-   **Candidate #4.17:** Karen Kennedy / Independent

-   **Candidate #4.18:** Van Jackson / Independent

-   **Candidate #4.19:** Debbie Brown / Independent

-   **Candidate #4.20:** Joseph Teller / Independent

-   **Candidate #4.21:** Greg Ward / Independent

-   **Candidate #4.22:** Lou Murphy / Independent

-   **Candidate #4.23:** Jane Newman / Independent

-   **Candidate #4.24:** Jack Callanann / Independent

-   **Candidate #4.25:** Esther York / Independent

-   **Candidate #4.26:** Glen Chandler / Independent

-   **Candidate #4.27:** Marcia Colgate / Independent

-   **Candidate #4.28:** Leslie Porter / Independent

-   **Candidate #4.29:** Molly Dalton / Independent

-   **Candidate #4.30:** David Davis / Independent

-   **Candidate #4.31:** May Peterson / Independent

-   **Candidate #4.32:** Patricia Dawkins / Independent

-   **Candidate #4.33:** Suzanne Adams / Independent

-   **Candidate #4.34:** Mary Miller / Independent

-   **Candidate #4.35:** Rosalind Leigh / Independent

-   **Candidate #4.36:** Elaine Henry / Independent

-   **Candidate #4.37:** Gail Moses / Independent

-   **Candidate #4.38:** Daniel Jones / Independent

-   **Candidate #4.39:** Don Maybee / Independent

-   **Candidate #4.40:** Lillian Cohen / Independent

-   **Candidate #4.41:** Richard Mitchell / Independent

-   **Candidate #4.42:** Pat York / Independent

-   **Candidate #4.43:** Linda Rappaport / Independent

-   **Candidate #4.44:** Mike Porter / Independent

-   **Candidate #4.45:** Margaret Sharp / Independent

-   **Candidate #4.46:** Cathy Steele / Independent

-   **Candidate #4.47:** Lawrence Smith / Independent

-   **Candidate #4.48:** Bill Kendrick / Independent

-   **Candidate #4.49:** Fred Stein / Independent

-   **Candidate #4.50:** Jerry Cole / Independent

```{=html}
<!-- -->
```
-   **Race #5:**

  ------------------------------------------ ----------------------------
  Title of Office                            Lieutenant-Governor

  District of Office                         Statewide

  Partisanship                               Partisan

  Minimum Votes Allowed                      0

  Maximum Votes Allowed                      1

  Maximum Write-in Votes Allowed             1
  ------------------------------------------ ----------------------------

-   **Candidate #5.1:** Chris Norberg / Blue

-   **Candidate #5.2:** Anthony Parks / Yellow

-   **Candidate #5.3:** Luis Garcia / Purple

-   **Candidate #5.4:** Charles Qualey / Orange

-   **Candidate #5.5:** George Hovis / Pink

-   **Candidate #5.6:** Burt Zirkle / Gold

```{=html}
<!-- -->
```
-   **Race #6:**

  -------------------------------------------- --------------------------
  Title of Office                              Registrar of Deeds

  District of Office                           Countywide

  Partisanship                                 Partisan

  Minimum Votes Allowed                        0

  Maximum Votes Allowed                        1

  Maximum Write-in Votes Allowed               1
  -------------------------------------------- --------------------------

-   **Candidate #6.1:** Laila Shamsi / Yellow

```{=html}
<!-- -->
```
-   **Race #7:**

  ------------------------------------------------ ----------------------
  Title of Office                                  State Senator

  District of Office                               31st District

  Partisanship                                     Partisan

  Minimum Votes Allowed                            0

  Maximum Votes Allowed                            1

  Maximum Write-in Votes Allowed                   1
  ------------------------------------------------ ----------------------

-   **Candidate #7.1:** Edward Shiplett / Blue

-   **Candidate #7.2:** Marty Talarico / Yellow

```{=html}
<!-- -->
```
-   **Race #8:**

  ------------------------------------------- ---------------------------
  Title of Office                             State Assemblyman

  District of Office                          54th District

  Partisanship                                Partisan

  Minimum Votes Allowed                       0

  Maximum Votes Allowed                       1

  Maximum Write-in Votes Allowed              1
  ------------------------------------------- ---------------------------

-   **Candidate #8.1:** Andrea Solis / Blue

-   **Candidate #8.2:** Amos Keller / Yellow

```{=html}
<!-- -->
```
-   **Race #9:**

  ----------------------------------------- -----------------------------
  Title of Office                           County Commissioners

  District of Office                        Countywide

  Partisanship                              Partisan

  Minimum Votes Allowed                     0

  Maximum Votes Allowed                     5

  Maximum Write-in Votes Allowed            5
  ----------------------------------------- -----------------------------

-   **Candidate #9.1:** Camille Argent / Blue

-   **Candidate #9.2:** Chloe Witherspoon / Blue

-   **Candidate #9.3:** Clayton Bainbridge / Blue

-   **Candidate #9.4:** Amanda Marracini / Yellow

-   **Candidate #9.5:** Charlene Hennessey / Yellow

-   **Candidate #9.6:** Eric Savoy / Yellow

-   **Candidate #9.7:** Sheila Moskowitz / Purple

-   **Candidate #9.8:** Mary Tawa / Purple

-   **Candidate #9.9:** Damian Rangel / Purple

-   **Candidate #9.10:** Valarie Altman / Orange

-   **Candidate #9.11:** Helen Moore / Orange

-   **Candidate #9.12:** John White / Orange

-   **Candidate #9.13:** Joe Lee / Pink

-   **Candidate #9.14:** Joe Barry / Pink

-   **Candidate #9.15** Martin Schreiner / Gray

```{=html}
<!-- -->
```
-   **Race #10:**

  ---------------------------------------- ------------------------------
  Title of Office                          Court of Appeals Judge

  District of Office                       Statewide, 4th seat

  Partisanship                             Non-partisan

  Minimum Votes Allowed                    0

  Maximum Votes Allowed                    1

  Maximum Write-in Votes Allowed           1
  ---------------------------------------- ------------------------------

-   **Candidate #10.1:** Michael Marchesani

```{=html}
<!-- -->
```
-   **Race #11:**

  ----------------------------------------- -----------------------------
  Title of Office                           Water Commissioners

  District of Office                        City of Springfield

  Partisanship                              Partisan

  Minimum Votes Allowed                     0

  Maximum Votes Allowed                     2

  Maximum Write-in Votes Allowed            2
  ----------------------------------------- -----------------------------

-   **Candidate #11.1:** Orville White / Blue

-   **Candidate #11.2:** Gregory Seldon / Yellow

```{=html}
<!-- -->
```
-   **Race #12:**

  --------------------------------------------- -------------------------
  Title of Office                               City Council

  District of Office                            City of Springfield

  Partisanship                                  Partisan

  Minimum Votes Allowed                         0

  Maximum Votes Allowed                         4

  Maximum Write-in Votes Allowed                4
  --------------------------------------------- -------------------------

-   **Candidate #12.1:** Harvey Eagle / Blue

-   **Candidate #12.2:** Randall Rupp / Blue

-   **Candidate #12.3:** Carroll Shry / Blue

-   **Candidate #12.4:** Beverly Barker / Yellow

-   **Candidate #12.5:** Donald Davis / Yellow

-   **Candidate #12.6:** Hugh Smith / Yellow

-   **Candidate #12.7:** Reid Feister / Yellow

```{=html}
<!-- -->
```
-   **Retention Question #1:**

  ------------------ ----------------------------------------------------
  Wording of         Retain **Robert Demergue** as Chief Justice of the
  Question           Supreme Court?

  ------------------ ----------------------------------------------------

-   **Retention Question #2:**

  ------------------ ----------------------------------------------------
  Wording of         Retain **Elmer Hull** as Associate Justice of the
  Question           Supreme Court?

  ------------------ ----------------------------------------------------

-   **Referendum #1:**

+-----------+----------------------------------------------------------+
| Title of  | PROPOSED CONSTITUTIONAL AMENDMENT C                      |
| pr        |                                                          |
| oposition |                                                          |
+-----------+----------------------------------------------------------+
| Wording   | Shall there be amendments to the State constitution      |
| of        | intended to have the collective effect of ensuring the   |
| pr        | separation of governmental power among the three         |
| oposition | branches of state government: the legislative branch,    |
|           | the executive branch and the judicial branch?            |
|           |                                                          |
|           | a\. Article III, Section 6 of the Constitution shall be  |
|           | amended to read as follows:                              |
|           |                                                          |
|           | Section 6. Holding of offices under other governments. - |
|           | Senators and representatives not to hold other appointed |
|           | offices under state government. \--No person holding any |
|           | office under the government of the United States, or of  |
|           | any other state or country, shall act as a general       |
|           | officer or as a member of the general assembly, unless   |
|           | at the time of taking such engagement that person shall  |
|           | have resigned the office under such government; and if   |
|           | any general officer, senator, representative, or judge   |
|           | shall, after election and engagement, accept any         |
|           | appointment under any other government, the office under |
|           | this shall be immediately vacated; but this restriction  |
|           | shall not apply to any person appointed to take          |
|           | deposition or acknowledgement of deeds, or other legal   |
|           | instruments, by the authority of any other state or      |
|           | country.                                                 |
|           |                                                          |
|           | No senator or representative shall, during the time for  |
|           | which he or she was elected, be appointed to any state   |
|           | office, board, commission or other state or quasi-public |
|           | entity exercising executive power under the laws of this |
|           | state, and no person holding any executive office or     |
|           | serving as a member of any board, commission or other    |
|           | state or quasi-public entity exercising executive power  |
|           | under the laws of this state shall be a member of the    |
|           | senate or the house of representatives during his or her |
|           | continuance in such office.                              |
|           |                                                          |
|           | b\. Article V of the Constitution shall be amended to    |
|           | read as follows: The powers of the government shall be   |
|           | distributed into three (3) separate and distinct         |
|           | departments: the legislative, the executive and the      |
|           | judicial.                                                |
|           |                                                          |
|           | c\. Article VI, Section 10 of the Constitution shall be  |
|           | deleted in its entirety.                                 |
|           |                                                          |
|           | d\. Article IX, Section 5 of the Constitution shall be   |
|           | amended to read as follows:                              |
|           |                                                          |
|           | Section 5. Powers of appointment.- The governor shall,   |
|           | by and with the advice and consent of the senate,        |
|           | appoint all officers of the state whose appointment is   |
|           | not herein otherwise provided for and all members of any |
|           | board, commission or other state or quasi-public entity  |
|           | which exercises executive power under the laws of this   |
|           | state; but the general assembly may by law vest the      |
|           | appointment of such inferior officers, as they deem      |
|           | proper, in the governor, or within their respective      |
|           | departments in the other general officers, the judiciary |
|           | or in the heads of departments.                          |
+-----------+----------------------------------------------------------+

-   **Referendum #2:**

  ------------- ----------------------------------------------------------
  Title of      PROPOSED CONSTITUTIONAL AMENDMENT D
  proposition   

  Wording of    Shall there be an amendment to the State constitution
  proposition   concerning recovery of damages relating to construction of
                real property improvements, and, in connection therewith,
                prohibiting laws that limit or impair a property owner\'s
                right to recover damages caused by a failure to construct
                an improvement in a good and workmanlike manner; defining
                \"good and workmanlike manner\" to include construction
                that is suitable for its intended purposes; and permitting
                exceptions for laws that limit punitive damages, afford
                governmental immunity, or impose time limits of specified
                minimum lengths on filing lawsuits?
  ------------- ----------------------------------------------------------

-   **Referendum #3:**

+-----------+----------------------------------------------------------+
| Title of  | PROPOSED CONSTITUTIONAL AMENDMENT H                      |
| pr        |                                                          |
| oposition |                                                          |
+-----------+----------------------------------------------------------+
| Wording   | Shall there be an amendment to the State constitution    |
| of        | allowing the State legislature to enact laws limiting    |
| pr        | the amount of damages for noneconomic loss that could be |
| oposition | awarded for injury or death caused by a health care      |
|           | provider? \"Noneconomic loss\" generally includes, but   |
|           | is not limited to, losses such as pain and suffering,    |
|           | inconvenience, mental anguish, loss of capacity for      |
|           | enjoyment of life, loss of consortium, and other losses  |
|           | the claimant is entitled to recover as damages under     |
|           | general law.                                             |
|           |                                                          |
|           | This amendment will not in any way affect the recovery   |
|           | of damages for ecomonic loss under State law. \"Economic |
|           | loss\" generally includes, but is not limited to,        |
|           | monetary losses such as past and future medical          |
|           | expenses, loss of past and future earnings, loss of use  |
|           | of property, costs of repair or replacement, the         |
|           | economic value of domestic services, loss of employment  |
|           | or business opportunities. This amendment will not in    |
|           | any way affect the recovery of any additional damages    |
|           | known under State law as exemplary or punitive damages,  |
|           | which are damages allowed by law to punish a defendant   |
|           | and to deter persons from engaging in similar conduct in |
|           | the future.                                              |
+-----------+----------------------------------------------------------+

-   **Referendum #4:**

+-----------+----------------------------------------------------------+
| Title of  | PROPOSED CONSTITUTIONAL AMENDMENT K                      |
| pr        |                                                          |
| oposition |                                                          |
+-----------+----------------------------------------------------------+
| Wording   | Shall there be an amendment to the State constitution    |
| of        | authorizing Madison and Fromwit Counties to hold         |
| pr        | referenda on whether to authorize slot machines in       |
| oposition | existing, licensed parimutuel facilities (thoroughbred   |
|           | and harness racing, greyhound racing, and jai alai) that |
|           | have conducted live racing or games in that county       |
|           | during each of the last two calendar years before        |
|           | effective date of this amendment? The Legislature may    |
|           | tax slot machine revenues, and any such taxes must       |
|           | supplement public education funding statewide. Requires  |
|           | implementing legislation.                                |
|           |                                                          |
|           | This amendment alone has no fiscal impact on government. |
|           | If slot machines are authorized in Madison or Fromwit    |
|           | counties, governmental costs associated with additional  |
|           | gambling will increase by an unknown amount and local    |
|           | sales tax-related revenues will be reduced by \$5        |
|           | milliion to \$8 million annually. If the Legislature     |
|           | also chooses to tax slot machine revenues, state tax     |
|           | revenues from Madison and Fromwit counties combined      |
|           | would range from \$200 million to \$500 million          |
|           | annually.                                                |
+-----------+----------------------------------------------------------+

-   **Referendum #5**

  ------------- ----------------------------------------------------------
  Title of      BALLOT MEASURE 101: Open Primaries
  proposition   

  Wording of    Requires primary elections where voters may vote for any
  proposition   state or federal candidate regardless of party
                registration of voter or candidate. The two
                primary-election candidates receiving most votes for an
                office, whether they are candidates with no party or
                members of same or different party, would be listed on
                general election ballot. Exempts presidential nominations.
                Fiscal Impact: No significant net fiscal effect on state
                and local governments.
  ------------- ----------------------------------------------------------

-   **Referendum #6:**

  ------------- ----------------------------------------------------------
  Title of      BALLOT MEASURE 106: Limits on Private Enforcement of
  proposition   Unfair Business Competition Laws

  Wording of    Allows individual or class action \"unfair business\"
  proposition   lawsuits only if actual loss suffered; only government
                officials may enforce these laws on public\'s behalf.
                Fiscal Impact: Unknown state fiscal impact depending on
                whether the measure increases or decreases court workload
                and the extent to which diverted funds are replaced.
                Unknown potential costs to local governments, depending on
                the extent to which diverted funds are replaced.
  ------------- ----------------------------------------------------------

End of logical specification for Pilot Test Ballot Specification.

Appendix B: Facilitator's Guide

## Pre Evaluation Administration

In the pre evaluation administration phase, the facilitator will be
responsible for:

-   Administering the consent and release form

-   Presenting the test introduction to the participant and ensuring
    that he/she understands the procedures. The following introductory
    material will be given to the participant.

## Test Introduction

The test facilitator will present the following information to
participants:

> You are here to be part of a test of a voting machine. We are
> conducting this test to see if typical voters can accomplish specific
> kinds of voting tasks --- ones that voters might attempt in a real
> election. This is not evaluation of you. We are evaluating the
> equipment. Any problems you encounter represent a design issue in
> these machines.
>
> We have made up a ballot for use in this evaluation. The names on the
> ballot are not real. And all political party names are referred to by
> colors. These colors have no relationship to the political parties you
> may know. They were also made up for this test.
>
> We will be giving you specific written instructions for what we want
> you to do. It is important you follow our instructions so that we have
> can properly test the machines. So, if the directions say to vote for
> a specific person, please try to vote for that person even if you
> prefer another name or party. If you can not do it exactly as the
> instructions tell you, do the best you can. If you can not follow the
> instructions exactly, try to remember what happened so we can talk to
> you about it after the evaluation. When you have completed the
> evaluation, return the instructions to me. We will then discuss this
> evaluation of the voting equipment and I'll get your thoughts about
> it. Once the evaluation begins, we will not be able to provide you
> with any help.
>
> Do you have any questions?

After the facilitator ensures that the participant understands what is
expected of them, the participant instructions will be given to them and
they will be shown to the machine.

## Voting

> Depending on the machine, the facilitator may be required to
> [start]{.underline} the voting process by acting the role of the poll
> worker.
>
> The facilitator will be responsible for timing. Event timing will
> start once the participant reaches the machine. If the facilitator is
> required to initiate the voting process as a poll worker, timing will
> begin when the ballot appears on the screen. If the participant starts
> the voting process, timing will begin when the participant does the
> first step (i.e., enters the card, receives the paper ballot, etc.).
> Once the participant has started the event of voting, no further
> interaction is allowed. The facilitator will not be permitted to
> answer any questions about the voting machine, the ballot, or the
> voting instructions. If the facilitator is asked for assistance, this
> should be the reply:
>
> "I'm sorry. I can't provide you with any help. Please do the best you
> can. If you are stuck and cannot continue, you can stop."
>
> Once the participant has completed the voting task by walking away
> from the machine, the facilitator will record the event time---the
> time on task will be from the start point as described above to the
> time the participant walks away from the machine.

## Post Evaluation Interview

In post test questioning phase, the facilitator will administer the
modified SUS and modified Cooper-Harper surveys. Once the participant
completes the surveys, the facilitator will interview the participants
to determine:

-   If the participant followed the voting instructions

-   If there were situations where the participant could not follow the
    instructions and what was done in those situations.

## Compensation

In the compensation phase, the facilitator will provide the participant
with his/her compensation and thank the participant for his/her time.

[^1]: This is separate from the issue of accessibility, which is being
    addressed under a separate effort.