# **Next VVSG Training Part 1, Chapter 3: Usability, Accessibility, and Privacy**

## October 15-17, 2007

## 

## Dr. Sharon Laskowski

## National Institute of Standards and Technology

## sharon.laskowski@nist.gov

## 

## 

## 

## 

# This presentation includes

## Key concepts for HFP requirements

## Overview of Part 1, Chapter 3

### And relevant requirements in Part 1, 4.2.4, 4.4.2

## Explanations of the more complex requirements

### Design requirements

### Usability performance requirements

## Definitions as needed

# Acronyms in this presentation

## Acc-VS: Accessible Voting Station

## DRE: Direct Recording Electronic

## EBM: Electronically-assisted Ballot Marker

## MMPB: Manually-Marked Paper Ballot

## VEBD: Voter-Editable Ballot Device

### VEBD-A: Audio VEBD

### VEBD-V: Video VEBD

## PCOS: Precinct-Count Optical Scanner

## ATI: Audio-Tactile Interface

## CVR: Cast Vote Record

## IVVR: Independent Voter-Verifiable Record

### (VVPR: Voter-Verifiable Paper Record)

## VVPAT: Voter-Verifiable Paper Audit Trail

## CIF: Common Industry Format for Usability Test Reports

## VPP: Voting Performance Protocol

## 

# Key Concept: Goal

## Goal of these requirements is to provide a voting system that voters can use comfortably, efficiently, and with justified confidence that they have cast their votes correctly.

## Focus is primarily on the voter's interaction with the voting system, but there are requirements for poll workers. 

# Key Concept: Universal Usability

## View the voting station as a public kiosk, designed for "everyone", not special purpose

## Many people have some special needs, but do not identify themselves as having disabilities

#### Aging population, those with language or reading issues

## Move as much accessibility into general voting station as possible

## **ALL usability requirements apply to the ACC-VS**

## 

# Key Concept: Accessibility

## Goal is to make the voting system independently accessible to as many voters as possible

## Harmonized with other accessibility standards, collaborated with the US Access Board

## Section 3.3 is organized according to the type of disability being addressed 

### NOTE: features intended primarily to address one kind of disability may assist voters with other kinds

## 

# Key Concept: Accessible Voter Verification

## Software independence (SI) for security and its implications for accessibility has been a thorny issue for the TGDC

### SI in current systems can only be addressed using paper IVVR

### But, paper by itself is not accessible: some voters cannot verify directly

### So, there are a number of requirements that address how to make the IVVR accessible

### This includes using "observational testing" to verify the reliability of indirect means of verification 

### 

# Key Concept: Design vs. Performance

## Design requirements

### Specify the "look and feel" of the voter interface, for general classes of voting systems

### Based on best practice from other similar domains

### Typically tested by inspection or expert review

## Performance requirements

### Based on usability testing best practices

### Specifies a benchmark which must be met when voters *interact* with the system

### Tested in a tightly controlled environment with human test participants

### We will explain the performance metrics and benchmarks in 3.2.1 in the final part of this presentation. 

#### 

# 3.2 General Usability Requirements

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# 3.3 Accessibility Requirements

# 

# 

# 

# 

# What about VVPAT (Voter-Verifiable Paper Audit Trail)?

## 

## [VVPAT]{.underline} is one implementation of paper IVVR, also called voter-verifiable paper records (VVPR)

## 

## Section 4.4.2 contains requirements specific to VVPAT.

## 

## All relevant usability and accessibility requirements apply to VVPAT.

## 

# 

# 

# 

# 

# 

# 

# 

# 

# 

# Usability Performance Requirements

## Goal: To develop a test method to distinguish systems with poor usability from those with good usability

### Based on performance not evaluation of the design

### Reliably detects and counts errors one might see when voters **interact** with a voting system

### Reproducible by test laboratories

### Technology-independent

### 

# Calculating benchmarks

## Given such a test method, benchmarks can be calculated: a system meeting the benchmarks has good usability and passes the test

### The values chosen for the benchmarks become the performance requirements

## 

# Usability testing for certification in a lab

## We are measuring the **performance of the** **system** in a **lab**

### We control for other variables, including the test participants

### We measure the effect of the system on usability

## The test ballot is designed to detect different types of usability errors and be typical of many types of ballots

## The test environment is tightly controlled, e.g., for lighting, setup, instructions, no assistance

## The test participants are chosen to reliably detect the same performance on the same system

# Usability testing for certification in a lab

## Test participants are told exactly how to vote, so errors can be measured

## The test results measure relative degree of usability between systems and are NOT intended to predict performance in a specific election

### Ballot is different

### Environment is different (e.g, help is provided)

### Voter demographics are different

## A general sample of the US voting population is never truly representative because all elections are "local". 

## 

## 

### 

### 

## 

### 

## 

# Components of the test method (Voting Performance Protocol)

## Well-defined test protocol that describes the number and characteristics of the "voters" participating in the test and how to conduct test,

## Test ballot that is relatively complex to ensure the entire voting system is evaluated and significant errors detected,

## Instructions to the "voters" on exactly how to vote so that errors can be accurately counted,

## Description of the test environment,

## Method of analyzing and reporting the results, and

## Performance benchmarks with associated threshold values.

# Performance Benchmarks: Recap of Research

## 

## Validity: tested on 2 different systems with 47 participants

### Test protocol detected differences between systems, produces errors that were expected.

## Repeatability/Reliability: 4 tests on same system, 195 participants, similar results

# Performance Benchmarks: Recap of Research

## Demographics

### Eligible to vote in the US

### Gender: 60% female , 40% male 

### Race: 20 % African American, 70% Caucasian, 10% Hispanic

### Education: 20 % some college, 50% college graduate, 30% post graduate

### Age: 30% 25-34 yrs., 35% 35-44 yrs., 35 % 45-54 yrs. 

### Geographic Distribution: 80% VA, 10% MD, 10% DC

## 

# Benchmark Tests

## 4 systems, May 19-20, June 1-2

### Selection of DREs, EBMs, PCOS

## 187 test participants

## 5 measurements

### 3 benchmark thresholds

### 2 values to be reported only

## 

# The Performance Measures Base Accuracy Score

## We first count the number of errors test participants made on the test ballot -- there are 28 voting opportunities: count how many were correct for each participant 

## We then calculate a **Base Accuracy Score:** the mean percentage of all ballot choices that are correctly cast by the test participants 

# We calculate 3 effectiveness measures: Total Completion Score

## 

## The percentage of test participants who were able to complete the process of voting and have their ballot choices recorded by the system. 

### 

### 

# Voter Inclusion Index (VII)^\*^

## A measure of overall voting accuracy that uses the Base Accuracy Score and the standard deviation.

### If 2 systems have the same Base Accuracy Score (BAS), the system with the larger variability gets a lower VII.

### The formula, where S is the standard deviation and LSL is a lower specification limit to spread out the measurement (we used .85), is: 

# Perfect Ballot Index (PBI)^\*^

## The ratio of the number of cast ballots containing no erroneous votes to the number of cast ballots containing at least one error. 

### This measure deliberately magnifies the effect of even a single error. It identifies those systems that may have a high Base Accuracy Score, but still have at least one error made by many participants. 

### This might be caused by a single voting system design problem, causing a similar error by the participants. The higher the value of the index, the better the performance of the system. 

## **\***range is 0 to infinity, if no errors at all. 

# Efficiency and Confidence Measures

## **Average Voting Session Time** -- mean time taken for test participants to complete the process of activating, filling out, and casting the ballot.

## **Average Voter Confidence** -- mean confidence level expressed by the voters that they believed they voted correctly and the system successfully recorded their votes.

## Neither of these measures were correlated with effectiveness.

## Most people were confident in the system and their ability to use the system. 

# Benchmark test results

# Benchmark test results

# Benchmark thresholds

## Voting systems, when tested by laboratories designated by the EAC using the methodology specified in this paper, must meet or exceed ALL these benchmarks:

### 

### Total Completion Score of 98%

### Voter Inclusion Index of .35 

### Perfect Ballot Index of 2.33

### 

## Systems C and D fail.

## Report time and confidence

# 

# 

# How "tough" should the benchmark thresholds be?

## The benchmark data here used 50 test participants, but the test protocol will call for 100 (to allow statistical assumption of normal distribution to calculate the VII confidence intervals)

### 100 participants will narrow the confidence intervals and thereby toughen the test.

## **Two points of view:**

### Proposed benchmarks do weed out poorly performing systems (and, it is relatively easy to raise thresholds)

### vs.

### This should be a forward-looking standard, new systems should be held to a higher standard 

#### (but what is the upper bound, given that humans always make some mistakes?)

### 

## 

# Additional Research

## Reproducibility: How much flexibility can be allowed in the test protocol?

### Will variability in test participants experience due to labs in different geographic regions affect results?

### Should we factor in older population or less educated population? 

### Benchmark thresholds are always tied to the demographics of the test participants to some extent 

## Accessible voting system performance?

## 

# Any Final Questions?