MR. KILLAM: My name is Bill Killam. I am the president of User Centered
Design, Incorporated. I am adjunct professor at the University of
Maryland and George Mason University in Human Factors Engineering. My
specific background with voting machines and probably the reason why you
asked me to speak today was in the actual hands-on experience with some
of the equipment out there currently being used or being proposed.

I was able to work with 16 machines last year and met with 16 of the
vendors at the IAECREOT convention. I met with everybody from their
presidents and their lead engineers down to their sales staff. I also
saw six machines and worked with them at the National Federation of the
Blind. They allowed me to work with their machines and to talk about the
testing work they had done and the findings they had. Finally, I
participated in the expert review of 16 machines at the University of
Maryland in their efforts with the National Science Foundation. I have
reviewed countless articles and research papers and helped write the
NIST Human Factors Report on Voting Systems. So, my comments are based
primarily on that research, if I can call it research. (I want to
clarify that statement in a moment.) But also on my background: I am a
Human Factors Engineer, board certified.

Before I start, I want to clarify the terms I will use. There is not
universal acceptance of these definitions. But the question posed refers
to "human factors, usability and accessibility". To my mind, and within
my company, we have always viewed accessibility as having two
components. The first being access. We see access as the precursor
before usability is considered. If you can\'t open the door, it doesn\'t
matter what\'s behind the door. So, we view access as a critical factor
but different from what we term "usability by people with disabilities".
Of course, this also suggests a separation between people with
disabilities and everyone else. We are aptly referred to on occasion as
the people who are temporarily able bodied. It\'s a continuum. When I
refer to accessibility in my comments, I am referring to access to the
device and usability across the board. Human Factors Engineering is the
study of all of that. So, I\'ll probably just say usability and
accessibility in my comments.

The first question posed was: "What are the most important issues in
usability, accessibility, and human factors." I'm concerned my comments
probably will sound like I\'m part of a different panel, but I will try
to stick with the subject at hand. I have four main points related to
that question. The first one is something I think many people here would
acknowledge, but I think it\'s worth repeating - that usability and
accessibility are not absolutes. They are on a continuum, as well. There
is no such thing as a perfectly usable device or perfectly accessibility
device. You can only tell that at a distance. We know we will always be
excluding some people. Sometimes the same person on a different day.
Knowing that, we have to accept that there will be some margin we will
never be able to accommodate. And dealing with individual cases can only
confuse us.

The second thing is we are talking about a system made up of multiple
components. In the case of voting, the system not only includes the
machines and the ballot, it includes the user, and everybody helping -
the poll worker, et cetera. As Kim Brace \[the first panel speaker\]
pointed out, by having data, collecting it, and being able to see where
something has changed, we can see the effect. The system will always be
affected by any change within it. A different set of voters, a different
ballot, a different machine, different poll workers, a different
environment will all effect the system. The issue is: how does it affect
it and is it still within range of the effects we would like to see. The
goal is, any time we make a change on purpose such as a new ballot
design or a new machine, the goal is we improve the usability and
accessibility. In the case of accessibility (and pure access in that
case), there is actually a mandate that that occur. The problem, and
biggest issue in my mind, is our inability to tell if we\'ve done that.
The current problem we have, although we have lots and lots of antidotal
data and data after the fact - wonderful postmortem analysis, but we
have no scientific, reliable, valid data to tell us where we are and if
we are changing anything. If we are introducing a new machine or ballot,
are we improving things or making it worse? Without the data, we have no
way to tell us what we have right now. We don\'t know what problems
currently exist but I think we all know that they do. We also don\'t
have any means of measuring. There\'s no pure definition of usability or
accessibility. There are some key concepts, there are some key
measurements you might make, but we have no definition that is universal
across the board or specific to voting.

The third one is we don\'t have any benchmarks. We would like to believe
whatever we introduce is making things better. But since we don\'t know
where we are, we can\'t tell where we\'re going or if we got there.
That\'s why I think that my answer to this first question leads more
into a discussion for this afternoon \[on usability testing\]. But that
to me is the primary goal. There are a lot of techniques associated with
solving that problem. The ability, no matter what we do, to be able to
measure these things I think is the critical issue.

The second question posed had to do with are communications - design of
the ballots, instruction material, polling place, et cetera, - critical.
I think the evidence is out there to suggest we have a problem in this
area. It\'s not a new one. It is a problem that has existed for years.
There are wonderful papers written just looking at the ballot
instructions. Probably what I consider a classic case was where the
instructions itself probably caused over-voting to the extent that there
were tons of spoiled ballots. It was not at a national level but at a
lower level. These are critical to the successful design and use of the
system. The problem we have to be careful is to think that it can
compensate. Though a poor set of instructions, inaccurate instructions,
invalid set of instructions, or bad ballot designs can ruin the voting
system, it can\'t make bad equipment better. I always look back to the
equipment. Maybe that is my engineering background.

If we go farther, the third question posed is one having to do with
training and documentation. Training is a last resort. And again,
training suffers even more problems in that it\'s transient. It has its
own usability issues of ease of learning and ease of recall. Any time
you train a set of people, immediately they start to forget. They
confabulate their answers. That is not unexpected. We believe a lot
sticks. To believe that can have a significant effect through \[on
usability or accessibility\] training is problematic. Though I think
it\'s again very important. Especially in the case of voting system this
becomes even more critical because we\'re talking about a population
that is not highly trained, they don\'t use this equipment on a regular
basis, they are often volunteers, they are often elderly, they\'re being
introduced to new technology. The training is critical. But again it
can\'t compensate for problems in the design.

I suspect that might be leaving plenty of time for Ginny who was looking
for three extra minutes. So, I\'ll get to my conclusion right away.

(Laughter)

I looked at these questions in order and I thought the order of
presentation of the questions was interesting, if not significant. I
think the key is in the design of the equipment. We could mandate the
user center design process to the vendors. (Not my company, the
concept.) I don\'t think it is possible or necessary. There are people
out there who are excellent designers and never would have to bring the
user in to test anything. We do, however, have to find a way to make
sure that we are getting back what we are asking for. That means knowing
what the criterion is for acceptance and having a way to measure whether
the vendors make it or not. I think then we step farther from the
equipment and start looking at signage, documentation, and ballot
design. Ballot design particularly cannot be separated. But to believe
that the instructions or great ballots on top of a machine with
problems... We can\'t expect any better than the machine can perform.
And finally, again, the third question being farther back from that -
training.

There is an expression we use in the industry which I debated over
whether to use it because I don\'t intend to be flippant, but I think
it\'s colorful enough and memorable enough. We refer on occasion of
putting lipstick on a bulldog. Good signage. Good documentation and
ballot design on top of a machine is something like putting lipstick on
a bulldog. We have to make sure it \[the equipment\] is not a bulldog. I
think that is the key to me in the answer.

There is plenty of data, compelling evidence that problems exists, and a
lot of sources for finding this \[data\]. I think the ability to do pure
scientific studies to really analyze and develop procedures for getting
this data, establishing the baseline, and testing against it will be key
to getting us where we need to be.

CHAIRPERSON QUESENBERY: Thank you. Does anyone on the panel have
questions.

MS. HILLMAN: I wanted to explore the use of the term "human factors".
What was the first application of the use of the term? I am going back
as far as I can think in terms of things being designed and how people
decided the knob should always be on the right versus the left.

MR. KILLAM: The term "human factors engineering"? It depends on who you
ask. I have seen references in Bob Bailey's book that can take you back
to the Phoenicians. I don\'t think we have to go back that far.

Probably the first true human factors engineers were a group of
psychologists that went to Wright Patterson Air Force Base after World
War II and worked on issues, particularly flight issues. That is the
origin of the term "applied psychologists". They later started the
Ergonomics Society which later became the Human Factors Society which
eventually used the European term "ergonomics". They are synonymous. In
the United States human factors and ergonomics are generally associated
with hardware and software. They merged and became The Human Factors and
Ergonomics Society.

So, most people would say it is an industry about 40 or 45 years old and
the term comes from psychologists getting into engineering. That\'s why
human factors itself is its shortened version - it\'s engineering
associated with the factors relating to humans.

MS. HILLMAN: Is there any group that would disclaim or not want to use
the term "human factors" when talking about voting systems or equipment?
Would they want to describe it another way?

MR. KILLAM: I don\'t think so. I think human factors has become more
universally talked about. There is a group of people who have degrees in
it and it\'s a professional affiliation. I think human factors has
become a common expression for this area. I think human factors captures
the notion looking at components related to humans tha are often not
considered.

The term system... Another colleague of mine had a discussion about the
use of the term "human system interface" and said that\'s not correct
because humans are part of the system. I think by introducing the term
and saying human factors, we recognize it in the system along with the
engineering factors, and software factors, and the society factors, and
the instruction factors - I think it makes sense. I would encourage its
use. I\'m not sure if you're thinking of an audience that would not like
to use it.

MS. HILLMAN: No. But if you were speaking to a group of community
activists and talking about this, how would you explain human factors in
a few simple words so they could walk out and figure out what their
contribution to the process would be.

MR. KILLAM: That is a good question. I have a few stories I use to try
to explain what human factors engineering is. And there is certainly a
definition, but I am not sure if it answers your question. The
definition of human factor engineering would be an application of the
knowledge of human capabilities and limitations so we know now to design
products better. I might use that.

The other part I might use, and my favorite story, is to talk about the
fact that it\'s not as obvious as it may seem. Its the case of a person
stuck in a room trying to communicate and it\'s too noisy to talk. The
answer \-- an answer to the problem from a human factors perspective is
to put your fingers in your ear. Sound waves bounce against the bones of
your elbow.... Actually, put your finger in your ear and point your
elbow so the sound will travel up your elbow into your ear, and you will
hear better. It\'s not an obvious thing. It's the way the human body
works that probably wouldn\'t be considered.

So, solutions to problems are not always what they seem to be, and the
problem that.... Not human factors being a problem but human
engineering. We can\'t engineer humans. We can engineer systems based on
factors related to humans. I am quick to point out it\'s human factors
engineering. We are not geneticists. I don\'t know if that answers your
question.

MS. HILLMAN: I think that works for me.

CHAIRPERSON QUESENBERY: I do have a question that will put you on the
spot a little bit. As you looked at this these 16 machines, you are one
of a few people who looked at a lot of machines and were able to see
them up close and touch them. What\'s your assessment of the overall
state of human factors for disabilities assistance that are currently
out there in three words or less?

MR. KILLAM: As politically correct as I can put it, there is plenty of
evidence to suggest that we are going to be making things worse \-- that
these designs probably have problems.

I will be quick to point out all of the data I have seen - the data and
the information that I am giving you is antidotal. It's an opinion. I
have a pen that I take out in every class as a demonstration, and it was
the greatest demonstration of how difficult something can be to figure
out, but I learned it immediately. The people that work at this stuff
\[human factors engineers\] are experts. We can look at things and say
this is a serious problem and it should be fixed. But, as we pointed out
in the human factors report, we have to be careful. I may be answering
your question longer than you intended maybe, but we have to be careful
when we use expert opinion to drive the needs to fix the problem. There
are a lot of usability problems that we classify in my company as
category one problems \-- usability problems prior to success. There is
a lot of ways to make things better, but if the user sets out and gets
to the end point, from some respects it is not a problem. We can always
make it better, but we can continue that forever. We have to be
concerned with type two and three problems.

Type two problems means the user thinks they did and they didn\'t or the
user partially did it. The roll-off effect based on the fact that the
machine is difficult to use and the person says "I am not going to do
anymore, I'm un comfortable with it." That is the issue we have to
detect.

The third one is complete failure. That is category three. Case in
point. Human factors issue. If the report is true (and we know from
recent evidence we don\'t want to report anything unless we verify our
sources), but I did read an article suggesting that in New York one
precinct had disabled the interlock system for the curtain on an old
lever voting machine. Until you pull that level, the curtain should not
open. They confessed. It happened some years ago. No one recalls why,
but they did disable it. Evidence suggests that there were 60,000...
...I wouldn\'t want to be quoted..., but a large percentage of people
walked away from the machine without voting. Was that an intention or
because the interlock was disabled? The user thought they were finished.
If we believe they walked away unintentionally, they thought they were
finished and walked away. That was somebody who did something not
realizing the effect on the human being without having an interlock in
place. We don\'t know. Until we can find out.... I think we have seen
evidence to suggest this could be a serious problem. How serious we
wouldn\'t know until we measure.

CHAIRPERSON QUESENBERY: Thank you. Jim, do you have any questions?

MR. ELEKES: I don\'t want to take a lot of time. What are the most
significant factors in your opinion relative to usability \-- and I\'ll
keep bringing this up all day unfortunately \-- by those who either are
sensory or physically disabled you have found in reviewing the 16
machines you have been able to observe?

MR. KILLAM: Unfortunately in most cases of the system I reviewed, we
have not looked at accessibility features so I am unequipped to answer
the question fully. I think the problems in that area are much more in
the intuitive side \-- cognitive side of the issue. I suspect the other
barriers to access are in some respects a technical problem. The
usability problems are much different, and I have not seen the solution
demonstrated in a way that I then could detect the usability problems. I
think the access problems still stand in the way.

MR. ELEKES: Cognitive. We\'re talking perhaps of access by a person who
is visually impaired the controls would have to be simple enough and
intuitive enough that the person would recognize as it were commonplace
similar to other technologies they use in activities of daily life.

MR. KILLAM: Yes. I think one of the areas and some of the problems I
have seen suggest that it\'s difficult for a person to be able to tell
where they are, how much is left, have they finished. I can\'t imagine
how they are trying to get that information across to the person who
can\'t see. I think the issue of feedback, the issue of progress status,
the issue of how much time do I have left. The kinds of things that can
affect roll-off voting or just even giving up completely... I don\'t
think we\'ve look at that, yet.

MR. ELEKES: Thank you.

CHAIRPERSON QUESENBERY: Thank you, Mr. Killam.