Verification through User Value:

Or 'How to Avoid Drinking Your Own Bathwater in Ubicomp Evaluations'

Christopher A. Miller and Harry B. Funk

Smart Information Flow Technologies\
2119 Oliver Avenue South\
Minneapolis, Minnesota, 55405-2440 U.S.A.\
*{cmiller, hfunk}@SIFTech.com*

Evaluation of any complex, knowledge-based aiding system is
extraordinarily difficult. Ubiquitous computing (ubicomp) systems expand
this problem. Evaluation can mean at least two things: the traditional
notions of 'validation' and 'verification'. *Validation* refers to
whether or not the system has been built and performs as it was
specified. *Verification* refers to whether or not it provides the
benefit intended. Validation is an evaluation of the as-built system
against the requirements and subsequent design, while verification is
(given that validation was passed), an evaluation of the design against
reality. For the purposes of this position paper, we will sidestep the
important issues of validation in order to concentrate on the more
complicated and, ultimately, more important issues associated with
verification.

The problem is that in active research domains such as ubicomp, any
design is essentially 'just' an educated guess about what correct
behavior should be. This has two profound effects on the evaluation
process. First, it means that the relative importance of performing good
verification is much greater than it might be in a better understood
engineering domain. Design becomes the advancing of a hypothesis about
correct behavior and verification, the experiment which tests the
hypothesis. Second, it means that the definition of 'correct behavior'
for the system in a test case becomes more problematic, as will be
discussed next.

Knowing what is 'correct behavior' is not trivial when designing a
highly complex, novel technology to be used by diverse human users in
varying circumstances. Ubicomp systems will inevitably include logic
about how to configure and adapt behavior and information presentation
to the set of contexts, devices and user needs they encounter. But,
given the complexity and potential interacting factors in the domain,
correct answers are rarely obvious for all possible situations. This is
confounded by the fact that the human interaction variables which must
be accounted for in evaluation are themselves not always rigorously
predictable and well understood. A team of designers will take their
best and studied approach to creating logic (and implementing it in
methods and metrics) that computationally determines correct behavior
for the system under a range of circumstances. This *is* the best, most
formal prediction available for generating correct behavior across
contexts; if there were a better one, the team would have used it. But,
this leaves the problem of identifying correct behavior in a given
evaluation scenario by some method different from, and ideally better
than, the computational method embedded in the newly-designed ubicomp
system. We refer to this as the *problem of circularity* in development
and test. It could also be called 'how do we avoid drinking our own
bathwater?'![](media/image1.wmf){width="3.1527777777777777in"
height="2.0395833333333333in"}

Fortunately, we've lived through this problem before in developing
adaptive systems in complex, high-criticality, real world domains like
piloting and industrial process control---in particular, the flight test
of the U.S. Army's Rotorcraft Pilot's Associate and the evaluation of
DARPA's Agile Information Control Environment. In those domains, as in
ubicomp, we formalized computational approaches to adapting the
presentation of information and the behavior of automation to the
context of use, user and available devices. Lessons learned there should
apply here as well.

The situation is illustrated in the figure below. For some set of
stimulus contexts, the act of system design is the creation of a logic
which provides a set of output responses. The problem is that there is
typically no true alternative to the logic or algorithms embedded in the
system for determinng what any general probe stimulus should produce as
a response. Hence, we are tempted to use the same logic to identify
correct responses for test that we have embedded in the system---but
this is circular.

Fortunately, the problem of circularity only applies to the general
case, not always to the specific. While it is frequently impossible to
acquire a better (or even different) model of correct behavior across
the full set of context stimuli, it may be possible in some *specific*
cases for us to know what is right or wrong. To the degree that the
system behaves inappropriately where an answer is known, then it is not
fully verified. If we can identify a robust and representative set of
verification cases that adequately sample the domain, and if the system
behaves appropriately on them, then we have some confidence that it is
verified.

So how do we know what is appropriate behavior---even for the specific
case---in a way independent from that implemented? A saving grace of
ubicomp systems is that they have a firm bottom line. In order to be
verified, a design must provide the benefit intended. For any
human-in-the-loop system is that it must provide *user value*. User
value can come through either (1) a more satisfying user experience, (2)
improved task performance, or (3) an expanded range of task performance
(i.e., the ability to do things impossible to do before, or do them in
locations previously impossible)---or through appropriate combinations
of these factors. In other words, if we can demonstrate enhanced user
value with a ubicomp system over a comparative case without one, then
*de facto* we have something worth having---and if it is the value
intended, then we have verified (at least that part of) the system.

Thus, we argue that verification of ubicomp systems *must* be done with
reference to a well-chosen, representative set of user-in-the-loop
scenarios for which appropriate and inappropriate behaviors have been
identified with reference to user value criteria. Metrics for these
criteria can frequently be established independently of the algorithms
used in the ubicomp system, at least for the specific scenario under
examination. For example, user satisfaction can be derived from
subjective Likert-scale based ratings or through the use of the demand,
frustration and effort subscales of NASA's Task Load Index (TLX) rating
method. Improved task performance metrics will be domain specific, but
they should likely include factors such as error rates and time of task
performance. Expanded range metrics might include an analysis of the
times and locations in which tasks are performed. In each case, the
comparison must be made between the ubicomp system and some realistic
baseline method that is of interest.

The result, then, will be a series of data points determining whether
the algorithms encoded in the ubicomp system are providing value in the
specific cases chosen. Furthermore, we will have avoided the problem of
circularity by obtaining objective performance criteria outside of the
implemented ubicomp reasoning. The problem now becomes, however,
choosing a useful and sufficient set of points, since it is generally
impossible to test the full space of either stimuli or responses. This
is the familiar 'bed of nails' approach to testing as illustrated in the
figure---we know a point in the stimulus space should be connected with
a point in the response space, but we are limited by cost and time in
the number of 'nails' we can drive through the system in order to test.
So how do we chose good points to probe?

One set of points can come from initial design. Complex, adaptive
systems such as ubicomp are typically built by an initial round of
knowledge acquisition and usability interviews that, ultimately, yield
use cases. In creating these requirements, we have typically captured
how users think they want the system to behave at specific points. Thus,
one weak set of verification cases can be based on those used in initial
design, provided these were created with user input. This set can, of
course, be made stronger by determining, through user-in-the-loop
simulation and testing the set of responses that actually produce user
value, since these may differ from what the user says s/he wants.
Fortunately, since we know the specifications of the designed system, we
needn't treat the creation of test cases as pure black-box
evaluation---that is, we needn't choose test points based only on their
distribution in the stimulus and response spaces. Instead, we can choose
cases specifically because they test components of system reasoning that
designers believe are significant or risky,---as illustrated by the
central probe passing through the internal component box in the
figure---or to ensure coverage of all system components.

A problem is that humans will adapt their performance and their very
definition of the job to the new technology. Hence evaluations should
pose a set of well-designed problem scenarios, but allow free
exploration of them by human users. Experience with the system over time
may be important to give users a chance to learn and adapt to the new
technology. Wizard of Oz approaches which simulate the appearance and
behavior of the technology before it exists prove very useful since they
enable the anticipation of problems well before they exist. For similar
reasons, it is important to be able to establish a flexible hierarchy of
metrics to adapt to sometimes unpredictable user-definitions of goals.
For example, choosing time to perform a task as a criterion for success
may prove problematic if the new technology enables users to explore
more possible task methods or decisions---task duration may not show an
improvement, but task satisfaction and perhaps quality of decisions will
improve, and error rates may decline.

In summary, we believe that there is no substitute for user-in-the-loop
evaluations of ubicomp systems since, not only does this get at the
bottom line of user value, but it is also one of the few methods
available to avoid the problem of circularity in verification.
Nevertheless, there are substantial problems in selecting, constructing,
running and interpreting such evaluations. We will be able to provide
lessons learned from our work in developing and fielding adaptive aiding
systems in high-criticality real-world domains in the context of the
workshop.