### Evaluating capture and access through authentic use

Heather Richter and Gregory Abowd

Georgia Institute of Technology

One of the challenges in evaluating ubiquitous computing technologies is
in understanding the real motivations behind their usage and their
impact on everyday activities. However, it is difficult to gain this
understanding in laboratory settings. People have difficulty envisioning
how they would really incorporate ubiquitous applications into their
lives. Evaluating whether a user can interact with such a technology
does not help in understanding why and how they would interact with the
technology on an everyday basis. Thus, in order to truly understand the
usefulness and impact of a ubiquitous application, it needs to be
evaluated in a realistic setting.

At Georgia Tech we have been evaluating a ubiquitous capture and access
system, eClass, through repeated, authentic use in classrooms \[1\].
EClass has been used for thousands of lectures, leading to deep
understanding of the impact of the system on both students and teachers.
However, this experience also taught us several lessons. First, we need
to design for evaluation. We were lucky we were able to maintain and
collect data from eClass for so long. Still, the data collected from
eClass was unwieldy and occasionally uncertain because of the way
actions were logged. Additionally, we need to evaluate from the
beginning, and evaluate the evaluation. While we collected data from
eClass early on, we did not analyze that data for a long time. Such
analysis could have led to more iterations on eClass and shown
additional evaluations or experiments to complement what was already
completed.

We are now examining capture and access in a meeting environment with a
new system called TeamSpace. The desire for long term, authentic usage
heavily influenced the design with obvious requirements of reliability,
robustness, and usability. Additionally, we attempted to better
integrate the system into current meeting and work practices so that
repeated usage is encouraged. For example, although a meeting capture
system need only focus on face-to-face meetings, the system can be used
for distributed meetings as many groups struggle with supporting them.
TeamSpace now appeals to both those with capture needs, and those with
needs of supporting distributed meetings. We are now ready to deploy the
system and evaluate its use.

As in many ubiquitous systems, TeamSpace has the potential to provide
benefit, yet there is no glaring need we are addressing. Additionally,
meetings, teams, and work vary greatly, and the real motivations for
using such a system are unclear. Thus, we wish to use TeamSpace to
understand the motivations and circumstances for ubiquitous meeting
capture and access. The evaluation will focus on when people use the
system, what tasks they are attempting to accomplish when reviewing
meetings, and how they are accomplishing those tasks. TeamSpace is
successful if it is used regularly and perceived as useful. With an
incomplete prototype, however, this will be challenging. Thus, the first
aspect of the evaluation will be to determine what aspects of the system
encourage repeat usage, and what tasks users are attempting. Thus, the
initial evaluation is going to depend heavily on a mix of both
quantitative and qualitative data gathered from system logs,
questionnaires, interviews, and observations of a team of people. Each
evaluation will lead us to iterate and improve TeamSpace, then do
iteratively more specific and focused evaluations.

Brotherton, Jason. "Enriching Everyday Activities through the Auotmated
Capture and Access of Live Experiences," Ph.D. Dissertation. Georgia
Institute of Technology, 2001.