# Presentation: 780778

## User Centered Evaluationof the Access Grid

- Emilee Patrick, Motorola Labs
- epatrick@labs.mot.com

## Overview

- Motorola Labs and the Access Grid
- Goal 1: Understand AG usage
- Goal 2: Answer specific research questions
- Goal 3: Improve AG Node User Interface

- Emilee Patrick: UCR and the AG

## Motorola Labs and the AG

- About me...
- The User Centered Research group
- The “Collaborative Spaces” project

*AG Node demo, September 2000*

**Motorola Labs**

- Evaluating the AG

**Notes:**
Background in psychology and Human-Computer Interaction

Part of the user-centered research group in Motorola labs – different from traditional Motorola approach of “if you invent it they will come”
We work on developing future applications based on user research and needs analysis

We do things like conduct observations in the users’ “natural environment”, we can learn details about the work by direct observation that we can’t get any other way -- even by asking them directly.

“collaborative spaces” project – research ways to improve support for distance collaboration, not package and ship the AG
Includes learning about VMC, and ways to make the AGN easier for an average person to use on their own

The purpose of this project is to conduct studies that will produce requirements for redesigning the Access Grid Node (AG Node), and future systems to support communication and collaboration over distance. 

## A ‘Collaborative’ Space is...

- A** ****physical**** **location, supporting **collaboration**** **by:
**connecting** people in different locations
- communicating **awareness** of others’ activities
- conveying a sense of **presence**** ** or “being there”
- building relationships through **persistent** connection

**Motorola Labs**

- Evaluating the AG

**Notes:**
different people have different definitions of the term
our definition of collaborative spaces – centers on a physical space with a persistent connection to other remote physical spaces
Connection conveys awareness of others’ activities and a sense of presence or ‘shared space’

a study from Carnegie Mellon University of psych researchers, 2x more frequent communication with colleague in office next door than even others on the same floor

A LOT of literature out there on computer-supported collaborative work and video-mediated communication (ask me for references – epatrick@labs.mot.com)
I’ll touch briefly on 2 topics today

## Goal 1: Understand AG Usage

- Real-world data collection
- on-site observations
- observations over the AG
- user survey
- automatic data logging
- Purpose is to...
- understand who is using the AG and for what
- investigate backchannel (MUD) usage
- identify issues appropriate for closer study

**Goal 1**

- Evaluating the AG

**Notes:**
Need to learn more about the strengths and weaknesses of the AG thru real-world data collection – help us concentrate our efforts

On-site: operators and presenters
AG: record and code interactive sessions
For type of communication
For problems observed
Survey/ logging: find out who is using the AG and for what
Record and Code conversations
Track meeting number/ duration/ participating locations

The MUD is important for understanding how the AG works in real practice
Running a node is very resource-intensive, understanding MUD use might give us an idea of how to make using an AG node easier for the average user

## Understanding the user’s world...

- Observe the user like:
- Borrowing from cultural anthropology...
- Go where the users work
- Watch the actual users work while they work
- Ask questions about it
- Capture it: record it, get samples, etc.

*Margaret Mead conducting field research in Bali, circa 1957*

**Goal 1**

- Evaluating the AG

**Notes:**
To understand the user and his environment, you have to go there and experience it.

Self-reported data is three levels removed from the truth: 
Users tell you what they think you want to hear or what they think is a socially preferred answer (especially when they are part of a group) 
Users tell you what they remember believing that they did (but memory is highly fallible, especially regarding the specifics of interaction behavior) 
Users can only report what they believe they did; not what they actually did, and people always rationalize their behavior when thinking about it after the fact; also they don't even notice many of the things they do 

## Automatic Logging

- How to measure AG participation?
- chat logs
- frequency/ content of use
- session management activity
- Ways to analyze logged data
- code types of activities
- state transition and time allocation diagram
- content analysis of discussion
- look for patterns of activity

**Goal 1**

- Evaluating the AG

## Issues

- observation should take place over long periods of time
- multiple people in several physical locations to observe simultaneously
- issues with recording and informed consent (both video and logging)
- logging can be expensive and time-consuming to implement

**Goal 1**

- Evaluating the AG

## Goal 2: Research Questions

- Proposed ‘focus’ research
- aid coordination of speaking turns
- provide awareness of how one’s location appears to others
- The ‘video quality’ question
- how much is enough?
- well... enough for what, exactly??
- Enabling technology research
- System: simplification, reliability, security/ encryption, integrate wireless devices
- Video: delay, quality, synch with directional audio
- Multicast: performance, many-to-many scalability,  dynamic QoS

**Goal 2**

- Evaluating the AG

**Notes:**
When speaking turns happen it can be disorienting to try and find the person talking
This may impede interactivity with novice users

Investigate factors that impact Focus, and how to aid coordination of speaking turns, under different circumstances
	number of participants
	pace of conversation
	image size/ placement
	video quality

Video quality and nonverbal cues: this is the question everyone asks. Vary video quality and see what nonverbal content comes through.

## Other Ideas

- Immersion and audience immediacy
- does increasing immediacy result in a greater feeling of shared space? does this really matter?
- Anonymity and privacy
- how do these factors impact trust formation, essential for effective and high-performing groups
- Selective attention and information overload
- how much is really too much for an AG display? how can organization help?

**Goal 2**

- Evaluating the AG

**Notes:**
Audience Immediacy: the presence of an audience has been shown to impact human behavior (social facilitation) and contribute to group cohesiveness. Do we see the same effect with the AG Node? If we make the Node environment more immersive,  does a remote audience seem more immediate?
Anonymity, Privacy, Trust: currently, there are no privacy (or security) mechanisms built in to the AG Node – other than leaving the room or shutting off your own cameras and microphones. With video a participant cannot be anonymous like they can in a MUD. It is possible for the AGN to transmit voice only – how does this affect impression formation?
Information Overload: an AG event can theoretically involve 30 different remote locations – is this useful? At what point to users become unable to keep track of what is going on? Can we build capabilities into the systems to help manage this problem?

## Issues

- Recruiting participants is complicated by the fact that a group is needed each time!!
- Hard to duplicate real conditions of use in an experimental setting
- Representative sample – who are the users?
- Technology is presently somewhat unreliable
- Coordinating between multiple sites is problematic

**Goal 2**

- Evaluating the AG

## Goal 3: AG Node User Interface

- Nodes are just plain hard to run
- Usability testing to identify interface problems
- Needs to wait until the other two goals are at least partially met

**Goal 3**

- Evaluating the AG

## Issues

- Same issues as experimental studies:
- a group is required for each session
- hard to duplicate real conditions of use
- Won’t yield useful results until the system is more robust

**Goal 3**

- Evaluating the AG

## The End!

- Emilee Patrick, Senior Human Factors Engineer
- User Centered Research, Motorola Labs
- Rm. 2230, 1301 E. Algonquin Rd. Schaumburg, IL 60196
- phone: 847.538.6886  fax: 847.576.3240

- Evaluating the AG

## References

- Beyer, H. and Holtzblatt, K. (1998). Contextual Design: Defining Customer Centered Systems, San Francisco, CA: Morgan Kauffman.
- Gutwin, C. and Greenberg, S. (2000). The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces. IEEE 9th International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET-ICE'00).
- Hackos, J.T., and Redish, J.C. (1998). User and Task Analysis for Interface Design. Wiley: New York.
- Nielsen, J. (1999). Voodoo Usability http://www.useit.com/alertbox/991212.html
- Ramage, M. and Ross, S. (1996). Evaluation Methods in the CSCW Literature. http://www.comp.lancs.ac.uk/computing/research/cseg/projects/evaluation/lit_methods.html

- Evaluating the AG