# Presentation: 134401

## Observations and Evaluations of Collaborative Systems

- <date/time>

- Sara Bly
- February 1, 2001

**Notes:**
Rick:  Small groups (3-7) and a small number of groups (3-7) with future development aimed at including visualization.
What is being done now?
What works?  What doesn’t?  Where are the barriers?  Where are the sweet spots?

Are small groups able to collaborate effectively using the Access Grid?
	What is “collaboration”?  What is “effective”?

Does persistence matter?
Does access to the AG node site matter? (Yes, of course!)

What is the significance of using the spatial metaphor?
What is the relationship between moving and interacting in the MOO versus in the Access Grid?
What is the MOO culture?  The AG culture?  The videoconferencing culture?
What is the value of the additional MOO channel?

What is the impact of different audio setups?
What behaviors carry over from face-to-face?  What new behaviors emerge?  What behaviors aren’t possible?

How much attention is spent during a session on technology rather than content?

## Outline

- Background
- Portholes
- Evaluation
- The Question
- The Data
- The Findings

## My Background in Collaborative Systems

- Media Space (audio, couches!, remote camera, persistence, informal, self-display)
- Shared Drawing
- Portholes
- Text-based MUDs
- Anchored Conversations

**Notes:**
My perspective on collaborative systems is based on many years of work with technologies and the support of work group activity.  At the Xerox Palo Alto Research Center, I was part of a multi-disciplinary group that evolved and used media space, adding shared drawing and awareness systems to our environment.  In more recent years, I’ve worked with Elizabeth Churchill at FX Palo Alto Laboratory to look at lightweight environments such as text-based MUDs which lead to our design for Anchored Conversations.

## Portholes

**Notes:**
Although the Media Space was a persistent environment supporting a group spread geographically across 2 sites, I want to show you some examples today from Portholes, an early awareness system.  
Note: people at both sites, common areas, play, visitor in my office.
These are from about 1992-1993

## Use of Portholes

- Users took advantage of its availability.
- to build cohesion and community remotely.
- to coordinate activities locally.
- It encouraged informal interaction.
- The effects were subtle.
***Evaluation methods are an issue!***

## Portholes: Building Community

## Portholes: Building Community

## Portholes: Building Community

## Portholes: Local Activity

## Portholes: Informal Interactions

**Notes:**
Note that it’s almost 12noon in England which is about 4am in California.

## Lessons From Portholes

- Ask the questions that matter.
- Remember to look for the simple things.
- Don’t let the unforeseen go unnoticed.
- Expect answers in many places and forms.

**Notes:**
So, what do Portholes and awareness systems have to do with evaluating collaborative systems or the Access Grid more specifically.
Simplicity:  remember to look for the simple stuff
	Paul’s message to EuroPARC about the system; the aerobics class
	In shared drawing, the quick mode changes.
Serendipity: don’t let the unforeseen go unnoticed
	Waving to Scott at 2am.
Answers may be in many different places and in many different forms.
	Christmas figures; Alan’s remark to me.

## Evaluation

- Identify the questions.
- Gather the data.
- Think, reflect, brainstorm, analyze.
***Include Context and Process.***

**Notes:**
How do we get at the simple, the unforeseen, the different places and forms of information?
Methods include the context and process as well as the task itself.

## Identify the Questions

- Find a Focus, a Perspective
- Fix on a Person, an Object, a Location
- Comparison with face-to-face?
- Evolution of work practice?
- Opportunities for new technologies?

**Notes:**
It’s obvious in quantitative methods that you need to start with your hypothesis.  It’s not always so obvious in qualitative methods that it’s equally important to know what you’re asking.
Group Elicitation Method (“Dump and Clump”)

Sharing notes and ideas – as they would face-to-face?  As they would in another system?  Or in whatever way that proves effective?

## Access Grid Development(RS)

- Focus on small group-to-group situations?
- Remove need for node operator (or training).
- Improvements in audio/video/text/etc.
- Improve sense of presence.
- Understand options in modality/bw trades.
- Applications support (e.g. style guide).
- Scalability.

## Relative Values?

- Spatial metaphors; Navigation scheme
- UI options; Camera and audio placement, count
- Roles of text channels (e.g. back channel)
- Public vs private channels, modes
- Role of asynchronous interaction
- [Types of interaction/collaboration? Users?  Domains?]

## Access Grid Cookbook

- When is the Access Grid appropriate?
- When is something else better?
- What are the guidelines for evaluation of applications?
- What else can be learned from human factors input?

## Gather the Data

- Systematic.
- Objective (recordings, logging).
- Contextual (in situ).
- We’re bad at self-reporting.
- We don’t notice everyday events.
- We may miss the serendipity.

## Our Data Methods Include

- Artifact interviews
- Contextual inquiry
- Day in the life exercises
- Observations (with notes, often video, multiple perspectives)
- Anecdotes and collections
- System logging

## Think, Reflect, Analyze

- Debrief
- Written Notes
- Collections
- Brainstorming
- Interactive Analysis

## References

- Contextual Inquiry / Design (Holtzblatt, et al)
- Field Studies Casebook
- New book coming from Preece et al
- CHI / CSCW / PDA (inc. tutorials)