# Presentation: 666458

## experience sampling method:  preliminary findings

- sunny consolvo & miriam walker
- intel research seattle
- sunny@intel-research.net
- user-centered evaluations of ubicomp workshop
- 4th int’l conference on ubiquitous computing
- september 29, 2002

**Notes:**
good morning
introduce self
today, I’ll be talking about work Miriam Walker and I did in applying experience sampling method to ubiquitous computing

## before we get started...

**this work is** **preliminary**
- wrapped up data collection this month
- expect future publication

- experience sampling method -- ubicomp 2002

**Notes:**
Before we get started, I want to point out that what I’ll be discussing today is PRELIMINARY

we just finished data collection in the 2nd week of September
we have yet to analyze the data
we expect to publish something about our experiences in the future, once we’ve had a chance to go through the data

I’ll try my best to handle your questions, but please be prepared for me to have to “get back to you”

## outline of talk

- intro to *experience sampling method (esm)*
- case study:  *the personal server*
- issues
- wrap-up

- experience sampling method -- ubicomp 2002

**Notes:**
I’ll first give you a brief introduction to the method

then I’ll tell you about how we applied it to the Intel Research Personal Server project

I’ll mention some issues we’ve encountered so far

and finally wrap-up and take your questions

## esm:  a brief introduction

- also called:
- beeper studies
- time sampling
*field study* technique from *psychology*
- used to understand areas such as...
- mood
- time use
- social interactions
- we’re exploring its uses for ubicomp

- experience sampling method -- ubicomp 2002

**Notes:**
experience sampling method (or esm) is sometimes called:
beeper studies
time sampling studies

it is a field study technique

originated in the field of psychology

traditionally used to understand areas such as mood, time use, and social interactions
for mood specifically, refers to things like
depression
drug treatments
drug cravings 
withdrawals
affects of exercise

we’re exploring its uses for ubicomp

## esm:  a brief introduction

- fill out questionnaire when alerted
- researcher is not present
- alerted several times per day
- business as usual until alerted
- duration is often 1-3 weeks

- experience sampling method -- ubicomp 2002

**Notes:**
what happens is...
participants fill out a questionnaire whenever they are alerted
some important things to note are that...
no researcher is present
participants are often alerted several times/day
in theory, participants go about their daily business until an alert sounds and they have to fill out a questionnaire

duration of study is often 1-3 weeks

## running esm:  tools

- need to choose tools for:
**alerting** participant
- random alerts
- timed alerts
- event-based/user-driven
**delivering** questionnaire
**capturing** participant responses

- experience sampling method -- ubicomp 2002

**Notes:**
to run esm, researchers need to choose tools for:

alerting the participant that it’s time to answer a questionnaire

delivering each questionnaire

capturing the participant’s responses to each questionnaire

depending on researcher needs, the type of alert will be...
random alerts
e.g., from the hours of 9am – 9pm, deliver 10 randomly spaced alerts
timed alerts
e.g., from the hours of 9am – 9pm, deliver 10 evenly timed alerts
event-based/user-driven alerts
e.g., when a certain event occurs for the participant, s/he triggers the questionnaire

## choosing tools:  alerting

- tools for *alerting* the participant include:
- cell phones
- pagers
- watches
- PDAs
- custom devices
- some desirable abilities:
- specify time period during which alerts may go off
- specify # of alerts per time period & overall

- experience sampling method -- ubicomp 2002

**Notes:**
tools that can be used to alert the participant that it’s time to complete a questionnaire include:
cell phones
pagers
watches
PDAs
custom made devices

it’s desirable that these tools give ability to:
specify time period during which alerts may go off
e.g., you don’t necessarily want alerts to be going off while the participant is sleeping
specify # of alerts per time period & overall
e.g., 10 alerts per day, 70 alerts for duration of study

## choosing tools:  delivering

- tools for *delivering questionnaires* include:
- paper booklets
- PDAs
- phones (traditional or mobile)
- live researcher
- recorded message
- audio recordings (e.g., cassette player)

- experience sampling method -- ubicomp 2002

**Notes:**
tools that can be used to deliver questionnaires to participants include:
paper booklets
PDAs
phones (traditional or mobile)
live researcher
e.g., participant calls special phone number when alerted and researcher on other end of phone reads off questionnaire
recorded message
e.g., participant calls special phone number when alerted and hears recorded message of questionnaire
audio recordings
e.g., when participant is alerted, they listen to an audio recording, perhaps via a cassette player

## choosing tools:  capturing

- tools for *capturing responses* include:
- paper booklets
- PDAs
- phones (traditional or mobile)
- live researcher
- recording service (e.g., Tellme)
- audio recordings (e.g., cassette player)
- cameras

- experience sampling method -- ubicomp 2002

**Notes:**
tools used to capture participant responses to the questionnaires include:
paper booklets
PDAs
phones (traditional or mobile)
live researcher records answers
recording service (e.g., Tellme)
audio recordings (e.g., cassette player)
cameras

## esm:  some advantages

- reduces effects of intrusive observers
- larger sample size & time interval
- reduces memory biases by asking about *current activity, feelings, etc.*
- it’s all about **now**

- experience sampling method -- ubicomp 2002

**Notes:**
side effects of having an observer watch the participant’s every move should be significantly reduced

researchers should be able to involve more participants over a greater time interval than would be feasible with most observational techniques

memory biases often experienced with interviews or diaries can be reduced by asking about current activity, feelings, etc.
it’s all about what’s happening NOW, who are you with NOW, how are you feeling NOW

## esm:  more advantages

- less time consuming than observations, yet...
- study activities *as they occur* *in the field*
- study activities and emotions *often not accessible* to observers
*not as disruptive* to others

- experience sampling method -- ubicomp 2002

**Notes:**
less time consuming than most  observations

however, researchers still get to 
study activities as they occur in the field
study activities and emotions they might not otherwise get to observe, e.g.,
on dates
at the movies
getting ready in the morning/evening
in fact, we had a participant fill out a questionnaire during sex
and it’s not as disruptive to people around participant as traditional observations
though it can be a little annoying, it’s not near as undesirable as having a person constantly present and taking notes or filming

## esm:  some disadvantages

- questionnaires must be brief
- participant must carry tools at all times
- power consumption and/or service availability
- certain events may only show up in very large samples sizes/durations
- more intrusive than interviews & surveys

- experience sampling method -- ubicomp 2002

**Notes:**
because questionnaires take place in daily life and participants are alerted several times per day, 
there’s only so much you can ask
a good rule of thumb is <= 2 min./questionnaire

also, 
participant must be carrying all of the tools at all times
depending on tools, power consumption and/or service availability could affect ability to complete questionnaire or study
certain events may only show up in very large samples sizes/durations
though less intrusive than observation, more intrusive than interviews, traditional surveys

## outline of talk

- intro to *experience sampling method (esm)*
- case study:*  personal server*
- issues
- wrap-up

- experience sampling method -- ubicomp 2002

**Notes:**
now I’ll tell you about how we applied it to the Intel Research Personal Server project

## case study:  personal server

*personal server* team was interested in:
- how people use information
**what** they want
**when** they want it
**where** they want it
- used esm to help collect this data

- experience sampling method -- ubicomp 2002

**Notes:**
personal server project team at intel research was interested in:
how people used information
what information they needed
when they needed that information
where they needed that information

we used esm to help collect this data for them

## pilot study #1

- 4 participants
- lab researchers & administrative staff
- palm m100s for
*alerting*
*delivering*
*capturing*
- original boston college ESP software
- for running ESM on palms
- duration:  11⁄2 - 4 days

- experience sampling method -- ubicomp 2002

**Notes:**
prior to running the actual study, we conducted (2) pilot studies:
the first pilot study included
4 participants, made up of 
lab researchers
administrative staff

they used
palm m100s for alerting, delivering, and capturing
open-source software for running ESM on palms called ESP
developed by Boston College

## pilot study #1:  problems

- palm dropped  batteries fell out  data lost
- only method of alert was beep (m100)
- participants wouldn’t carry it in several situations
- once volume of beep was set, it couldn’t be changed
- only response option:  multi-choice, single answer
- difficult for participants to answer some questions
- e.g., “where are you?”
- some participants wanted a bag to carry the palm
- participants wanted to know about missed questionnaires

- experience sampling method -- ubicomp 2002

**Notes:**
we had a couple of instances where the palm dropped  which almost guaranteed that the batteries fell out  leading to a complete loss of data

only method of alert was via beep (m100)
participants wouldn’t carry it in several situations
meetings
movies
nicer restaurants
once volume was set, it couldn’t be changed
this is because for reasons including limiting power consumption, the participant was locked out of all features except the ESP software

only response option:  multi-choice, single answer (ESP software)
difficult for participants to answer some questions
e.g., “where are you?”

some participants wanted a bag to carry the palm

participants wanted to know if they missed a questionnaire

## pilot study #2

- 4 participants
- lab researchers, administrative staff, family
- modified iESP software
- palm m500s for *alerting* and *delivering*
- for *capturing*:
- 2 participants carried *fold-out keyboards* (for palm)
- 2 participants carried *paper booklets*
- all 4 carried *digital cameras*
- for describing where & what they were doing
- duration: 11⁄2 - 4 days

- experience sampling method -- ubicomp 2002

**Notes:**
4 participants
lab researchers, administrative staff, family
iESP software—which is software we modified with the help of a student at UW, started from existing ESP software
palm m500s for alerting and delivering
for capturing:
2 participants carried fold-out keyboards (for palm)
2 participants carried paper booklets
all 4 carried digital cameras
for describing where & what they were doing
duration: 11⁄2 - 4 days

## pilot study #2:  justifications

- upgrading to m500s:
- ability to alert via *beep* and/or *vibrate*
- almost impossible for batteries to fall out
*iESP *software added functionality, including:
- freeform text entry
- multi-choice, multi-answer
- new formats for multi-choice, single answer
- slider (e.g., “on a scale of 1 to 5...”)
- ability to change volume, vibrate, etc. during study
- ability to check how many questionnaires have been completed vs. asked during study

- experience sampling method -- ubicomp 2002

## our implementation:  the study

- 31 participants
- recruited from local community
- pre/post-study interviews + 1 week with equipment
- 10 questionnaires/day
- 70 questionnaires total
- incentive (in USD)
- $50 for participating and returning equipment
- $1 per completed questionnaire
- $120 total possible

- experience sampling method -- ubicomp 2002

**Notes:**
31 participants completed the study
recruited from local community
primary criteria was that they regularly used a cell phone
recruiting form was on website, so assumed some familiarity with internet
matched target users for personal server as closely as possible given that we did our own recruiting

one hour long pre- and post-study interviews + one week of questionnaires:
10 questionnaires/day
70 questionnaires total

$50 USD for participating and returning equipment 
$1 per completed questionnaire
per questionnaire incentive was to try to get participants to fill out as many questionnaires as possible
total incentive possible was $120

## the study (continued)

- palm m500 to *alert*, *deliver*, & *capture*
- modified *iESP *software
- more about *capturing*:
- also used camera
- free-form data entry:
*paper booklet* for all 31 participants
*graffiti */ *soft keyboard* for all 31 participants
*full-sized fold-out keyboard* for 9 of 31 participants

- experience sampling method -- ubicomp 2002

**Notes:**
we used a palm m500 for alerting, delivering, & capturing
modified existing open-source software, esp
based on feedback from pilot testers and data we wanted to collect
miriam and I advised micah brodsky of UW who modified actual software

more about capturing:
also used camera
free-form data entry:
paper booklet for all 31 participants
graffiti and/or soft keyboard on palm for all 31 participants
full-sized fold-out keyboard on palm for 9 of 31 participants

## what the participants carried

- experience sampling method -- ubicomp 2002

**Notes:**
here’s a look at all of the equipment the participants carried
9 of the 31 participants carried all of this
22 carried all but the fold-out keyboard

for those of you not familiar with the palm’s foldout keyboard, here’s what it looks like
it is a full-sized keyboard that folds up to be a little larger than the palm itself

## type of data collected

- responses to following types of questions:
- yes / no
- multiple choice, single answer
- multiple choice, multiple answers
- freeform text entry
- looks a lot like survey data except
- multiple responses from same participant in variety of circumstances over time

- experience sampling method -- ubicomp 2002

## other data collected

- qualitative data from interviews
- participants’ photos
- some photos were *of interest* to the participant
- other photos documented *where* s/he was or *what* s/he was doing

- experience sampling method -- ubicomp 2002

**Notes:**
qualitative data from pre- and post-study interviews
participants took photo(s) while filling out questionnaires
some photos were of interest to the participant
other photos documented where they were / what they were doing

## questions on the palm

- experience sampling method -- ubicomp 2002

## more questions

- experience sampling method -- ubicomp 2002

## participant photos:  in transit

- experience sampling method -- ubicomp 2002

**Notes:**
several of the photos participants took were of people—friends, colleagues, themselves
because only the participants signed release forms, I can’t show you any of those photos

I can show you some other common subjects

## participant photos:  at home

- experience sampling method -- ubicomp 2002

## participant photos:  at work

- experience sampling method -- ubicomp 2002

## participant photos:  retail

- experience sampling method -- ubicomp 2002

## participant photos:  pets

- experience sampling method -- ubicomp 2002

## outline of talk

- intro to *experience sampling method (esm)*
- case study:  *personal server*
- issues
- wrap-up

- experience sampling method -- ubicomp 2002

**Notes:**
now I’ll talk about some issues we’ve discovered so far

## our tool decisions...

- chose palm because:
- no service problems—cost or coverage
- power consumption not an issue for duration
- know when participant fills out questionnaire
- probabilities and contingencies can be used
- if *a*, then *b*
- if *b*, then not *c*
*d* 50% of time
- don’t have to transcribe data

- experience sampling method -- ubicomp 2002

**Notes:**
chose palm because:
no service problems—cost or coverage
it’s a one-time cost
power consumption not an issue for duration
know when participant fills out questionnaire
probabilities and contingencies can be used, e.g.
if participant answers “a,” ask “b”
if “b” is asked, don’t ask “c”
ask “d” 50% of the time
don’t have to transcribe data

## tool decisions

- used camera to see if it would help participants describe...
**where** they were
**what** they were doing
- photos used to jog memories during post-study interviews

- experience sampling method -- ubicomp 2002

**Notes:**
used camera to see if it would help participants describe
where they were
what they were doing
a bonus:  proved very useful in jogging their memories during post-study interviews

## according to the participants...

- 1/31 **would not** participate again
- 2/31 would only do it for **more money**
- 28/31 would participate again
- “it was cool”
- “I felt like a spy”
- “friends were jealous of the [fold-out] keyboard”
- a lot to carry
- mixed feelings about carry bag
- several claimed they would be purchasing palms
- only one participant wanted voice recording option
- when asked, most did not respond favorably

- experience sampling method -- ubicomp 2002

**Notes:**
1/31 would not participate in a study like this again
2/31 would only do it for more money
28/31 would participate in a study like this again
“it was cool”
“I felt like a spy”
“friends were jealous of the [fold-out] keyboard”

a lot to carry
mixed feelings about carry bag
several claimed they would be purchasing palms
we found this interesting, as they were locked out of all palm features EXCEPT the esm software

only one participant wanted voice recording option
when asked, most did not respond favorably
they said that the palm was much more discrete, and that having to speak the answers meant that they wouldn’t have been able to complete as many questionnaires, particularly while
in meetings
in movies
at a restaurant
etc.

## from our perspective...

- woo-hoo!  we got all of the equipment back
- only 1/31 palms was broken
- of the 31 participants and 70 questionnaires:
- 28 participants completed 50+ questionnaires
- 13 of whom completed 60+ questionnaires
- 2 participants completed 25- questionnaires
- 1 participant performed a hard reset and returned the palm with no data or software

- experience sampling method -- ubicomp 2002

**Notes:**
from our perspective
we were nervous we might not get all of the equipment back because the value of the equipment was greater than the maximum incentive we were offering
only 1/31 palms was broken
data was lost for only 1/31 participants
who performed a hard reset
of the 31 participants and 70 questionnaires:
28 participants completed 50+ questionnaires
13 of whom completed 60+ questionnaires
2 participants completed 25- questionnaires

## outline of talk

- intro to *experience sampling method (esm)*
- case study:  *personal server*
- issues
- wrap-up

- experience sampling method -- ubicomp 2002

**Notes:**
I’ll do a quick wrap-up then take your questions

## wrap up:  we’ll use it again

- pleased with results so far
- intend to use esm again

- experience sampling method -- ubicomp 2002

**Notes:**
we already have plans to use it on another project

## wrap-up:  pda software

- original **ESP** software available at:
- _[http://www2.bc.edu/~barretli/esp/](http://www2.bc.edu/~barretli/esp/)_
**iESP** software (**includes** upgrades):
- launching on intel research seattle web site in Oct or Nov 2002
- if you want to be notified...
- business card with “iESP notification” written on it
- legibly fill out notification sign-up sheet

- experience sampling method -- ubicomp 2002

**Notes:**
if you want to run an ESM study with Palms in the next couple of months...
the open-source ESP software from Boston College is available for download
developed by Drs. Lisa and Daniel Barrett
works for Palm Pilots and Handspring Visors (requirements listed on site)
does not contain the upgrades I discussed

iESP, which is the software we modified with a student at UW, will be available for download later this fall
iESP contains the upgrades I discussed
it will be open-source as well
if you’re interested in being notified of the site launches...
give me your business card with “iESP notification” written on the front or back
legibly fill out sign-up sheet I’m passing around

## acknowledgements

- drs. lisa & daniel barrett
- justin silves
- gaetano borriello
- ken anderson
- rahul shah
- trevor paring
- roy want
- john light
- our pilot testers
- and our colleagues at intel research seattle

- experience sampling method -- ubicomp 2002

**Notes:**
our sincere thanks go to...

Drs. Lisa & Daniel Barrett of Boston College—authors of the original ESP software who have been very cooperative with us

Justin Silves of intel research seattle provided tremendous administrative support including recruiting and scheduling participants, and typing up interview notes

Gaetano Borriello, IRS lab director, who not only supported the research but also paid for everything

Ken Anderson a psychologist from Intel People & Practices Research who advised us on the changes we made to the method

Rahul Shah, an intern at IRS from UC Berkeley who wrote a perl script to help us make the data more usable

Roy Want, Trevor Paring, & John Light of the Intel Research personal server team

our pilot testers

and our colleagues at intel research seattle for putting up with a lot of beeping palms

## questions?

- experience sampling method -- ubicomp 2002

## examples of study questions

- where is your cell phone?  in my...
- hand
- pocket/belt clip
- bag
- car
- home
- office
- somewhere else and with me
- somewhere else and not with me

- experience sampling method -- ubicomp 2002

## examples of study questions

- in the last 30 minutes, which of the following would you have liked to have access to but didn’t have available?
- web pages
- movie times
- restaurant reviews
- personal calendar
- telephone number(s)
- computer files
- music files
- shopping list
- “to do” list

- experience sampling method -- ubicomp 2002