Towards

Task-based

Personal

Information

Management

Evaluations

David

Elsweiler

Department

Computer

and

Information

Sciences,

University

of

Strathclyde

dce@cis.strath.ac.uk

Ian

Ruthven

Department

Computer

and

Information

Sciences,

University

of

Strathclyde

ir@cis.strath.ac.uk

ABSTRACT

Personal

Information

Management

(PIM)

is

a

rapidly

growing

area

of

research

concerned

with

how

people

store,

manage

and

re-find

information.

A

feature

of

PIM

research

is

that

many

systems

have

been

designed

to

assist

users

manage

and

re-find

information,

but

very

few

have

been

evaluated.

This

has

been

noted

by

several

scholars

and

explained

by

the

difficulties

involved

in

performing

PIM

evaluations.

The

difficulties

include

that

people

re-find

information

from

within

unique

personal

collections;

researchers

know

little

about

the

tasks

that

cause

people

to

re-find

information;

and

numerous

privacy

issues

concerning

personal

information.

In

this

paper

we

aim

to

facilitate

PIM

evaluations

by

addressing

each

of

these

difficulties.

In

the

first

part,

we

present

a

diary

study

of

information

re-finding

tasks.

The

study

examines

the

kind

of

tasks

that

require

users

to

re-find

information

and

produces

a

taxonomy

of

re-finding

tasks

for

email

messages

and

web

pages.

In

the

second

part,

we

propose

a

task-based

evaluation

methodology

based

on

our

findings

and

examine

the

feasibility

of

the

approach

using

two

different

methods

of

task

creation.

Categories

and

Subject

Descriptors

[Information

Search

and

Retrieval]:

General

Terms

Measurement,Management,Experimentation,

Human

Factors

INTRODUCTION

Personal

Information

Management

(PIM)

is

a

rapidly

growing

area

of

research

concerned

with

how

people

store,

manage

and

re-find

information.

PIM

systems

the

methods

and

procedures

by

which

people

handle,

categorize,

and

retrieve

information

on

a

day-to-day

basis

are

becoming

increasingly

popular.

However

the

evaluation

of

these

PIM

systems

is

problematic.

One

of

the

main

difficulties

is

caused

by

the

personal

nature

of

PIM.

People

collect

information

as

a

natural

consequence

of

completing

other

tasks.

This

means

that

the

collections

people

generate

are

unique

to

them

alone

and

the

information

within

a

collection

is

intrinsically

linked

with

the

owner"s

personal

experiences.

As

personal

collections

are

unique,

we

cannot

create

evaluation

tasks

that

are

applicable

to

all

participants

in

an

evaluation.

Secondly,

personal

collections

may

contain

information

that

the

participants

are

uncomfortable

sharing

within

an

evaluation.

The

precise

nature

of

this

information

what

information

individuals

would

prefer

to

keep

private

varies

across

individuals

making

it

difficult

to

base

search

tasks

on

the

contents

of

individual

collections.

Therefore,

experimenters

face

a

number

of

challenges

in

order

to

conduct

realistic

but

controlled

PIM

evaluations.

A

particular

feature

of

PIM

research

is

that

many

systems

have

been

designed

to

assist

users

with

managing

and

re-finding

their

information,

but

very

few

have

been

evaluated;

a

situation

noted

by

several

scholars

Recently,

however,

researchers

have

started

to

focus

on

ways

to

address

the

problem

of

PIM

evaluation.

For

example,

Kelly

proposes

that

numerous

methodologies

must

be

taken

to

examine

and

understand

the

many

issues

involved

in

PIM,

although,

she

makes

explicit

reference

to

the

need

for

laboratory

based

PIM

studies

and

a

common

set

of

shared

tasks

to

make

this

possible.

Capra

also

identifies

the

need

for

controlled

PIM

lab

evaluations

to

complement

other

evaluation

techniques,

placing

specific

emphasis

on

the

need

to

understand

PIM

behaviour

at

the

task

level.

In

this

paper,

we

attempt

to

address

the

difficulties

involved

to

faciliate

controlled

laboratory

PIM

evaluations.

In

the

first

part

of

this

paper

we

present

a

diary

study

of

information

re-finding

tasks.

The

study

examines

the

kind

of

tasks

that

require

users

to

re-find

information

and

produces

a

taxonomy

of

re-finding

tasks

for

email

messages

and

web

pages.

We

also

look

at

the

features

of

the

tasks

that

make

re-finding

difficult.

In

the

second

part,

we

propose

a

task-based

evaluation

methodology

based

on

our

findings

and

examine

the

feasibility

of

the

approach

using

different

methods

of

task

creation.

Thus,

this

paper

offers

two

contributions

to

the

field:

an

increased

understanding

of

PIM

behaviour

at

the

task

level

and

an

evaluation

method

that

will

facilitate

further

investigations.

RELATED

WORK

A

variety

of

approaches

are

available

to

study

PIM.

Naturalistic

approaches

study

participants

performing

naturally,

completing

their

own

tasks

as

they

occur,

within

familiar

environments.

These

approaches

allow

researchers

to

overcome

many

of

the

difficulties

caused

by

the

personal

nature

of

PIM.

As

the

tasks

performed

are

real

and

not

simulated,

the

participants

can

utilise

their

own

experiences,

previous

knowledge

and

information

collections

to

complete

the

tasks.

A

benefit

of

the

approach

is

that

data

can

be

captured

continuously

over

extended

time

periods

and

measurements

can

be

taken

at

fixed

points

in

time

within

these

Naturalistic

approaches

can

be

applied

by

conducting

fieldwork

ethnographic

methods

as

suggested

by

or

via

log

file

analysis

Both

ethnographic

and

fieldwork

methods

require

the

presence

of

an

experimenter

to

assess

how

PIM

is

performed,

which

raises

a

number

of

issues.

Firstly,

evaluation

in

this

way

is

expensive;

taking

long

time

periods

to

study

small

numbers

of

participants

and

these

small

samples

may

not

be

representative

of

the

behaviour

of

larger

populations.

Secondly,

because

participants

cannot

be

continually

observed,

experimenters

must

choose

when

to

observe

and

this

may

affect

the

findings.

An

alternative

strategy

to

conducting

naturalistic

evaluations

is

to

utilise

log

file

analysis.

This

approach

makes

use

of

logging

software

that

captures

a

broad

sampling

of

user

activities

in

the

context

of

natural

use

of

a

system.

In

a

novel

PIM

search

tool

was

deployed

to

users

and

the

log

data

provided

detailed

information

about

the

nature

of

user

queries,

interactions

with

the

query

interface

and

about

properties

of

the

items

retrieved.

Log

file

analysis

is

a

powerful

methodology

as

it

allows

the

capture

of

a

large

quantity

of

detailed

information

about

how

users

behave

with

the

system

without

the

expense

and

distracting

influence

of

an

observer.

Nevertheless,

there

are

limitations

to

this

strategy.

Firstly,

to

attain

useful

results,

the

deployed

prototype

must

be

something

that

people

would

use

i.e.

it

has

to

be

a

fully

functional

piece

of

software

that

offers

improvement

on

the

systems

ordinarily

available

to

participants.

Developing

a

research

prototype

to

this

standard

is

beyond

the

resources

of

many

researchers.

Further,

caution

must

be

taken

when

analysing

logs,

as

the

captured

data

shows

nothing

about

the

goals

and

intentions

that

the

user

had

at

the

time.

It

is,

therefore,

difficult

to

make

any

concrete

statements

about

the

reasons

for

the

behaviour

depicted

in

the

logs.

This

reveals

a

need

to

complement

naturalistic

studies

with

controlled

experiments

where

the

experimenter

can

relate

the

behaviour

of

study

participants

to

goals

associated

with

known

search

tasks.

Laboratory-based

studies

simulate

users"

real

world

environment

in

the

controlled

setting

of

the

laboratory,

offering

the

ability

to

study

issues

that

are

tightly

defined

and

narrow

in

scope.

One

difficulty

in

performing

this

kind

of

evaluation

is

sourcing

collections

to

evaluate.

Kelly

proposes

the

introduction

of

a

shared

test

collection

that

would

provide

sharable,

reusable

data

sets,

tasks

and

metrics

for

those

interested

in

conducting

PIM

research.

This

may

be

useful

for

testing

algorithms

in

a

way

similar

to

TREC

in

mainstream

IR

However,

a

shared

collection

would

be

unsuitable

for

user

studies

because

it

would

not

be

possible

to

incorporate

the

personal

aspects

of

PIM

while

using

a

common,

unfamiliar

collection.

One

alternative

approach

is

to

ask

users

to

provide

their

own

information

collections

to

simulate

familiar

environments

within

the

lab.

This

approach

has

been

applied

to

study

the

re-finding

of

personal

photographs

email

messages

and

web-bookmarks

The

usefulness

of

this

approach

depends

on

how

easy

it

is

to

transfer

the

collection

or

gain

remote

access.

Another

solution

is

to

use

the

entire

web

as

a

collection

when

studying

web

page

re-finding

This

may

be

appropriate

for

studying

web

page

re-finding

because

previous

studies

have

shown

that

people

often

use

web

search

engines

for

this

purpose

A

second

difficulty

in

performing

PIM

laboratory

studies

is

creating

tasks

for

participants

to

perform

that

can

be

solved

by

searching

a

shared

or

personal

collection.

Tasks

relate

to

the

activity

that

results

in

a

need

for

information

and

are

acknowledged

to

be

important

in

determining

user

behaviour

A

large

body

of

work

has

been

carried

out

to

understand

the

nature

of

tasks

and

how

the

type

of

task

influences

user

information

seeking

behaviour.

For

example,

tasks

have

been

categorised

in

terms

of

increasing

complexity

and

task

complexity

has

been

suggested

to

affect

how

searchers

perceive

their

information

needs

and

how

they

try

to

find

information

Other

previous

work

has

provided

methodologies

that

allow

the

simulation

of

tasks

when

studying

information

seeking

behaviour

However,

little

is

known

about

the

kinds

of

tasks

that

cause

people

to

search

their

personal

stores

or

re-find

information

that

they

have

seen

before.

Consequently,

it

is

difficult

to

devise

simulated

work

task

situations

for

PIM.

The

exception

is

the

study

of

personal

photograph

management,

where

Rodden"s

work

on

categorising

personal

photograph

search

tasks

has

facilitated

the

creation

of

simulated

work

task

situations

There

have

been

other

suggestions

as

to

how

to

classify

PIM

tasks.

For

example,

asked

participants

to

classify

tasks

based

on

how

frequently

they

perform

the

task

type

in

their

daily

life

and

how

familiar

they

were

with

the

location

of

the

sought

after

information

and

several

scholars

have

classified

information

objects

by

the

frequency

of

their

use

e.g.

While

these

are

interesting

properties

that

may

affect

how

a

task

will

be

performed,

they

do

not

give

experimenters

enough

scope

to

devise

tasks.

Personal

collections

are

one

reason

why

task

creation

is

so

difficult.

Rodden"s

photo

task

taxonomy

provides

a

solution

here

because

it

allows

tasks,

tailored

to

private

collections

to

be

categorised.

Systems

can

then

be

compared

across

task

types

for

different

users

Unfortunately,

no

equivalent

taxonomy

exists

for

other

types

of

information

object.

Further,

other

types

of

object

are

more

sensitive

to

privacy

than

photographs;

it

is

unlikely

that

participants

would

be

as

content

to

allow

researchers

to

browse

their

email

collections

to

create

tasks

as

they

were

with

photographs

in

This

presents

a

serious

problem

how

can

researchers

devise

tasks

that

correspond

to

private

collections

without

an

understanding

of

the

kinds

of

tasks

people

perform

or

jeopardising

the

privacy

of

study

participants?

A

few

methods

have

been

proposed.

For

example,

studied

email

search

by

asking

participants

to

re-find

emails

that

had

been

sent

to

every

member

in

a

department;

allowing

the

same

tasks

to

be

used

for

all

of

the

study

participants.

This

approach

ensured

that

privacy

issues

were

avoided

and

participants

could

use

things

that

they

remember

to

complete

tasks.

Nevertheless,

the

systems

were

only

tested

using

one

type

of

task

participants

were

asked

to

find

single

emails,

each

of

which

shared

common

properties.

In

section

we

show

that

people

perform

a

wider

range

of

email

re-finding

tasks

than

this.

In

generic

search

tasks

were

artificially

created

by

running

evaluations

over

two

sessions.

In

the

first

session,

participants

were

asked

to

complete

work

tasks

that

involved

finding

some

unknown

information.

In

the

second

session,

participants

completed

the

same

tasks

again,

which

naturally

involved

some

re-finding

behaviour.

The

limitations

of

this

technique

are

that

it

does

not

allow

participants

to

exploit

any

personal

connections

with

the

information

because

the

information

they

are

looking

for

may

not

correspond

to

any

other

aspect

of

their

lives.

Further,

if

time

is

utilised

by

a

system

or

interface

being

tested

the

approach

is

unsuitable

because

all

of

the

objects

found

in

the

first

session

will

have

been

accessed

within

the

same

time

period.

Our

review

of

evaluation

approaches

motivates

a

requirement

for

controlled

laboratory

experiments

that

allow

tightly

defined

aspects

of

systems

or

interfaces

to

be

tested.

Unfortunately,

it

has

also

been

shown

that

there

are

difficulties

involved

in

performing

this

type

of

evaluation

it

is

difficult

to

source

collections

and

to

devise

tasks

that

correspond

to

private

collections,

while

at

the

same

time

protect

the

privacy

of

the

study

participants.

In

the

following

section

we

present

a

diary

study

of

refinding

tasks

for

email

and

web

pages.

The

outcome

is

a

classification

of

tasks

similar

to

that

devised

by

Rodden

for

personal

photographs

In

section

we

build

on

this

work

by

examining

methods

for

creating

tasks

that

do

not

compromise

the

privacy

of

participants

and

discuss

how

our

work

can

facilitate

task-based

PIM

user

evaluations.

We

show

that

by

collecting

tasks

using

electronic

diaries,

not

only

can

we

learn

about

the

tasks

that

cause

people

to

re-find

personal

information,

but

we

can

learn

about

the

contents

of

private

collections

without

compromising

the

privacy

of

the

participants.

This

knowledge

can

then

be

used

to

construct

tasks

for

use

in

PIM

evaluations.

METHOD

Diary

Studies

are

a

naturalistic

technique,

offering

the

ability

to

capture

factual

data,

in

a

natural

setting,

without

the

distracting

influence

of

an

observer.

Limitations

of

the

technique

include

difficulties

in

maintaining

participant

dedication

levels

and

convincing

participants

that

seemingly

mundane

information

is

useful

and

should

be

reported

suggest

that

the

effects

of

the

negatives

can

be

limited,

however,

with

careful

design

and

good

implementation.

In

our

diary

study,

we

followed

the

suggestions

in

to

achieve

the

best

possible

data.

To

this

end,

we

restricted

the

recorded

tasks

to

web

and

email

re-finding.

By

asking

users

to

record

fewer

tasks

it

was

anticipated

that

participant

apathy

would

be

reduced

and

dedication

levels

maintained.

The

participants

were

provided

with

a

personalised

web

form

in

which

they

could

record

details

about

their

information

needs

and

the

contexts

in

which

these

needs

developed.

Web

forms

were

deployed

rather

than

paperbased

diaries

because

to

re-find

web

and

email

information

the

user

would

be

at

a

computer

with

an

Internet

connection

and

there

would

be

no

need

to

search

for

a

paper-based

diary

and

pen.

The

diary

form

solicited

the

following

information:

whether

the

information

need

related

to

re-finding

a

web

page

or

an

email

message

and

a

description

of

the

task

they

are

performing.

This

description

was

to

contain

both

the

information

that

the

participant

wished

to

find

and

the

reason

that

they

needed

the

information.

To

help

with

this,

the

form

gave

three

example

task

descriptions,

which

were

also

explained

verbally

to

each

participant

during

an

introductory

session.

The

experimenter

ensured

that

the

participants

understood

that

the

tasks

to

be

recorded

were

not

limited

to

the

types

shown

in

the

examples.

The

examples

were

supplied

purely

to

get

participants

thinking

about

the

kinds

of

things

they

could

record

and

to

show

the

level

of

and

type

of

details

expected.

The

form

also

asked

participants

to

rate

each

task

in

terms

of

difficulty

(on

a

scale

from

where

was

very

easy

and

was

very

hard).

Finally,

they

were

asked

when

was

the

last

time

they

looked

at

the

sought

after

information.

Again,

they

were

able

to

choose

from

options

(less

than

a

day

ago,

less

than

a

week

ago,

less

than

a

month

ago,

less

than

a

year

ago,

more

than

a

year

ago).

Time

information

was

used

to

examine

the

frequency

with

which

the

participants

re-found

old

and

new

information,

and

when

combined

with

difficulty

ratings

created

a

picture

of

whether

or

not

the

time

period

between

accessing

and

re-accessing

impacted

on

how

difficult

the

participants

perceived

tasks

to

be.

participants,

recruited

by

mass

advertisement

through

departmental

communication

channels,

research

group

meetings

and

undergraduate

lectures,

were

asked

to

digitally

record

details

of

their

information

re-finding

tasks

over

a

period

of

approximately

weeks.

The

final

population

consisted

of

academic

staff

members,

research

staff

members,

research

students

and

undergraduate

students.

The

ages

of

participants

ranged

from

As

both

personal

and

work

tasks

were

recorded,

the

results

collected

cover

a

broad

range

of

re-finding

tasks.

RESULTS

Several

analyses

were

performed

on

the

captured

data.

The

following

sections

present

the

findings.

Firstly,

we

examine

the

kinds

of

re-finding

tasks

that

were

performed

both

when

searching

on

email

and

on

the

web.

Next,

we

consider

the

distribution

of

tasks

which

kinds

of

tasks

were

performed

most

often

by

participants.

Lastly,

we

explore

the

kinds

of

re-finding

tasks

that

participants

perceived

as

difficult.

Nature

of

Web

and

Email

Re-finding

Tasks

During

the

study

tasks

were

recorded.

of

these

tasks

were

email

based,

were

webbased.

As

with

most

diary

studies,

the

number

of

tasks

recorded

varied

extensively

between

particpants.

The

median

number

of

tasks

per

participant

was

(interquartile

range

More

web

tasks

were

recorded

than

email

tasks

This

means

that

on

average

each

participant

recorded

approximately

one

task

every

two

days.

From

the

descriptions

supplied

by

the

participants,

we

found

similar

features

in

the

recorded

tasks

for

both

email

and

web

re-finding.

Based

on

this

observation

a

joint

classification

scheme

was

devised,

encompassing

both

email

and

web

tasks.

The

tasks

were

classified

as

one

of

three

types:

lookup

tasks,

item

tasks

and

multi-item

tasks.

Lookup

tasks

involve

searching

for

specific

information

from

within

a

resource,

for

example

an

email

or

a

web

page,

where

the

resource

may

or

may

not

be

known.

Some

recorded

examples

of

lookup

tasks

were:

Looking

for

the

course

code

for

a

class

it"s

used

in

a

script

that

is

run

to

set

up

a

practical.

I"d

previously

obtained

this

about

weeks

ago

from

our

website.

I

am

trying

to

determine

the

date

by

which

I

step

down

as

an

External

Examiner.

This

is

in

an

email

somewhere

Looking

for

description

of

log

format

from

system

R

developed

for

student

project.

I

think

he

sent

me

in

it

an

email

Item

tasks

involve

looking

for

a

particular

email

or

web

page,

perhaps

to

pass

on

to

someone

else

or

when

the

entire

contents

are

needed

to

complete

the

task.

Some

recorded

examples

of

item

tasks

were:

Looking

for

SIGIR

paper

to

give

to

another

student

Find

the

receipt

of

an

online

airline

purchase

required

to

claim

expenses

I

need

the

peer

evaluation

forms

for

the

MIA

class

E

sent

me

them

by

email

To

clarify,

lookup

tasks

differ

from

item

tasks

in

two

ways

in

the

quantity

of

information

required

and

in

what

the

user

knows

about

what

they

are

looking

for.

Lookup

tasks

involve

a

need

for

a

small

piece

of

information

e.g.

a

phone

number

or

an

ingredient,

and

the

user

may

or

may

not

know

exactly

the

resource

that

contains

this

information.

In

item

tasks

the

user

knows

exactly

the

resource

they

are

looking

for

and

needs

the

entire

contents

of

that

resource.

Multi-item

tasks

were

tasks

that

required

information

that

was

contained

within

numerous

web

pages

or

email

messages.

Often

these

tasks

required

the

user

to

process

or

collate

the

information

in

order

to

solve

the

task.

Some

recorded

examples

were:

Looking

for

obituaries

and

other

material

on

the

novelist

John

Fowles,

who

died

at

the

weekend.

Accessed

the

online

Guradian

and

IMES

Trying

to

find

details

on

Piccolo

graphics

framework.

Remind

myself

of

what

it

is

and

what

it

does.

Looking

to

build

a

GUI

within

Eclipse

I

am

trying

to

file

my

emails

regarding

IPM

and

I

am

looking

for

any

emails

from

or

about

this

journal

There

were

a

number

of

tasks

that

were

difficult

to

classify.

For

example,

consider

the

following

recorded

task:

re-find

AS"s

paper

on

graded

relevance

assessments

because

I

want

to

see

how

she

presented

her

results

for

a

paper

I

am

writing

This

task

actually

consists

of

two

sub-tasks:

item

task(refind

the

paper)

and

lookup

task

(look

for

specific

information

within

the

paper).

It

was

decided

to

treat

this

as

a

lookup

task

because

the

user"s

ultimate

goal

was

to

access

and

use

the

information

within

the

resource.

There

were

a

number

of

examples

of

combined

tasks,

mainly

of

the

form

item

then

lookup,

but

there

were

also

examples

of

item

then

multi-item.

For

example:

re-find

Kelkoo

website

so

that

I

can

re-check

the

prices

of

hair-straighteners

for

my

girlfriend

A

second

source

of

ambiguity

came

from

tasks

such

as

finding

an

email

containing

a

URL

as

a

means

of

re-accessing

a

web

page.

It

was

also

decided

to

categorise

these

as

lookup

tasks

because

in

all

cases

these

were

logged

by

participants

as

email

searches

and,

within

this

context,

what

they

were

looking

for

was

information

within

an

email.

Another

problem

was

that

some

of

the

logs

lacked

the

detail

required

to

perform

a

categorisation

e.g.

searching

for

how

to

retrieve

user"s

selection

from

a

message

box.

Decided

to

use

some

other

means

Such

tasks

were

labelled

as

U

for

unclassifiable.

To

verify

the

consistency

of

the

taxonomy,

the

tasks

were

recategorised

by

the

same

researcher

after

a

delay

of

two

weeks.

The

agreement

between

the

results

of

the

two

analyses

was

largely

consistent

Further,

we

asked

a

researcher

with

no

knowledge

of

the

project

or

the

field

to

classify

a

sample

of

tasks.

The

second

researcher

achieved

a

agreement.

We

feel

that

this

high

agreement

on

a

large

number

of

tasks

by

more

than

one

researcher

provides

evidence

for

the

reliability

of

the

classification

scheme.

The

distribution

of

task

types

is

shown

in

table

Overall,

lookup

and

item

tasks

were

the

most

common,

with

multiitem

tasks

only

representing

of

those

recorded.

The

distribution

of

the

task

types

was

different

for

web

and

email

re-finding.

The

majority

of

email

tasks

involved

looking

for

information

within

an

email

(lookup),

in

contrast

to

web

tasks

where

the

majority

of

tasks

involved

looking

for

a

single

web

page

(item).

Another

distinction

was

the

number

of

recorded

multi-item

tasks

for

web

and

email.

Multi-item

tasks

were

very

rare

for

email

re-finding

(only

of

email

tasks

involved

searching

for

multiple

resources),

but

comparatively

common

for

web

re-finding

Lookup

Item

Multi-item

Unclass.

Email

Web

All

Table

The

distribution

of

task

types

In

addition

to

the

three-way

classification

described

above,

the

recorded

tasks

were

classified

with

respect

to

the

temperature

metaphor

proposed

by

which

classifies

information

as

one

of

three

temperatures:

hot,

warm

and

cold.

We

classified

the

tasks

using

the

form

data.

Information

that

had

been

seen

less

than

a

day

or

less

than

a

week

before

the

task

were

defined

as

hot,

information

that

had

been

seen

less

than

a

month

before

the

task

as

warm,

and

information

that

had

been

seen

less

than

a

year

or

more

than

a

year

before

the

task

as

cold.

Unfortunately,

a

technical

difficulty

with

the

form

only

allowed

of

the

tasks

to

be

classified.

The

remainder

were

defined

as

U

for

unclassifiable.

A

cross-tabulation

of

task

types

and

temperatures

is

shown

in

table

Hot

Warm

Cold

Unclass.

Email

Web

All

Table

The

distribution

of

temperatures

Most

of

the

tasks

that

caused

people

to

re-find

web

pages

and

email

messages

involved

searching

for

information

that

has

been

accessed

in

the

last

week.

However

there

were

also

a

number

of

re-finding

tasks

that

involved

searching

for

older

information:

of

the

tasks

recorded

for

email

and

for

web)

involved

searching

for

information

accessed

in

the

last

month

and

of

the

tasks

recorded

for

email

and

for

web)

were

looking

for

even

older

information.

This

is

important

with

respect

to

evaluation

because

there

is

psychological

evidence

suggesting

that

people

remember

less

over

time

e.g.

This

means

that

users

may

find

searching

for

older

information

more

difficult

or

perhaps

alter

their

seeking

strategy

when

looking

for

hot,

warm

or

cold

information.

What

tasks

are

difficult?

We

looked

for

patterns

in

the

recorded

data

to

determine

if

certain

tasks

were

perceived

as

more

difficult

than

others.

For

example,

we

examined

whether

the

media

type

affected

how

difficult

the

participants

perceived

the

task

to

be.

There

was

no

evidence

that

participants

found

either

email

or

web

tasks

more

difficult.

We

also

investigated

whether

the

type

of

task

or

the

length

of

time

between

accessing

and

re-accessing

made

a

task

more

difficult.

Figure

shows

this

information

graphically.

Figure

Difficulty

ratings

for

task

types

From

figure

it

does

not

appear

that

any

particular

task

type

was

perceived

as

difficult

with

respect

to

the

others,

although

there

is

a

suggestion

that

lookup

tasks

were

perceived

more

difficult

when

looking

for

cold

information

than

hot

and

item

tasks

were

perceived

more

difficult

for

warm

information

than

hot.

To

assess

the

relationship

between

information

temperature

and

the

perceived

difficulty,

we

used

Mood"s

median

tests

to

determine

whether

the

rank

of

difficulty

scores

was

in

agreement

for

the

information

temperatures

being

compared

For

the

look-up

task

data,

there

was

evidence

that

hot

tasks

were

perceived

easier

than

cold

and

that

warm

tasks

were

perceived

easier

than

cold

but

there

was

no

evidence

to

distinguish

between

the

difficulty

ratings

of

hot

and

warm

For

the

item

task

data,

there

was

evidence

that

hot

and

cold

tasks

were

rated

differently

but

no

evidence

to

distinguish

between

hot

and

warm

or

warm

and

cold

These

tests

confirm

that

the

length

of

time

between

accessing

and

re-accessing

the

sought

after

information

indeed

influenced

how

difficult

participants

perceived

the

task

to

be.

Nevertheless,

the

large

number

of

tasks

of

all

types

and

temperatures

rated

by

participants

as

easy

i.e.

suggests

that

there

are

other

factors

that

influence

how

difficult

a

task

is

perceived

to

be.

To

learn

about

these

factors

would

require

the

kind

of

user

evaluations

proposed

by

the

kind

of

evaluations

facilitated

by

our

work.

Summary

In

the

first

part

of

this

paper,

we

described

a

diary

study

of

web

and

email

re-finding

tasks.

We

examined

the

types

of

task

that

caused

the

participants

to

search

their

personal

stores

and

found

three

main

categories

of

task:

tasks

where

the

user

requires

specific

information

from

within

a

single

resource,

tasks

where

a

single

resource

is

required,

and

tasks

that

require

information

to

be

recovered

from

multiple

resources.

It

was

discovered

that

look-up

and

item

tasks

were

recorded

with

greater

frequency

than

multi-item

tasks.

Although

no

evidence

was

found

that

web

or

email

tasks

were

more

difficult,

there

was

some

evidence

showing

that

the

time

between

accessing

and

re-accessing

affected

how

difficult

the

participants

perceived

tasks

to

be.

These

findings

have

implications

for

evaluating

PIM

behaviour

at

the

task

level.

The

remainder

of

this

paper

concentrates

on

this,

discussing

what

the

findings

mean

with

respect

to

performing

task-based

PIM

user

evaluations.

TASK-BASED

PIM

EVALUATIONS

The

findings

described

in

section

are

useful

with

respect

to

evaluation

because

they

provide

experimenters

with

enough

knowledge

to

conduct

controlled

user

evaluations

in

lab

conditions.

Greco-Latin

square

experimental

designs

can

be

constructed

where

participants

are

assigned

n

tasks

of

the

three

types

described

above

to

perform

on

their

own

collections

using

x

systems.

This

would

allow

the

performance

of

the

systems

or

the

behaviour

of

the

participants

using

different

systems

to

be

analysed

with

respect

to

the

type

of

task

being

performed

(look-up,

item,

or

multi-item).

In

the

following

sections

we

evaluate

the

feasibility

of

this

approach

when

employing

different

methods

of

task

creation.

Using

Real

Tasks

One

method

of

creating

realistic

re-finding

tasks

without

compromising

the

privacy

of

participants

is

to

use

real

tasks.

Diary-studies,

similar

to

that

described

above,

would

allow

experimenters

to

capture

a

pool

of

tasks

for

participants

to

complete

by

searching

on

their

own

collections.

This

is

extremely

advantageous

because

it

would

allow

experimenters

to

evaluate

the

behaviour

of

real

users,

completing

real

search

tasks

on

real

collections

while

in

a

controlled

environment.

There

is

also

the

additional

benefit

that

the

task

descriptions

would

not

make

any

assumptions

about

what

the

user

would

remember

in

a

real

life

situation

because

they

would

only

include

the

information

that

had

been

recorded

i.e.

the

information

that

was

available

when

the

user

originally

performed

the

task.

Nevertheless,

to

gain

these

benefits

we

must,

firstly,

confirm

that

the

task

descriptions

recorded

are

of

sufficient

quality

to

enable

the

task

to

be

re-performed

at

a

later

date.

Secondly,

we

must

ensure

that

a

diary-study

would

provide

experimenters

with

enough

tasks

to

construct

a

balanced

experimental

design

that

would

satisfy

their

data

needs.

To

examine

the

quality

of

recorded

tasks,

weeks

after

the

diary

study

had

completed,

we

asked

of

our

participants,

selected

randomly

from

the

pool

of

those

who

recorded

enough

tasks,

to

re-perform

of

their

own

tasks.

The

tasks

were

selected

randomly

from

the

pool

of

those

available.

The

issued

tasks

consisted

of

email

and

web

tasks,

of

which

were

lookup

tasks,

were

item

tasks,

and

were

multi-item

tasks.

The

issued

tasks

represented

a

broad-sampling

of

the

complete

set

of

recorded

tasks.

They

also

included

tasks

with

vague

descriptions

e.g.

a

software

key

for

an

application

I

required

to

reinstall.

to

find

a

quote

to

use

in

a

paper.

Cannot

remember

the

person

or

the

exact

quote

The

usefulness

of

such

tasks

would

rely

on

the

memories

of

participants

i.e.

would

the

recorder

of

task

remember

which

application

he

referred

to

and

would

the

recorder

of

remember

enough

about

the

context

in

which

the

task

took

place

to

re-perform

the

task?

Presented

with

the

tasks

exactly

as

they

recorded

them,

the

participants

were

asked

to

re-perform

each

task

with

any

system

of

their

choice.

Of

the

tasks

issued,

were

completed

without

problems,

of

the

tasks

were

not

completed

because

the

description

recorded

was

insufficent

to

recreate

the

task,

and

tasks

were

not

completed

because

the

task

was

too

difficult

or

the

required

web

page

no

longer

existed.

Experimenters

are

likely

to

be

interested

in

the

final

group

of

tasks

because

it

is

important

to

discover

what

makes

a

task

difficult

and

how

user

behaviour

changes

in

these

circumstances.

Therefore,

from

the

tasks

tested,

only

tasks

were

not

of

sufficient

quality

to

be

used

in

an

evaluation

situation.

Further,

there

did

not

seem

to

be

any

issue

of

the

type,

temperature

or

difficulty

ratings

affecting

the

quality

of

the

task

descriptions.

These

findings

suggest

that

the

participants

who

recorded

most

tasks

in

the

diary

study

also

recorded

tasks

with

sufficient

quality.

However,

did

the

diary

study

generate

enough

tasks

to

satisfy

the

needs

of

experimenters?

Participant

Tasks

Lookup

Item

Multi-item

Unclass.

Table

The

quantities

of

recorded

email

tasks

Participant

Tasks

Lookup

Item

Multi-item

Unclass.

Table

The

quantities

of

recorded

web

tasks

Naturally

the

exact

number

of

tasks

required

to

perform

a

user

evaluation

will

depend

on

the

goals

of

the

evaluation,

the

number

of

users

and

the

number

of

systems

to

be

tested

etc.

However,

for

illustrative

purposes

we

chose

tasks

as

a

cut-off

point

for

our

data.

From

tables

and

which

show

the

quantities

of

email

and

web

tasks

recorded

for

each

participant,

we

can

see

that

of

the

participants,

only

recorded

or

more

email

tasks

and

recorded

or

more

web

tasks.

This

means

that

many

of

the

recruited

participants

could

not

actually

participate

in

the

final

evaluation.

This

is

a

major

limitation

of

using

recorded

tasks

in

evaluations

because

participant

recruitment

for

user

tests

is

challenging

and

it

may

not

be

possible

to

recruit

enough

participants

if

experimenters

lose

between

half

and

two-thirds

of

their

populations.

Further,

there

was

some

imbalance

in

the

numbers

of

recorded

tasks

of

different

types.

Some

participants

recorded

several

lookup

tasks

but

very

few

item

tasks

and

others

recorded

several

item

tasks

but

few

lookup

tasks.

There

was

also

a

specific

lack

of

multi-item

email

tasks.

This

situation

makes

it

very

difficult

for

experimenters

to

prepare

balanced

experimental

designs.

Therefore,

even

though

our

first

test

suggests

that

the

quality

of

recorded

tasks

was

sufficient

for

the

participants

to

re-perform

the

tasks

at

a

later

stage,

the

number

of

tasks

recorded

was

probably

too

low

to

make

this

a

viable

option

for

experimental

task

creation.

However,

it

may

be

possible

to

increase

the

number

of

tasks

recorded

by

frequently

reminding

participants

or

by

making

personal

visits

etc.

Using

Simulated

Tasks

Based

on

Real

Tasks

Another

benefit

of

diary-studies

is

that

they

provide

information

about

the

contents

and

uses

of

private

collections

without

invading

participants"

privacy.

This

section

explores

the

possibility

of

using

a

combination

of

the

knowledge

gained

from

diary

studies

and

other

attributes

known

about

participants

to

artificially

create

re-finding

tasks

corresponding

to

the

taxonomy

defined

in

section

We

explain

the

techniques

used

and

demonstrate

the

feasibility

of

creating

simulated

tasks

within

the

context

of

a

user

evaluation

investigating

email

re-finding

behaviour.

Space

limitations

prevent

us

from

reporting

our

findings;

instead

we

concentrate

on

the

methods

of

task

creation.

As

preparation

for

the

evaluation,

we

performed

a

second

diary-study,

where

new

participants,

consisting

of

post-graduate

students

and

under-graduate

students,

recorded

email

tasks

over

a

period

of

approximately

weeks.

The

collected

data

revealed

several

patterns

that

helped

with

the

creation

of

artificial

tasks.

For

example,

students

in

both

groups

recorded

tasks

relating

to

classes

that

they

were

taking

at

the

time

and

often

different

participants

recorded

tasks

that

involved

searching

for

the

same

information.

This

was

useful

because

it

provided

us

with

a

clue

that

even

though

some

of

the

participants

did

not

record

a

particular

task,

it

was

possible

that

the

task

may

still

be

applicable

to

their

collections.

Other

patterns

revealed

included

that

students

within

the

same

group

often

searched

for

emails

containing

announcements

from

the

same

source.

For

example,

several

undergraduate

students

recorded

tasks

that

included

re-finding

information

relating

to

job

vacancies.

There

were

also

tasks

that

were

recorded

by

participants

in

both

groups.

For

example,

searching

for

an

email

that

would

re-confirm

the

pin

code

required

to

access

the

computer

labs.

To

supplement

our

knowledge

of

the

participants"

email

collections,

we

asked

participants

from

each

group

to

provide

email

tours.

These

consisted

of

short

minute

sessions,

where

participants

were

asked

to

explain

why

they

use

email,

who

sends

them

email,

and

their

organisational

strategies.

This

approach

has

been

used

successfully

in

the

past

as

a

non-intrusive

means

to

learn

about

how

people

store

and

maintain

their

personal

information

Originally,

we

had

planned

to

ask

more

participants

to

provide

tours,

but

we

found

tours

per

group

was

sufficient

for

our

needs.

Again,

patterns

emerged

that

helped

with

task

creation.

We

found

content

overlap

within

and

between

groups

that

confirmed

many

of

our

observations

from

the

diary

study

data.

For

example,

the

students

who

gave

tours

revealed

that

they

received

emails

from

lecturers

for

particular

class

assignments,

receipts

for

completed

assignments,

and

various

announcements

from

systems

support

and

about

job

vacancies.

Importantly,

the

participants

were

also

able

to

confirm

which

other

students

had

received

the

same

information.

This

confirmed

that

many

of

tasks

recorded

during

the

diary

study

were

applicable,

not

only

to

the

recorder,

but

to

every

participant

in

or

both

groups.

Based

on

this

initial

investigatory

work,

a

set

of

tasks

of

each

type

in

our

taxonomy)

was

created

for

each

group

of

participants.

We

also

created

a

set

of

tasks

for

a

third

group

of

participants

that

consisted

of

research

and

academic

staff

members,

based

on

our

knowledge

of

the

emails

our

colleagues

receive.

Where

possible

we

used

the

information

recorded

in

the

diary

study

descriptions

to

provide

a

context

for

the

task

i.e.

a

work

task

or

motivation

that

would

require

the

task

to

be

performed.

When

the

diary

study

data

did

not

provide

sufficient

context

information

to

supply

the

participants

with

a

robust

description

of

the

information

need,

we

created

simulated

work

task

situations

according

to

the

guidelines

of

A

further

advantage

of

using

simulated

tasks

in

this

way,

rather

than

real-tasks,

is

that

some

of

the

users

will

not

have

performed

the

task

in

the

recent

past

and

this

allows

the

examination

of

tasks

that

look

for

information

of

different

temperatures.

If

only

real-tasks

had

been

used

all

of

the

participants

would

have

performed

the

tasks

during

the

period

of

the

diary

study.

The

created

tasks

were

used

in

a

final

evaluation,

where

we

examined

the

email

re-finding

behaviour

of

users

with

three

different

email

systems.

users

in

each

group)

performed

tasks

each

task

of

each

type

on

each

system)

using

their

own

personal

collections

in

a

Greco-Latin

square

experimental

design.

Performing

a

PIM

evaluation

in

this

way

allowed

the

examination

of

re-finding

behaviour

in

a

way

not

possible

before

we

were

able

to

observe

the

email

re-finding

strategies

employed

by

real

users,

performing

realistic

tasks,

on

their

own

collections

in

a

controlled

environment.

The

study

revealed

that

the

participants

remembered

different

attributes

of

emails,

demostrated

different

finding

behaviour,

and

exhibited

different

levels

of

performance

when

asked

to

complete

tasks

of

the

different

types

in

the

taxonomy.

The

key

to

both

the

task

creation

and

the

analysis

of

the

results

was

our

taxonomy,

which

provided

the

template

to

create

tasks

and

also

a

means

to

compare

the

behaviour

and

performance

of

different

users

(and

systems)

performing

different

tasks

of

the

same

type.

Some

of

the

findings

of

the

evaluation

will

be

published

in

Summarising

the

approach,

to

conduct

a

user

experiment

using

our

methodology,

researchers

would

be

required

to

perform

the

following

steps:

a

diary

study

as

above

the

recorded

tasks

looking

for

overlap

between

the

participants.

the

gained

knowledge

about

the

contents

of

participants"

collections

by

asking

a

selection

of

the

participants

to

provide

a

tour

of

their

collection.

the

knowledge

gained

to

devise

tasks

of

the

three

different

types

defined

within

the

taxonomy.

More

Information

about

this

and

the

diary

forms

required

can

be

found

at

tailed

information

on

how

to

use

the

research

described

in

this

paper

to

perform

task-based

PIM

evaluations

can

be

found

at

our

website

(see

footnote

CONCLUSIONS

This

paper

has

focused

on

overcoming

the

difficulties

involved

in

performing

PIM

evaluations.

The

personal

nature

of

PIM

means

that

it

is

difficult

to

construct

balanced

experiments

because

participants

each

have

their

own

unique

collections

that

are

self-generated

by

completing

other

tasks.

We

suggested

that

to

incorporate

the

personal

aspects

of

PIM

in

evaluations,

the

performance

of

systems

or

users

should

be

examined

when

users

complete

tasks

on

their

own

collections.

This

approach

itself

has

problems

because

task

creation

for

personal

collections

is

difficult:

researchers

don"t

know

much

about

the

kinds

of

re-finding

tasks

people

perform

and

they

don"t

know

what

information

is

within

individual

personal

collections.

In

this

paper

we

described

ways

of

overcoming

these

challenges

to

facilitate

task

based

PIM

user

evaluations.

In

the

first

part

of

the

paper

we

performed

a

diary

study

that

examined

the

tasks

that

caused

people

to

re-find

email

messages

and

web

pages.

The

collected

data

included

a

wide

range

of

both

work

and

non-work

related

tasks,

and

based

on

the

data

we

created

a

taxonomy

of

web

and

email

re-finding

tasks.

We

discovered

that

people

perform

three

main

types

of

re-finding

task:

tasks

that

require

specific

information

from

within

a

single

resource,

tasks

that

require

a

single

complete

resource,

and

tasks

that

require

information

to

be

recovered

from

multiple

resources.

In

the

second

part

of

the

paper,

we

discussed

the

significance

of

the

taxonomy

with

respect

to

PIM

evaluation.

We

demonstrated

that

balanced

experiments

could

be

conducted

comparing

system

or

user

performance

on

the

task

categories

within

the

taxonomy.

We

also

suggested

two

methods

of

creating

tasks

that

can

be

completed

on

personal

collections.

These

methods

do

not

compromise

the

privacy

of

study

participants.

We

examined

the

techniques

suggested,

firstly

by

simulating

an

experimental

situation

participants

were

asked

to

re-perform

their

own

tasks

as

they

recorded

them,

and

secondly,

in

the

context

of

a

full

evaluation.

Performing

evaluations

in

this

way

will

allow

systems

that

have

been

proposed

to

improve

users"

ability

to

manage

and

re-find

their

information

to

be

tested,

so

that

we

can

learn

about

the

needs

and

desires

of

users.

Thus,

this

paper

has

offered

two

contributions

to

the

field:

an

increased

understanding

of

PIM

behaviour

at

the

task

level

and

an

evaluation

method

that

will

facilitate

further

investigations.

ACKNOWLEDGMENTS

We

would

like

to

thank

Dr

Mark

Baillie

for

his

insightful

comments

and

help

analysing

the

data.

REFERENCES

R.

Boardman,

Improving

tool

support

for

personal

information

management,

Ph.D.

thesis,

Imperial

College

London,

P.

Borlund,

The

iir

evaluation

model:

A

framework

for

evaluation

of

interactive

information

retrieval

systems,

Information

Research

no.

paper

no.

K.

Bystr¨om

and

K.

J¨arvelin,

Task

complexity

affects

information

seeking

and

use,

Information

Processing

and

Management

no.

R.

G.

Capra

and

M.

A.

Perez-Quinones,

Re-finding

found

things:

An

exploratory

study

of

how

users

re-find

information,

Tech.

report,

Virginia

Tech,

R.

G.

Capra

and

M.

A.

Perez-Quinones,

Using

web

search

engines

to

find

and

refind

information,

Computer

no.

R.

G.

Capra

and

M.

A.

Perez-Quinones,

Factors

and

evaluation

of

refinding

behaviors.,

SIGIR

Workshop

on

Personal

Information

Management,

August

Seattle,

Washington,

E.

Cutrell,

D.Robbins,

S.Dumais,

and

R.Sarin,

Fast,

flexible

filtering

with

phlat,

Proc.

SIGCHI

(New

York,

NY,

USA),

ACM

Press,

pp.

M.

Czerwinski,

E.

Horvitz,

and

S.

Wilhite,

A

diary

study

of

task

switching

and

interruptions,

Proc.

SIGCHI

pp.

S.

Dumais,

E.

Cutrell,

J.

Cadiz,

G.

Jancke,

R.

Sarin,

and

D.C.

Robbins,

Stuff

i"ve

seen:

a

system

for

personal

information

retrieval

and

re-use,

Proc.

SIGIR

pp.

D.

Elsweiler

and

I.

Ruthven,

Memory

and

email

re-finding,

In

preparation

for

ACM

TOIS

CFP

special

issue

on

Keeping,

Re-finding,

and

Sharing

Personal

Information

D.

Elsweiler,

I.

Ruthven,

and

C.

Jones,

Dealing

with

fragmented

recollection

of

context

in

information

management,

Context-Based

Information

Retrieval

Workshop

in

D.

Elsweiler,

I.

Ruthven,

and

C.

Jones,

Towards

memory

supporting

personal

information

management

tools,

(to

appear

in)

Journal

of

the

American

Society

for

Information

Science

and

Technology

D.

Harman,

What

we

have

learned,

and

not

learned,

from

trec,

Proc.

ECIR

P.

Ingwersen,

Information

retrieval

interaction,

Taylor

Graham,

D.

Kelly,

B.

Bederson,

M.

Czerwinski,

J.

Gemmell,

W.

Pratt,

and

M.

Skeels

(eds.),

Pim

workshop

report:

Measurement

and

design,

D.

Kelly

and

J.

Teevan,

(to

appear

in)

personal

information

management,

ch.

Understanding

what

works:

Evaluating

personal

information

management

tools,

Seattle:

University

of

Washington

Press.,

B.

H.

Kwasnik,

How

a

personal

document"s

intended

use

or

purpose

affects

its

classification

in

an

office,

no.

SI,

M.W.

Lansdale,

The

psychology

of

personal

information

management.,

Appl

Ergon

no.

L.

Palen

and

M.

Salzman,

Voice-mail

diary

studies

for

naturalistic

data

capture

under

mobile

conditions,

CSCW

Proceedings

of

the

ACM

conference

on

Computer

supported

cooperative

work,

M.

Ringel,

E.

Cutrell,

S.

Dumais,

and

E.

Horvitz,

Milestones

in

time:

The

value

of

landmarks

in

retrieving

information

from

personal

stores.,

Proc.

INTERACT

G.

Robertson,

M.

Czerwinski,

K.

Larson,

D.

C.

Robbins,

D.

Thiel,

and

M.

van

Dantzich,

Data

mountain:

using

spatial

memory

for

document

management,

Proc.

UIST

K.

Rodden,

How

do

people

organise

their

photographs,

BCS

IRSG

Annual

Colloquium

on

Information

Retrieval

Research,Glasgow,

Scotland,

D.C.

Rubin

and

A.E.

Wenzel,

One

hundred

years

of

forgetting:

A

quantitative

description

of

retention,

Psychological

Bulletin

A.

J.

Sellen

and

R.

H.

R.

Harper,

The

myth

of

the

paperless

office,

MIT

Press,

Cambridge,

MA,

USA,

P.

Vakkari,

Task

complexity,

problem

structure

and

information

actions:

Integrating

studies

in

on

information

seeking

and

retrieval.,

Information

Processing

and

Management

P.

Vakkari,

A

theory

of

task-based

information

retrieval,

Journal

of

Documentation

no.

O O B I I O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I I O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

human factor
re-find information
privacy issue
individual collection
naturalistic approach
laboratory-based study

personal information management
measurement
experimenter
taxonomy
email message
user evaluation