Continuous

Media

in

Discrete

Objects

Multimedia

for

Computer

Supported

Collaborative

Work

John

Riedl

Vahid

Mashayekhi

Computer

Science

Department

University

of

Minnesota

Minneapolis

MN

-LRB-

-RRB-

October

We

gratefully

acknowledge

the

support

of

the

National

Science

Foundation

-LRB-

grant

number

NSF

IRI

-RRB-

and

the

research

funds

of

the

Graduate

School

of

the

University

of

Minnesota

Contents

Introduction

Previous

work

Overview

Design

Considerations

Continuous

Media

in

Discrete

Objects

Floor

Control

Speech

Chunk

Detection

High-level

tools

for

SuiteSound

Suite

Distributed

persistent

objects

Filters

High-level

constructs

for

manipulating

flows

Sound

as

a

Suite

base

object

Coupling

with

other

users

Design

and

Use

of

SuiteSound

Sound

on

the

Sun

SparcStation

Recording

Playing

Sharing

the

Audio

Device

Experience

Teleconf

Annotator

CSI

Conclusions

and

Future

Work

Problems

and

Advantages

of

Continuous

Sound

in

Discrete

Objects

Future

work

Experiments

Enhancements

and

extensions

Continuous

Media

in

Discrete

Objects

Multimedia

for

Computer

Supported

Collaborative

Work

Abstract

Multimedia

has

the

same

potential

to

revolutionize

human-computer

interfaces

that

bitmapped

workstations

have

realized

over

the

last

decade

Achieving

this

potential

requires

the

development

of

programming

environments

with

integrated

support

for

multimedia

SuiteSound

is

one

such

environment

SuiteSound

is

built

in

the

Suite

object-based

system

on

a

conventional

UNIX

operating

system

SuiteSound

objects

incorporate

multimedia

by

creating

flows

and

filters

Flows

are

streams

of

multimedia

data

moving

through

a

sequence

of

objects

They

bridge

the

gap

between

objects

representing

the

state

of

an

entity

at

a

discrete

point

in

time

and

space

and

continuous

media

such

as

live

audio

or

video

Filters

are

intermediate

objects

between

the

source

and

destination

of

a

flow

They

take

a

flow

as

input

perform

one

of

several

operations

such

as

multiplex-in

multiplex-out

gain

control

record

or

silence

detection

on

it

and

send

the

resulting

flow

to

its

destination

In

effect

they

provide

a

virtual

device

interface

for

the

application

programmer

that

is

uniform

and

independent

of

any

physical

device

involved

-LSB-

AHL

-RSB-

This

paper

describes

the

design

and

implementation

of

SuiteSound

on

the

Sun

SparcStation

Three

SuiteSound

applications

are

described

in

detail

a

teleconferencing

tool

a

voice

annotator

and

a

collaborative

software

inspection

tool

Experiences

using

the

SuiteSound

environment

and

applications

are

detailed

a

retrospective

of

the

difficulties

and

benefits

of

including

multimedia

in

an

object-based

system

is

given

and

suggestions

for

future

work

are

included

Introduction

Computer

Supported

Collaborative

Work

-LRB-

CSCW

-RRB-

is

the

study

of

methods

for

enhancing

cooperation

among

computer

users

by

providing

tools

that

explicitly

support

the

user

interaction

and

sharing

of

information

-LSB-

-RSB-

Suite

is

an

object-based

system

with

powerful

tools

for

developing

distributed

collaborative

applications

-LSB-

-RSB-

This

paper

describes

SuiteSound

a

tool

for

manipulating

multimedia

flows

such

as

for

digital

audio

in

Suite

Adding

digital

audio

to

Suite

enhances

distributed

collaboration

by

helping

participants

work

together

more

naturally

Since

SuiteSound

is

integrated

with

Suite

applications

can

use

sound

in

the

way

that

is

most

natural

for

them

For

instance

Collaborative

Software

Inspection

-LRB-

CSI

-RRB-

-LSB-

-RSB-

is

a

tool

that

supports

distributed

software

inspection

using

Suite

for

the

software

modules

being

inspected

and

SuiteSound

for

audio

-LRB-

section

-RRB-

CSI

offers

audio

teleconferencing

for

participants

to

discuss

the

documents

during

the

distributed

inspection

meeting

and

audio

annotations

for

participants

to

make

comments

on

the

documents

while

preparing

for

the

meeting

There

are

many

applications

for

multimedia

flows

in

an

object-oriented

environment

Digital

teleconferencing

has

already

proven

to

be

an

effective

way

to

hold

meetings

without

traveling

-LSB-

-RSB-

Multimedia

electronic

mail

systems

such

as

Diamond

-LSB-

-RSB-

ease

communication

between

participants

especially

those

who

are

not

used

to

using

computers

At

Lawrence

Livermore

Laboratories

researchers

are

building

a

collaborative

notebook

for

physical

scientists

The

scientists

will

be

able

to

incorporate

still

images

live

audio

and

even

live

video

in

their

notebooks

and

readily

share

the

notes

with

other

researchers

-LSB-

FCB

-RSB-

Distributed

education

is

also

possible

with

students

at

remote

sites

seeing

video

images

of

the

instructor

and

participating

in

the

class

through

sound

transmitted

back

to

the

classroom

For

instance

collaborative

visualization

tools

will

allow

the

students

to

interactively

explore

algorithms

with

their

instructors

-LSB-

-RSB-

Digital

audio

will

enable

instructors

to

respond

to

questions

verbally

rather

than

having

to

type

replies

SuiteSound

supports

development

of

applications

like

these

by

integrating

flexible

easyto-use

digital

audio

with

a

flexible

easy-to-use

object-based

system

SuiteSound

applications

readily

combine

live

or

recorded

audio

with

editable

representations

of

user

objects

This

paper

describes

the

design

implementation

and

early

experience

with

SuiteSound

along

with

some

sample

applications

The

SuiteSound

software

is

available

for

others

who

would

like

to

experiment

with

multimedia

flows

in

an

object-based

system

Purdue

University

gives

away

the

Suite

system

along

with

The

University

of

Minnesota

gives

away

SuiteSound

to

sites

that

already

have

Previous

work

Researchers

have

developed

authoring

systems

that

provide

multimedia

support

in

creating

and

editing

of

multimedia

documents

The

MAEstro

system

provides

a

distributed

multimedia

authoring

system

where

the

user

can

create

multimedia

documents

through

the

use

Dewan

Computer

Science

Department

Purdue

University

pd@cs.purdue.edu

Riedl

Computer

Science

Department

University

of

Minnesota

riedl@cs.umn.edu

of

media

editors

The

MAEstro

applications

run

on

Sun

and

NeXT

workstations

with

combined

support

for

editing

of

text

CD

audio

video

discs

and

MIDI

music

-LSB-

-RSB-

MediaView

is

an

editable

multimedia

publication

system

that

supports

the

cut

copy

and

paste

operations

to

text

audio

images

and

video

It

is

developed

on

a

NeXT

computer

using

the

object-oriented

development

system

called

NeXTstep

-LSB-

-RSB-

The

Diamond

multimedia

message

system

supports

creating

editing

transmitting

printing

and

managing

multimedia

documents

A

document

may

contain

various

media

such

as

text

graphics

images

and

speech

-LSB-

-RSB-

Systems

such

as

MAEstro

MediaView

and

Diamond

provide

a

media-rich

environment

for

the

users

to

develop

documents

with

components

like

text

audio

graphics

color

images

and

live

video

Other

researchers

have

experimented

with

distributed

digital

audio

PhoneTalk

and

Vtalk

are

multi-user

systems

that

run

on

Sun

Sparc

Stations

for

the

specific

purpose

of

teleconferencing

-LSB-

-RSB-

The

Etherphone

system

supports

locally

distributed

computing

environments

with

multiple

workstations

and

multiple

networks

In

the

Etherphone

system

each

workstation

is

associated

with

an

Etherphone

that

digitizes

packetizes

and

encrypts

the

voice

and

transmits

it

over

the

Ethernet

The

voice

manager

in

the

Etherphone

system

provides

facilities

for

recording

editing

and

playing

stored

voice

-LSB-

-RSB-

Still

other

systems

have

applied

digital

audio

to

collaboration

and

decision-making

For

instance

the

MERMAID

system

is

designed

for

collaborative

decision-making

with

multiple

remote

participants

-LSB-

-RSB-

Data

voice

and

video

are

transmitted

using

N-ISDN

channels

with

per

channel

The

voice

and

data

use

one

B-channel

and

video

the

other

Users

can

talk

freely

with

voices

mixed

and

transmitted

to

all

users

simultaneously

In

Mermaid

synchronization

routing

and

distribution

are

done

separately

for

voice

video

and

data

This

allows

all

three

to

run

at

higher

speeds

but

makes

the

synchronization

of

the

media

more

difficult

Researchers

are

also

focusing

on

developing

sharable

audio

servers

for

supporting

multimedia

applications

-LSB-

AHL

-RSB-

describes

a

client-server

model

that

uses

an

audio

server

for

supporting

shared

access

to

audio

hardware

introduces

a

protocol

that

provides

applications

with

a

device-independent

interface

to

the

audio

capabilities

of

the

workstation

-LRB-

virtual

devices

-RRB-

and

provides

primitives

for

synchronizing

the

various

media

components

-LSB-

-RSB-

describes

an

extension

to

the

X

Window

System

to

support

audio

The

modified

server

accepts

audio

requests

from

across

the

network

and

executes

them

on

Sun

SparcStations

Overview

Our

work

extends

previous

research

in

four

ways

First

SuiteSound

provides

an

objectoriented

interface

for

building

applications

that

integrate

multimedia

with

traditional

objects

promoting

simplicity

and

re-usability

Second

our

work

supports

multimedia

flows

with

multiple

sources

and

multiple

sinks

and

many

different

ways

to

merge

or

split

the

flows

We

will

design

and

develop

a

high-level

specification

language

for

the

application

programmer

to

describe

the

type

of

filtering

operation

he

or

she

wishes

to

perform

on

a

flow

Third

we

are

investigating

methods

of

concurrency

control

that

allow

end-users

to

effectively

control

multiple

multimedia

streams

at

their

workstations

Fourth

we

are

performing

experiments

to

measure

the

costs

of

different

implementation

strategies

for

multimedia

systems

on

conventional

operating

systems

Section

describes

issues

in

incorporating

multimedia

in

an

object-based

system

on

a

conventional

operating

system

Many

of

these

ideas

are

independent

of

the

particular

system

we

have

chosen

for

our

implementation

Section

discusses

the

design

and

implementation

of

SuiteSound

SuiteSound

demonstrates

several

ways

in

which

asynchronous

behavior

can

be

effectively

incorporated

in

an

object-based

system

Section

describes

our

experience

with

three

applications

Teleconf

Annotator

and

CSI

We

also

talk

about

several

simple

pilot

tests

we

have

performed

to

understand

the

strengths

and

weaknesses

of

SuiteSound

Finally

section

summarizes

our

experience

incorporating

multimedia

flows

in

a

conventional

operating

system

discusses

some

of

the

lessons

we

have

learned

and

proposes

future

work

Design

Considerations

Continuous

Media

in

Discrete

Objects

Object-based

systems

are

being

used

for

many

tasks

because

of

the

benefits

of

improved

encapsulation

-LSB-

-RSB-

However

objects

are

inherently

discrete

conceptually

containing

the

representation

of

the

state

of

a

system

at

a

fixed

point

in

space

and

time

Continuous

media

such

as

voice

and

video

flow

across

time

with

no

convenient

boundaries

for

object

representation

Objects

can

be

used

to

represent

discrete

parts

of

the

flow

but

do

not

readily

represent

the

entire

flow

SuiteSound

breaks

continuous

media

into

chunks

Rather

than

using

individual

objects

to

represent

these

chunks

individual

objects

represent

points

in

space

past

which

the

chunks

must

flow

over

time

In

SuiteSound

continuous

media

is

processed

through

flows

and

filters

Flows

represent

connections

between

objects

through

which

continuous

media

can

move

from

a

source

to

a

destination

Filters

are

objects

that

take

flows

as

input

process

the

flow

and

produce

a

modified

flow

as

output

SuiteSound

provides

a

virtual

device

interface

on

top

of

physical

audio

devices

Filters

modify

the

functionality

of

the

virtual

device

but

provide

the

same

interface

so

additional

filters

can

be

applied

to

the

modified

flow

Flows

carry

data

from

the

output

of

one

virtual

device

to

the

input

of

another

Programmers

build

SuiteSound

applications

by

instantiating

filters

and

joining

them

with

flows

much

as

Unix

programmers

create

new

tools

from

Unix

utilities

by

joining

them

with

pipes

Once

a

flow

is

established

it

moves

automatically

through

the

filter

The

permanent

state

of

the

object

never

represents

more

than

a

brief

interlude

-LRB-

usually

to

second

-RRB-

in

the

continuous

flow

of

the

media

Source

and

sink

objects

can

operate

either

asynchronously

performing

callbacks

to

other

objects

at

physical

device

speeds

or

synchronously

transferring

data

to

other

objects

at

their

request

Generally

a

flow

has

either

an

asynchronous

source

or

an

asynchronous

sink

driving

the

flow

If

the

source

is

asynchronous

it

pushes

data

through

the

flow

towards

the

sink

If

the

sink

is

asynchronous

it

pulls

data

through

the

flow

from

the

source

For

instance

the

Teleconf

object

in

floor

control

mode

uses

a

real-time

source

with

sinks

accepting

data

passively

Conversely

the

Annotator

object

when

playing

uses

a

real-time

source

to

pull

the

data

at

the

correct

rate

There

are

several

unusual

flow

types

that

do

not

fit

this

model

For

instance

copying

data

between

two

Annotator

objects

can

be

done

with

a

non-real-time

flow

Also

some

applications

have

both

real-time

sources

and

sinks

and

must

flow-control

between

them

-LRB-

see

section

-RRB-

SuiteSound

provides

a

mechanism

for

creating

flows

but

does

not

restrict

the

user

to

a

particular

way

of

using

them

Floor

Control

Floor

control

in

a

collaborative

system

comprises

a

set

of

policies

to

control

access

to

the

set

of

shared

objects

Floor

control

policies

usually

permit

or

deny

access

to

all

the

objects

in

the

collaboration

simultaneously

conceptually

controlling

the

entire

floor

at

once

More

general

concurrency

control

policies

separate

the

objects

into

domains

that

can

be

accessed

individually

-LSB-

-RSB-

There

are

three

reasons

to

use

floor

control

for

on-line

sound

listener

overload

bandwidth

and

more

effective

conferencing

Listener

overload

refers

to

the

difficulty

in

understanding

multiple

sound

sources

simultaneously

When

visual

cues

are

unavailable

to

help

decide

who

should

speak

next

it

is

common

for

two

speakers

to

begin

at

the

same

time

With

no

stereo

separation

to

help

locate

the

speakers

the

listener

has

a

difficult

time

interpreting

the

intermingled

voice

Floor

control

can

replace

the

visual

cues

with

explicit

requests

to

speak

Bandwidth

is

an

issue

because

live

sound

runs

at

relatively

high

data

rates

Having

open

channels

between

all

pairs

of

participants

is

very

expensive

in

computing

and

networking

resources

Floor

control

can

reduce

the

requirements

from

open

channels

to

n

open

channels

Finally

floor

control

can

actively

support

more

effective

interaction

Systems

such

as

Colab

-LSB-

SFB

-RSB-

have

demonstrated

the

benefits

of

explicit

support

for

decision

making

systems

Floor

control

tactics

can

ensure

that

everyone

gets

a

turn

to

speak

or

can

allow

a

meeting

chair

to

directly

control

the

meeting

There

are

several

interesting

types

of

floor

control

for

on-line

audio

FIFO

The

floor

is

passed

according

to

a

queue

of

requests

in

a

First-In-First-Out

manner

The

advantage

of

this

mode

is

its

fairness

Everybody

gets

an

equal

chance

to

hold

the

floor

Moderated

In

this

mode

one

participant

has

clear

authority

over

the

meeting

and

other

participants

and

chooses

the

next

floor-holder

from

an

internal

queue

that

is

only

visible

to

him/her

The

moderator

may

wish

to

take

the

floor

back

at

any

time

to

give

to

another

user

The

floor

always

goes

back

to

the

moderator

after

a

floor-holder

is

done

speaking

Free

In

this

mode

a

number

of

participants

may

speak

simultaneously

and

the

result

of

their

contention

is

unpredictable

We

may

have

the

voice

come

out

garbled

at

the

other

end

FIFO

with

priorities

A

variation

of

FIFO

that

allows

for

placement

on

the

queue

to

be

done

according

to

a

pre-determined

scheme

For

instance

the

vice

president

would

get

a

chance

to

speak

before

the

student

employee

if

both

had

a

comment

Baton

The

floor

holder

decides

who

gets

the

control

next

The

disadvantage

of

this

mode

is

the

subjective

nature

of

this

choice

as

two

or

more

participants

may

monopolize

the

floor

leading

to

starvation

for

some

participants

Speech

Chunk

Detection

Many

of

the

floor

control

modes

can

be

simplified

for

users

by

automatically

requesting

the

floor

when

speech

is

detected

and

automatically

releasing

the

floor

when

a

silent

period

is

detected

For

instance

FIFO

mode

could

be

implemented

with

no

user

interaction

Floor

request

would

be

automatically

made

when

sound

is

detected

and

conversely

released

when

no

sound

Information

about

silent

periods

could

also

be

used

as

feedback

to

the

user

For

instance

a

floor-holder

who

has

been

silent

for

a

few

seconds

could

hear

a

beep

on

his

workstation

to

remind

him

to

release

the

floor

There

are

many

possible

algorithms

for

determining

non-silent

periods

Most

are

based

on

watching

the

linear

audio

signal

for

sections

of

pre-determined

length

with

non-zero

amplitude

We

call

non-silent

periods

speech

chunks

We

are

investigating

techniques

based

on

exponential

averaging

of

the

linear

signal

with

different

parameters

for

the

averaging

factor

and

the

length

of

the

non-zero

signal

required

to

indicate

speech

High-level

tools

for

SuiteSound

In

this

sub-section

we

introduce

Suite

briefly

describe

the

new

design

of

high-level

tools

for

SuiteSound

including

a

system

for

specifying

flows

and

the

integration

of

SuiteSound

objects

with

the

Suite

generalized

editing

model

We

also

describe

the

experiments

needed

to

measure

the

effectiveness

and

the

load

that

these

extensions

will

put

on

networks

and

workstations

Suite

Distributed

persistent

objects

Suite

is

a

software

system

designed

and

developed

at

Purdue

University

by

Prasun

Dewan

and

his

research

team

-LSB-

-RSB-

Suite

supports

remote

procedure

calls

-LRB-

RPC

-RRB-

active

persistent

data

and

management

of

direct

manipulation

user

interfaces

A

prototype

of

Suite

has

been

implemented

on

top

of

UNIX

TCP/IP

NFS

and

X

-LSB-

-RSB-

The

Suite

object

model

is

an

extension

of

UNIX

allowing

distributed

shared

protected

and

persistent

objects

Suite

objects

coexist

with

existing

components

of

the

operating

system

access

conventional

resources

and

can

be

invoked

by

conventional

processes

-LSB-

-RSB-

Filters

High-level

constructs

for

manipulating

flows

The

new

design

for

SuiteSound

includes

a

higher-level

specification

system

for

connecting

filters

Currently

the

filters

are

specified

procedurally

and

the

SuiteSound

program

includes

code

for

moving

the

messages

from

the

source

to

the

sink

The

higher-level

specification

system

is

based

on

a

library

of

filters

that

are

used

repeatedly

in

SuiteSound

applications

multiplex-in

Merge

several

input

flows

into

a

single

output

flow

by

adding

the

sounds

together

The

merged

sounds

are

synchronized

by

timestamps

on

each

second

-LRB-

Kbits

-RRB-

of

data

multiplex-out

Split

a

single

input

flow

into

multiple

output

flows

by

making

multiple

copies

of

the

input

discard

silent

periods

Remove

silent

periods

from

a

flow

This

filter

reproduces

a

flow

with

silent

periods

longer

than

a

second

removed

Timestamps

are

used

to

reproduce

the

flow

on

the

receiving

end

gain

control

Change

the

loudness

of

a

flow

Volume

control

filters

are

often

used

just

before

a

multiplex

in

to

adjust

the

relative

volumes

of

the

inputs

record

Record

sound

for

later

playback

Record

filters

buffer

sound

in

memory

but

store

larger

segments

on

disk

A

library

of

these

filters

is

built

and

a

registration

service

enables

users

to

add

filters

to

a

flow

by

creating

an

instance

of

the

filter

and

specifying

its

endpoints

in

the

flow

Filters

work

the

same

way

they

do

in

the

current

implementation

but

the

library

filters

are

written

to

be

general-purpose

so

programmers

can

easily

reuse

filters

in

new

applications

For

instance

Teleconf

could

be

implemented

using

the

library

in

two

steps

Create

multiplex-out

filters

for

each

participant

with

input

from

the

sound

device

and

output

to

the

multiplex-in

filters

of

each

other

participant

Create

multiplex-in

filters

for

each

participant

with

input

from

the

multiplex-out

filters

of

the

other

participants

and

output

to

the

sound

device

Extensions

to

existing

applications

would

also

be

easier

Volume

controls

could

be

added

to

the

Teleconf

application

by

creating

a

volume

filter

for

each

input

of

the

multiplex-in

filters

and

creating

a

Suite

dialog

manager

-LRB-

DM

-RRB-

to

modify

the

volumes

Implementing

each

filter

as

a

separate

Suite

object

would

result

in

performance

problems

so

we

use

the

idea

of

merged

server

design

developed

in

RAID

-LSB-

-RSB-

so

that

many

filters

can

run

within

a

single

address

space

Sound

as

a

Suite

base

object

Dialog

managers

in

Suite

are

generalized

editors

with

built-in

support

for

editing

base

objects

such

as

integers

characters

strings

and

sequences

of

base

objects

Users

build

applications

by

structuring

the

relationships

among

a

collection

of

base

objects

We

provide

sound

as

a

dialog

manager

base

type

Applications

designers

can

put

sound

buttons

in

applications

representing

editable

sound

objects

Users

can

choose

to

play

or

record

the

sound

or

can

perform

the

unelide

operation

on

the

button

to

open

a

waveform

editor

on

the

recorded

sound

to

perform

indexing

and

search

operations

Applications

structure

the

relationship

among

sound

buttons

through

SuiteSound

filters

For

instance

to

add

sound

annotations

to

any

Suite

application

the

programmer

extends

the

editable

data

with

one

or

more

sound

buttons

and

instructs

the

dialog

manager

to

display

the

sound

buttons

Coupling

with

other

users

SuiteSound

needs

to

be

integrated

with

the

coupling

concurrency

control

and

access

control

methods

used

for

other

object

types

in

Suite

Sound

is

coupled

like

other

dialog

manager

objects

allowing

users

to

choose

the

relationship

between

their

views

of

buttons

For

example

one

user

can

choose

to

only

make

complete

edited

annotations

available

-LRB-

visible

-RRB-

to

other

users

-LRB-

transmit

mode

in

Suite

's

coupling

model

-RRB-

while

another

user

can

choose

to

allow

other

users

to

listen

to

speech

in

chunks

separated

by

silent

periods

-LRB-

syntactically

complete

in

Suite

's

coupling

model

-RRB-

We

are

also

working

on

extensions

to

Suite

to

support

concurrency

control

and

access

control

Attempted

operations

will

be

checked

by

the

concurrency

controller

which

may

choose

to

delay

or

even

reject

them

For

instance

if

a

user

is

listening

to

audio

annotations

through

the

audio

device

he

or

she

may

specify

a

desired

behavior

when

another

user

tries

to

initiate

a

teleconference

session

Based

on

the

desired

behavior

the

concurrency

controller

can

reject

the

teleconference

request

delay

it

or

interrupt

the

annotation

by

starting

the

teleconference

immediately

Access

control

will

be

used

to

ensure

that

only

privileged

users

can

record

or

play

sound

on

an

audio

device

Access

control

for

audio

devices

is

in

some

ways

the

reverse

of

access

control

for

other

devices

since

writing

to

the

device

may

be

annoying

but

does

not

destroy

important

information

while

reading

from

the

device

is

especially

sensitive

because

it

violates

the

owner

's

privacy

Design

and

Use

of

SuiteSound

The

basic

goal

of

SuiteSound

is

to

provide

flexible

access

to

audio

devices

within

a

distributed

object-oriented

framework

Each

participating

workstation

runs

two

SuiteSound

objects

one

for

recording

from

the

microphone

and

one

for

playing

to

the

speaker

The

objects

provide

simple

facilities

to

initiate

flows

as

source

or

sink

respectively

Once

a

flow

is

started

data

can

either

be

delivered

automatically

from

a

source

to

the

next

link

in

the

flow

or

pulled

automatically

by

a

sink

from

the

previous

link

in

the

flow

Applications

structure

the

flow

as

they

desire

SuiteSound

provides

the

mechanism

for

creating

and

managing

flows

but

does

not

enforce

a

particular

structure

on

their

use

Applications

we

have

tested

to

date

split

flows

merge

flows

store

flows

and

replay

flows

Many

other

manipulations

are

possible

including

sophisticated

sound

processing

to

improve

quality

or

reduce

network

traffic

For

instance

we

are

experimenting

with

a

flow

filter

that

will

burst

a

flow

into

chunks

of

sequenced

sound

leaving

the

silent

periods

out

to

reduce

network

traffic

A

sequencer

filter

at

the

sink

will

reconstruct

the

original

flow

so

the

timing

is

correct

for

the

listener

The

flow

mechanism

provides

application

designers

with

a

higher-level

view

of

the

sound

device

freeing

them

to

build

better

applications

that

are

integrated

with

the

object-based

environment

Sound

flows

are

constructed

as

a

series

of

objects

linked

together

with

a

source

at

one

end

and

a

sink

at

the

other

In

most

applications

either

the

source

or

the

sink

is

chosen

to

provide

the

real-time

clock

rate

for

the

flow

The

designated

clock

source

issues

callbacks

to

the

next

link

in

the

flow

delivering

data

if

the

clock

source

is

also

a

sound

source

and

requesting

data

if

the

clock

source

is

a

sound

sink

These

method

invocations

from

a

sound

object

to

an

application

filter

are

named

callbacks

because

the

method

to

be

invoked

is

passed

to

another

object

which

calls-back

the

original

object

Callbacks

are

similar

to

Swift

's

upcalls

-LSB-

-RSB-

in

that

they

are

invocations

from

a

lower

layer

of

a

protocol

stack

to

a

higher

layer

Within

an

application

callbacks

are

used

to

pass

the

data

from

one

filter

to

the

next

in

the

flow

until

it

arrives

at

the

sink

Currently

all

applications

either

use

a

physical

audio

device

as

a

sink

in

which

case

the

sound

is

played

at

the

end

of

the

flow

or

a

recording

filter

in

which

case

the

sound

is

usually

made

part

of

another

flow

at

some

later

time

Future

applications

may

use

other

sinks

such

as

an

audio-to-text

converter

Sound

on

the

Sun

SparcStation

The

Sun

SparcStation

provides

a

physical

audio

device

that

is

capable

of

playing

and

recording

sound

at

per

second

Each

octet

represents

a

single

bit

Pulse

Code

Modulated

-LRB-

PCM

-RRB-

sample

from

the

physical

device

compressed

to

bits

using

law

companding

SunOS

provides

a

library

of

functions

for

operating

on

the

law

data

including

functions

for

converting

it

to

and

from

PCM

so

it

can

be

manipulated

linearly

For

instance

in

free

floor

mode

SuiteSound

's

teleconf

utility

converts

to

PCM

to

add

simultaneous

voice

signals

and

converts

back

to

law

before

playing

on

the

device

The

Sparcstation

includes

a

built-in

speaker

but

a

microphone

is

optional

Standard

dynamic

microphones

and

headphones

work

fine

Headsets

are

a

substantial

advantage

because

they

keep

the

microphone

near

the

speaker

's

mouth

while

leaving

his

or

her

hands

free

to

use

a

mouse

or

The

interface

to

the

audio

device

is

through

two

special

files

called

dev/audio

and

dev/audioctl

These

files

are

opened

and

closed

like

other

files

Once

open

they

can

be

read

or

written

with

the

standard

I/O

calls

Reading

from

the

device

delivers

encoded

digital

sound

data

from

the

microphone

while

writing

to

the

device

delivers

sound

data

to

the

speaker

Special

control

calls

are

available

for

checking

the

state

of

the

audio

device

changing

the

play

or

record

gains

pausing

during

playing

or

etc.

These

control

calls

can

be

done

on

an

opened

audio

device

or

can

be

issued

through

the

audioctl

device

At

most

one

process

can

have

the

audio

device

open

for

reading

and

at

most

one

process

can

have

it

open

for

writing

However

the

audioctl

device

is

sharable

so

one

application

can

issue

control

calls

without

interfering

with

other

applications

that

are

playing

and

recording

sound

data

The

specifics

of

the

audio

device

interface

may

change

in

future

releases

of

SunOS

However

SuiteSound

is

compatible

with

any

interface

based

on

the

general

framework

of

a

device

to

which

blocks

of

sound

can

be

read

or

written

The

same

SuiteSound

interface

can

be

implemented

on

many

physical

devices

Recording

Sound

is

transmitted

in

a

flow

in

data

structures

known

as

sound

bytes

Figure

shows

the

C

declaration

for

this

data

structure

The

Suite

object

compiler

recognizes

this

struct

as

a

declaration

of

a

special

data

type

called

an

opaque

sequence

An

opaque

sequence

is

a

octet

is

an

eight-bit

byte

in

Santa

Cruz

CA

is

willing

to

make

special

headsets

with

an

attached

microphone

that

work

with

the

Sun

SparcStation

undocumented

feature

is

that

close

calls

will

hang

if

the

audio

device

is

paused

typedef

struct

f

unsigned

num

bytes

unsigned

char

byte

opq

g

sound

bytes

type

Figure

This

data

structure

is

used

for

transmitting

sound

in

Suite

variable-sized

array

with

a

user-maintained

count

field

Opaque

sequences

are

adapted

from

regular

Suite

sequences

for

data

that

does

not

require

conversion

on

transmission

The

count

field

is

used

in

the

RPC

stubs

to

generate

correct

code

to

pack

and

unpack

the

sequence

Recorded

data

must

be

moved

from

the

physical

device

to

the

first

filter

in

the

flow

The

filter

object

that

starts

the

flow

identifies

itself

to

the

record

object

Each

time

the

record

object

has

data

available

it

delivers

the

data

by

invoking

method

UserRecordCallback

in

the

filter

object

There

are

two

basic

approaches

to

deciding

when

to

deliver

data

The

simplest

approach

is

to

read

data

every

time

it

is

available

on

the

sound

device

This

approach

delivers

sound

in

the

lowest

possible

latency

but

uses

all

of

the

computing

resources

of

the

workstation

to

do

so

It

is

only

appropriate

if

the

sound

application

is

the

only

application

on

the

workstation

In

the

more

common

case

that

the

sound

application

is

supporting

other

applications

sound

is

delivered

periodically

in

chunks

of

user-specified

size

Since

most

communications

costs

are

proportional

to

number

of

packets

rather

than

number

of

bytes

there

is

a

tradeoff

between

computing

cost

and

latency

of

sound

delivery

For

most

applications

the

packets

should

be

no

smaller

than

bytes

-LRB-

about

second

-RRB-

and

no

larger

than

bytes

-LRB-

about

second

-RRB-

The

methods

provided

or

used

by

the

sound

object

for

recording

are

Async

Method

StartRecording

Begin

delivering

data

from

the

sound

device

as

fast

as

possible

Async

Method

StartSynchronousRecording

Begin

delivering

data

from

the

sound

device

periodically

at

each

invocation

of

the

background

process

Async

Method

UserRecordCallback

Invoked

in

filter

process

to

deliver

sound

data

Async

Method

StopRecording

Terminate

recording

Async

Method

UserStopRecordCallback

Invoked

in

filter

process

after

last

sound

packet

has

been

delivered

UserStopRecordCallback

is

needed

so

the

filter

object

can

be

sure

the

recording

is

completed

Otherwise

data

that

has

already

been

recorded

from

the

physical

device

when

StopRecording

is

issued

may

be

lost

Playing

The

same

sound

bytes

data

structures

are

used

for

playing

sound

as

for

recording

it

Again

there

are

two

options

for

transmitting

data

between

the

filter

and

sound

objects

If

the

filter

device

is

receiving

the

sound

from

a

real-time

source

it

usually

uses

the

synchronous

mode

of

the

sound

device

to

play

the

data

at

the

same

rate

it

is

being

received

from

the

source

Otherwise

the

sound

device

is

placed

in

asynchronous

mode

so

it

will

ask

for

data

at

the

rate

that

it

can

be

played

on

the

physical

device

The

methods

provided

and

used

by

the

sound

device

for

playing

are

Async

Method

StartPlaying

Start

playing

in

asynchronous

mode

Data

is

requested

from

the

filter

as

needed

Async

Method

StopPlaying

Stop

asynchronous

play

Async

Method

UserPlayCallback

Invoked

in

the

filter

object

to

request

data

Async

Method

PlayData

Data

from

the

filter

object

in

response

to

an

asynchronous

callback

Async

Method

StartSynchronousPlaying

Start

playing

in

synchronous

mode

Data

is

played

as

it

is

delivered

by

the

filter

Async

Method

StopSynchronousPlaying

Stop

synchronous

play

Async

Method

UserStopPlayCallback

Invoked

in

the

filter

object

after

the

last

data

is

played

Async

Method

SynchronousPlay

Play

this

data

as

soon

as

possible

Sharing

the

Audio

Device

SuiteSound

objects

can

be

accessed

transparently

across

the

network

The

sound

is

played

or

recorded

on

the

workstation

on

which

the

SuiteSound

object

is

physically

located

To

simplify

access

to

correct

SuiteSound

object

they

are

named

suite-sound

<operation>

<hostname>

For

instance

a

workstation

named

hannibal

normally

runs

SuiteSound

objects

named

suite-sound-play-hannibal

and

suite-sound-record-hannibal

SuiteSound

currently

only

supports

one

active

reader

and

one

active

writer

at

a

time

SuiteSound

opens

the

audio

device

for

reading

or

writing

when

an

object

requests

recording

or

playing

When

one

object

stops

recording

or

playing

another

object

can

gain

control

of

the

SuiteSound

object

This

interface

can

be

extended

in

several

ways

to

improve

access

control

Users

with

transient

applications

-LCB-

such

as

alarm

clocks

or

mail

biffers

-LCB-

would

prefer

that

these

short-term

requests

could

over-ride

long-term

applications

such

as

teleconf

Objects

should

be

able

to

block

and

wait

until

a

SuiteSound

application

releases

the

object

Objects

that

use

SuiteSound

could

be

assigned

priorities

so

the

queue

would

better

represent

user

's

desires

A

simple

filter

application

could

allow

compatible

objects

to

share

the

device

by

overlaying

the

sound

For

instance

background

music

could

continue

while

an

audio

alarm

clock

plays

its

message

Such

a

filter

is

used

within

the

teleconf

tool

to

merge

simultaneous

speech

Experience

This

section

describes

our

experience

using

Teleconf

Annotator

and

CSI

and

discusses

some

issues

concerning

the

use

of

sound

as

a

medium

Teleconf

Teleconf

is

an

audio

teleconferencing

tool

built

on

SuiteSound

Teleconf

objects

manage

floor

control

for

the

users

and

pass

sound

among

themselves

before

playing

it

on

the

appropriate

workstations

Figure

depicts

the

interconnections

between

SuiteSound

objects

during

a

teleconference

Each

workstation

runs

two

SuiteSound

processes

that

manage

the

record

and

play

capabilities

of

the

device

In

addition

each

user

has

two

Teleconf

objects

one

for

routing

sound

to

other

users

and

one

for

receiving

sound

from

other

A

Master

object

manages

floor

control

not

processing

any

sound

itself

but

informing

the

Teleconf

objects

when

they

should

transmit

or

receive

sound

The

teleconferencing

tool

currently

supports

two

floor

control

modes

FIFO

queuing

and

no

floor

control

In

FIFO

mode

participants

press

a

button

in

a

Suite

dialog

manager

when

they

wish

to

speak

Then

they

join

a

queue

of

participants

waiting

to

speak

in

the

Master

object

The

participant

at

the

front

of

the

queue

holds

the

floor

The

Teleconf

object

for

the

floor-holding

participant

invokes

the

StartSynchronousRecording

method

on

the

participant

's

SuiteSound

object

Teleconf

objects

on

all

other

workstations

invoke

the

StartSynchronousPlaying

method

on

their

SuiteSound

objects

The

floor-holding

Teleconf

object

then

begins

receiving

sound

from

UserRecordCallback

methods

invoked

by

its

SuiteSound

object

This

sound

is

transmitted

to

each

of

the

other

Teleconf

objects

which

pass

it

to

their

SuiteSound

objects

with

SynchronousPlay

method

invocations

When

the

participant

is

finished

speaking

he

or

she

presses

a

button

in

a

Suite

dialog

manager

to

release

the

floor

The

Teleconf

objects

all

stop

their

SuiteSound

If

there

is

another

participant

in

the

queue

the

Master

assigns

the

floor

again

and

the

process

is

repeated

The

Master

is

still

involved

in

free

mode

but

now

each

member

of

the

queue

is

granted

the

floor

simultaneously

If

there

is

any

floor

holder

all

participant

Teleconf

objects

invoke

StartPlayingon

their

SuiteSound

object

Participants

invoke

StartSynchronousRecording

on

their

SuiteSound

objects

and

begin

transmitting

sound

to

all

other

Teleconf

objects

Now

however

the

Teleconf

objects

have

multiple

incoming

streams

representing

audio

data

that

is

being

recorded

simultaneously

They

convert

the

data

to

linear

form

and

add

it

discusses

limitations

in

the

UNIX

process

model

that

make

it

necessary

to

have

two

Teleconf

objects

the

original

design

the

floor

was

kept

active

if

another

participant

was

expected

to

gain

the

floor

The

new

design

is

simpler

and

the

slight

increase

in

control

messages

does

not

significantly

affect

performance

playingsync/async

playing

playing

playingsync/async

recording

floor

Exit

Conference

Request/Release

floor

Exit

conference

Request/Release

SuiteSound

record@hanna

SuiteSound

play@hanna

Teleconfrecord@hanna

Teleconfplay@hanna

master

Teleconfrecord@alex

Teleconfplay@alex

SuiteSound

record@alex

SuiteSound

play@alex

DialogueManager

DialogueManager

floor

requests

floorgranted

floorgranted

floor

requests

recording

Figure

The

SuiteSound

teleconferencing

tool

Figure

The

Teleconf

and

SuiteSound

objects

together

with

other

data

received

since

the

last

UserPlayCallback

from

the

SuiteSound

object

When

the

next

UserPlayCallback

arrives

the

summed

data

is

passed

to

the

sound

object

to

play

Figure

shows

on

all

of

the

objects

involved

in

the

use

of

Teleconf

Not

all

of

these

dialogue

managers

are

visible

in

a

normal

Teleconf

session

The

only

window

that

has

to

be

visible

is

the

Teleconf

window

-LRB-

the

middle

window

-RRB-

that

supports

the

requesting

and

releasing

the

floor

The

dialog

manager

on

the

left

provides

raw

information

on

the

play

and

record

volume

the

method

of

playing/recording

and

the

sampling

rate

The

dialog

manger

on

the

right

labeled

dm-Master

shows

the

queue

of

active

participants

and

shows

a

value

of

true

for

the

floor

control

-LRB-

meaning

that

we

are

in

FIFO

mode

-RRB-

Note

that

the

rest

of

the

windows

and

icons

are

the

window

manager

that

is

running

on

the

SparcStation

and

is

not

part

of

the

teleconferencing

tool

The

following

are

some

of

our

observations

regarding

the

use

of

the

teleconferencing

tool

Number

of

participants

in

FIFO

mode

We

have

found

out

that

when

using

FIFO

the

waiting

period

for

the

individual

participant

to

voice

his/her

comments

becomes

larger

as

the

number

of

participants

grows

The

waiting

can

lead

to

a

participant

getting

distracted

and

not

being

able

to

follow

the

line

of

discussion

to

a

participant

's

comments

not

being

relevant

to

the

present

discussion

anymore

-LRB-

as

the

line

of

discussion

changes

-RRB-

or

to

a

participant

simply

forgetting

his

or

her

comment

In

our

opinion

the

maximum

number

of

people

who

could

hold

a

discussion

effectively

and

comfortably

is

four

Restrictiveness

of

FIFO

FIFO

is

found

to

be

too

restrictive

by

our

users

and

in

many

instances

takes

the

spontaneity

away

from

the

users

Delay

We

held

our

first

meeting

in

the

same

room

and

therefore

were

able

to

hear

our

own

voice

coming

out

of

other

machine

a

second

later

This

delay

was

noticeable

by

the

participants

including

the

speaker

However

when

participants

are

distributed

they

no

longer

hear

their

own

voice

and

therefore

can

not

detect

a

delay

-LRB-

as

they

do

not

know

when

the

sound

was

transmitted

-RRB-

User

interface

The

use

of

Teleconf

should

be

transparent

to

the

users

Because

the

tool

is

usually

used

to

support

some

other

activity

-LRB-

e.g.

software

inspection

-RRB-

the

user

will

often

be

watching

parts

of

the

screen

other

than

the

Teleconf

dialogue

manager

Teleconf

should

mimic

face-to-face

speech

as

much

as

possible

by

providing

audible

feedback

instead

of

textual

feedback

Teleconf

will

support

this

notion

in

two

ways

Upon

receiving

the

floor

Teleconf

will

inform

the

user

by

calling

the

user

's

name

at

his

or

her

workstation

In

FIFO

mode

a

floor

holder

who

is

inactive

for

more

than

two

seconds

-LRB-

something

that

could

happen

if

a

person

has

forgotten

to

release

the

floor

-RRB-

the

user

will

hear

a

beep

every

two

seconds

Merged

sound

In

free

mode

the

speech

of

two

speakers

speaking

simultaneously

is

merged

We

intended

that

the

speech

would

be

garbled

so

the

speakers

would

understand

and

resolve

the

problem

and

so

that

listeners

would

know

that

someone

else

was

waiting

to

speak

Surprisingly

simple

speech

was

sometimes

understandable

when

merged

and

it

was

sometimes

possible

to

identify

the

speakers

in

merged

speech

-LRB-

an

advantage

for

later

floor

control

negotiation

-RRB-

Free

mode

Prior

to

making

enhancements

to

read/write

XDR

functions

the

performance

of

the

tool

in

Free

mode

even

with

only

two

participants

was

intolerable

The

words

were

chopped

off

at

the

end

and

the

delay

was

more

than

noticeable

The

enhancements

resolved

the

problems

mentioned

above

and

made

the

tool

usable

However

Free

mode

puts

considerable

load

on

the

machines

and

as

the

number

of

participants

grows

to

more

than

three

the

machines

will

be

unable

to

keep

up

We

concur

with

other

researchers

that

teleconferencing

with

no

floor

control

is

difficult

with

more

than

three

participants

-LSB-

-RSB-

We

are

performing

experiments

to

measure

the

load

per

user

in

free

mode

We

are

also

investigating

extensions

to

free

mode

in

which

sound

data

representing

relatively

silent

periods

is

not

transmitted

reducing

the

load

considerably

Tables

and

show

a

summary

of

the

execution

profile

of

all

functions

in

SuiteSound

and

Teleconf

objects

in

Free

mode

that

take

more

than

of

the

total

execution

time

The

information

presented

are

the

name

of

the

functions

percentage

of

the

running

time

of

the

program

spent

in

the

function

the

number

of

times

the

function

was

invoked

and

the

average

time

in

milliseconds

spent

in

each

function

and

its

descendants

per

call

Originally

the

sound

bytes

were

declared

to

be

of

type

character

This

created

performance

problems

as

each

byte

was

converted

individually

resulting

in

several

procedure

calls

and

high

time

and

space

overheads

Extending

Suite

with

the

opaque

data

type

and

declaring

the

sound

bytes

to

be

of

type

opaque

has

improved

the

performance

noticeably

Now

a

single

procedure

call

moves

the

entire

block

of

sound

data

into

the

XDR

data

structures

and

the

Function

name

Percent

time

calls

time

per

call

-LRB-

ms

-RRB-

WriteBytes

Write

-LRB-

System

call

-RRB-

Select

-LRB-

System

call

-RRB-

Table

Profile

of

SuiteSound

record

object

functions

Function

name

Percent

time

calls

time

per

call

-LRB-

ms

-RRB-

XDR

routines

XDR

-LRB-

-RRB-

XDR

-LRB-

-RRB-

Profile

-LRB-

mcount

-RRB-

Read

-LRB-

System

call

-RRB-

Write

-LRB-

System

call

-RRB-

Select

-LRB-

System

call

-RRB-

Table

Profile

of

Teleconf

functions

before

XDR

enhancement

unnecessary

conversions

are

avoided

There

is

a

comparative

table

provided

for

the

Teleconf

object

before

the

enhancements

were

done

to

the

XDR

functions

These

tables

do

not

tell

the

whole

story

The

original

Teleconf

object

in

free

mode

with

two

participants

took

of

the

CPU

time

of

a

Sun

Sparcstation

After

the

enhancement

the

same

Teleconf

object

delivers

the

same

sound

data

in

of

the

CPU

time

In

both

cases

most

of

the

time

is

spent

transmitting

the

data

but

in

the

latter

case

the

XDR

overhead

is

gone

Function

name

Percent

time

calls

time

per

call

-LRB-

ms

-RRB-

ReadBytes

AddSoundBytes

WriteBytes

Read

-LRB-

System

call

-RRB-

Write

-LRB-

System

call

-RRB-

Select

-LRB-

System

call

-RRB-

Table

Profile

of

Teleconf

functions

after

XDR

enhancement

UserPlayCallback

UserRecordCallback

Synch

Recording

Async

Playing

ExitStart/Stop

Playing

Start/Stop

Recording

managerDialogue

Play@alex

SuiteSound

record@alex

SuiteSound

Annotator

Figure

The

annotator

tool

design

Annotator

The

Annotator

tool

is

a

single-user

application

used

for

recording

and

playing

back

voice

annotations

The

voice

entries

are

entered

into

a

flexible

size

array

in

a

sequential

order

and

users

can

choose

to

play

the

recorded

entries

by

selecting

the

index

of

any

previously

recorded

message

The

information

stored

and

displayed

for

each

annotation

are

the

voice-entry-id

a

key

string

that

describes

the

annotation

and

the

time

of

creation

The

annotator

is

implemented

using

the

asynchronous

play

and

synchronous

record

features

of

the

SuiteSound

object

In

record

mode

the

Annotator

object

calls

the

synchronous

method

StartSynchronousRecording

in

the

SuiteSound

object

The

SuiteObject

opens

the

audio

device

for

reading

and

starts

returning

sound

bytes

to

the

caller

object

using

the

UserRecordCallback

routine

every

half

second

The

received

bytes

are

stored

into

the

current

entry

of

the

SoundAnnotations

array

until

there

is

no

more

room

is

available

For

practical

considerations

we

decided

on

an

annotation

to

be

around

one

hundred

seconds

-LRB-

i.e.

bytes

-RRB-

with

a

maximum

of

ten

voice

entries

allowed

per

session

The

playing

of

information

is

done

asynchronously

Once

an

index

is

chosen

the

Annotator

object

invokes

the

asynchronous

method

StartPlaying

in

SuiteSound

that

checks

the

status

of

the

audio

device

and

prepares

it

for

writing

The

SuiteSound

object

calls

UserPlayCallback

every

half

second

to

receive

sound

at

bytes

chunks

at

a

time

The

sound

chunk

is

sent

back

to

the

SuiteSound

object

for

playing

using

the

asynchronous

method

PlayData

This

is

done

until

there

is

no

more

data

available

on

that

entry

at

which

point

we

stop

playing

Figure

depicts

the

interconnections

between

SuiteSound

objects

during

an

annotation

session

As

is

the

case

with

the

Teleconf

object

each

workstation

runs

two

SuiteSound

processes

that

manage

the

record

and

play

capabilities

of

the

device

The

user

has

one

annotator

object

A

simple

directory

and

lookup

assistance

is

provided

with

Annotator

that

finds

a

voice

entry

-LRB-

i.e.

plays

that

entry

-RRB-

by

looking

up

its

identification

number

or

the

value

of

the

key

string

There

will

be

two

types

of

operations

supported

in

Annotator

Editing

operations

somehow

change

the

present

content

of

the

sound

annotations

Record

Recording

of

an

entry

Concatenate

Concatenating

to

an

existing

entry

Replace

Replacing/overwriting

an

existing

entry

Length

Returning

the

approximate

length

of

an

entry

in

seconds

The

non-editing

operations

in

no

way

change

the

content

of

the

sound

annotations

but

may

change

the

looks

of

things

-LRB-

i.e.

sorting

the

entries

-RRB-

Play

Playing

an

existing

entry

Loop

Looping/replaying

over

the

same

entry

Sort

Sorting

the

entries

based

on

two

keys

voice-entry-id

or

key

string

A

heavy

user

of

the

Annotator

will

be

the

CSI

tool

Annotator

can

provide

asynchronous

support

for

CSI

by

giving

the

reviewers

a

second

way

of

recording

their

comments

electronically

-LRB-

the

first

being

textual

-RRB-

The

voice

annotations

are

persistent

and

can

be

retrieved

later

by

the

producer

CSI

Collaborative

Software

Inspection

-LRB-

CSI

-RRB-

tool

supports

the

process

of

software

inspection

in

a

distributed

collaborative

environment

Traditionally

software

inspection

has

been

done

with

all

the

participants

present

in

a

meeting

room

and

the

inspection

done

on

paper

forms

and

documents

This

approach

is

expensive

in

terms

of

traveling

scheduling

human

resources

pre-meeting

preparation

of

material

and

post-meeting

recording

of

the

inspection

results

We

have

designed

and

built

a

distributed

collaborative

software

inspection

relaxing

the

requirement

that

all

the

participants

be

in

the

same

room

providing

automated

support

and

adding

structure

to

the

process

CSI

allows

geographically

distributed

participants

to

collaborate

in

the

inspection

and

provides

automated

support

for

all

inspection

activities

-LSB-

-RSB-

Participants

communicate

through

CSI

dialog

managers

Teleconf

and

voice

annotation

The

asynchronous

activities

of

software

inspection

include

the

distribution

of

target

material

and

the

creation

of

reviewer

's

fault

lists

The

target

material

is

on-line

and

accessible

by

reviewers

who

can

use

the

browser

to

view

it

and

annotate

the

lines

of

the

target

material

appropriately

These

annotations

help

the

producer

in

the

creation

of

the

integrated

fault

list

that

will

be

used

during

the

meeting

The

synchronous

activities

of

software

inspection

include

discussion

of

the

correlated

faults

reaching

a

consensus

on

the

faults

recording

the

action

items

and

finally

determining

the

status

of

the

inspection

The

target

material

is

brought

on-line

in

a

window

on

the

screen

of

all

the

participants

Participants

can

comment

on

the

target

material

either

through

a

sound

annotation

or

through

typed

annotations

in

a

shared

notepad

The

recorder

can

also

enter

action

items

into

an

action

list

that

is

on-line

and

visible

to

all

the

participants

The

recorder

has

a

form

that

allows

recording

of

information

on

the

attendees

status

of

the

meeting

and

the

final

decision

made

concerning

the

target

material

being

inspected

System

activity

-LRB-

such

as

loading

of

files

and

objects

updating

of

data

structures

in

objects

unloading

of

objects

-RRB-

is

recorded

automatically

and

is

available

for

future

statistical

studies

We

have

several

observations

on

the

use

of

sound

in

CSI

Preferred

mode

In

software

inspection

participants

play

specific

roles

One

such

role

is

that

of

a

moderator

The

moderator

ensures

that

the

meeting

follows

the

established

guidelines

and

does

not

stray

from

its

line

of

objectives

Having

a

moderated

floor

control

policy

is

best

suited

for

software

inspection

as

it

allows

the

moderator

to

enforce

certain

protocols

-LRB-

i.e.

when

a

participants

has

held

the

floor

for

too

long

the

moderator

may

choose

to

take

the

floor

away

or

simply

be

able

to

interrupt

the

speaker

long

enought

to

change

the

direction

of

the

discussion

-RRB-

Visual

cues

We

tried

convening

in

the

same

room

for

the

first

time

to

make

sure

that

things

would

go

smoothly

We

found

out

that

people

were

accustomed

to

receiving

visual

cues

and

in

effect

would

turn

around

to

see

one

another

while

speaking

The

later

meetings

were

held

-LRB-

truly

-RRB-

distributed

We

found

out

that

when

people

do

not

see

one

another

they

are

more

likely

to

focus

on

the

actual

work

they

are

doing

and

consequently

are

more

productive

Conclusions

and

Future

Work

Problems

and

Advantages

of

Continuous

Sound

in

Discrete

Objects

SuiteSound

has

proven

a

convenient

tool

for

building

object-based

applications

that

include

audio

The

mechanism

provided

has

made

it

relatively

easy

to

build

audio

applications

and

has

proven

flexible

enough

to

support

every

application

we

have

attempted

The

flow

model

is

convenient

especially

since

applications

can

initiate

a

flow

and

then

continue

with

their

other

work

while

the

flow

passes

through

asynchronously

Suite

has

been

a

good

platform

for

building

collaborative

applications

and

the

SuiteSound

extension

enhances

these

applications

However

we

have

found

four

limitations

in

our

implementation

three

introduced

by

limitations

of

the

underlying

UNIX

process

model

and

one

by

the

lack

of

dynamic

binding

in

Suite

The

problem

with

the

UNIX

process

model

is

that

it

does

not

currently

support

multiple

threads

of

control

within

an

address

space

This

limitation

means

that

methods

within

an

object

can

interfere

with

the

execution

of

other

methods

Three

examples

of

this

problem

are

Suite

supports

both

synchronous

and

asynchronous

method

invocation

In

synchronous

method

invocation

the

invoking

method

waits

for

a

reply

from

the

invoked

method

before

continuing

During

this

wait

no

other

method

can

be

executed

thus

blocking

any

flows

that

are

in

progress

Therefore

SuiteSound

objects

can

not

use

synchronous

methods

Real-time

flows

will

sometimes

run

as

fast

as

an

object

can

process

them

blocking

other

methods

from

executing

Thus

a

user

may

not

be

able

to

stop

the

flow

from

a

dialog

manager

We

used

a

feature

of

Suite

by

which

the

event

dispatcher

can

be

extended

by

an

application

to

implement

strict

round-robin

processing

of

method

invocations

Flows

with

a

cycle

in

them

will

sometimes

fill

up

the

TCP/IP

streams

used

by

Suite

for

communications

resulting

in

deadlock

To

avoid

this

problem

our

applications

do

not

construct

cyclic

flows

For

instance

the

Teleconf

play

and

record

objects

are

separate

to

avoid

creating

a

false

cycle

of

these

problems

would

exist

in

an

operating

system

that

supports

multiple

threads

of

control

in

each

address

space

-LSB-

-RSB-

The

fourth

problem

is

that

callback

methods

must

be

statically

bound

For

instance

when

an

object

asks

a

SuiteSound

record

object

to

issue

periodic

callbacks

the

name

of

the

callback

method

is

fixed

at

compile

time

We

work

around

this

problem

by

using

fixed

names

for

callback

procedures

An

object

tells

its

own

name

to

the

SuiteSound

object

which

then

knows

the

callback

method

to

invoke

Unfortunately

this

makes

it

difficult

to

handle

multiple

simultaneous

flows

This

problem

is

difficult

to

solve

since

the

Suite

object

compiler

checks

type

compatibility

between

methods

at

compile

time

Future

work

Experiments

We

are

interested

in

experimenting

with

audio

flows

to

measure

their

cost

and

effectiveness

on

networks

of

different

types

Interesting

possible

experiments

include

Network

requirements

Watch

the

actual

network

load

while

SuiteSound

applications

are

running

Use

a

packet

filter

on

a

separate

workstation

attached

to

the

same

ethernet

to

measure

the

load

caused

by

the

sound

packets

Computing

requirements

Measure

the

computing

costs

of

live

audio

on

modern

workstations

In

practice

much

of

the

cost

of

communication

is

computation

-LSB-

-RSB-

Since

audio

is

most

often

used

in

support

of

other

applications

it

is

important

that

it

not

put

too

much

load

on

the

system

Wide-area

network

Introduce

artificial

delay

and

see

if

the

participants

are

able

to

follow

the

conversation

This

experiment

simulates

the

behavior

of

the

system

in

an

overloaded

network

or

across

a

wide-area

network

User

interface

Team

up

a

novice

user

with

a

more

experienced

user

of

the

tool

and

make

a

list

of

common

mistakes

and

needs

This

should

help

in

understanding

the

type

of

user

interface

that

is

most

suitable

for

the

sound

media

For

instance

we

have

already

discovered

that

it

is

best

if

most

of

the

feedback

provided

by

the

system

is

audible

output

Our

experience

is

that

usually

sound

is

meant

to

support

some

other

activity

and

the

users

are

concentrating

on

the

other

activity

Floor

control

Measure

the

effectiveness

and

satisfaction

of

participants

solving

problems

under

several

different

floor

control

policies

to

understand

the

needs

of

the

different

types

of

meetings

and

how

a

particular

floor

control

policy

can

best

support

a

meeting

type

Collect

data

on

metrics

such

as

service

time

delay

and

performance

and

try

to

match

a

meeting

type

with

its

most

suitable

floor

control

policy

Enhancements

and

extensions

Teleconf

Annotator

and

CSI

are

relatively

new

tools

and

as

they

are

used

and

tested

more

we

will

be

enhancing

and

extending

them

further

Some

of

our

work

will

include

integrating

SuiteSound

into

Suite

adding

general

purpose

filters

for

performing

operations

on

the

flows

adding

more

forms

of

floor

control

to

Teleconf

and

making

its

feedback

to

the

user

as

audible

as

possible

Annotator

will

support

different

editing

operations

such

as

substring

and

indexing

Other

researchers

have

noted

that

sound

devices

are

difficult

to

share

because

of

their

serial

nature

-LSB-

AHL

-RSB-

However

in

the

Suite

environment

many

different

applications

may

want

to

use

the

sound

device

For

instance

if

a

user

is

listening

to

audio

annotations

through

the

audio

device

another

user

may

try

to

initiate

a

teleconference

session

Based

on

the

user

's

specified

behavior

SuiteSound

can

reject

the

teleconference

request

delay

it

overlay

an

audio

message

describing

the

originator

of

the

request

or

interrupt

the

annotation

by

starting

the

teleconference

immediately

Synchronizing

sound

is

especially

important

in

the

distributed

environment

where

communication

delays

are

unpredictable

We

plan

to

experiment

with

timestamps

inserted

in

the

multimedia

flow

We

are

currently

planning

one

timestamp

per

second

Two

problems

with

this

approach

are

the

differences

between

clocks

on

different

machines

and

the

gaps

between

timestamps

Our

experience

so

far

has

been

that

small

variances

in

synchrony

do

not

disturb

users

The

Teleconf

Master

is

an

interesting

example

of

a

collaborative

tool

in

Suite

in

that

its

implementation

is

completely

independent

of

the

Teleconf

application

We

plan

to

incorporate

the

same

master

object

into

an

existing

text

talk

tool

to

provide

similar

floor

control

This

is

an

example

of

extending

reusability

in

object-based

systems

to

meta-objects

that

control

the

behavior

of

other

objects

We

are

working

with

Prasun

Dewan

and

his

research

group

at

Purdue

University

to

develop

concurrency

control

techniques

that

support

the

entire

range

of

collaborative

activity

using

adaptability

techniques

to

change

concurrency

control

techniques

dynamically

-LSB-

-RSB-

Part

of

that

support

is

multimedia

flows

integrated

with

the

objects

in

the

system

We

are

actively

exploring

concurrency

control

techniques

that

are

effective

at

presenting

users

with

a

consistent

view

of

all

the

objects

in

the

system

including

multimedia

flows

References

-LSB-

-RSB-

S.R.

Ahuja

J.

Ensor

and

D.

Horn

The

Rapport

multimedia

conferencing

system

In

Proceedings

of

Conference

on

Office

Information

Systems

March

-LSB-

AHL

-RSB-

Susan

Angebarnndt

Richard

Hyde

Daphne

Luong

Nagendra

Siravara

and

Chris

Schmandt

Integrating

audio

and

telephony

in

a

distributed

workstation

environment

In

Proceedings

of

the

Summer

USENIX

Conference

pages

-LCB-

Nashville

TN

-LSB-

-RSB-

Bharat

Bhargava

John

Dilley

and

John

Riedl

Raid

A

robust

and

adaptable

distributed

system

In

ACM

SIGOP

workshop

-LRB-

Making

distributed

systems

work

-RRB-

Amsterdam

September

-LSB-

-RSB-

Bharat

Bhargava

Enrique

Mafla

John

Riedl

and

Bradley

Sauder

Implementation

and

measurements

of

an

efficient

communication

facility

for

distributed

database

systems

In

Proceedings

of

the

IEEE

Data

Engineering

Conference

Los

Angeles

CA

February

-LSB-

-RSB-

Bharat

Bhargava

and

John

Riedl

A

model

for

adaptable

systems

for

transaction

processing

IEEE

Transactions

on

Knowledge

and

Data

Engineering

December

-LSB-

-RSB-

Marc

Brown

Groupware

IEEE

Computer

May

-LSB-

-RSB-

D.

Clark

The

structuring

of

systems

using

upcalls

In

Proceedings

of

the

ACM

symposium

on

operating

systems

principles

December

-LSB-

-RSB-

David

R.

Cheriton

and

W.

Zwaenepoel

Distributed

process

groups

in

the

V

kernel

TOCS

-LRB-

-RRB-

-LCB-

May

-LSB-

-RSB-

P.

Dewan

and

R.

Choudhary

Flexible

user

interface

coupling

in

a

collaborative

system

Proceedings

of

the

ACM

CHI

's

Conference

April

-LSB-

-RSB-

P.

Dewan

A

guide

to

Suite

Technical

report

Software

Engineering

Research

Center

at

Purdue

Univesity

February

-LSB-

-RSB-

George

Drapeau

and

Howard

Greenfield

MAEstro

A

distributed

multimedia

authoring

environment

In

Proceedings

of

the

Summer

USENIX

Conference

pages

-LCB-

Nashville

TN

-LSB-

-RSB-

J.

Drake

V.

Mashayekhi

J.

Riedl

and

W.

Tsai

A

Distributed

Collaborative

Software

Inspection

Tool

Design

Prototype

and

Early

Trial

Technical

report

University

of

Minnesota

August

-LSB-

-RSB-

P.

Dewan

and

E.

Vasilik

An

object

model

for

conventional

operating

systems

Usenix

Computing

Systems

December

-LSB-

-RSB-

C.A.

Ellis

and

S.J.

Gibbs

Concurrency

control

in

groupware

systems

In

Proceedings

fo

the

ACM

SIGMOD

Conference

in

Groupware

systems

May

-LSB-

-RSB-

C.

Ellis

S.

Gibbs

and

G.

Rein

Groupware

Communications

of

the

ACM

January

-LSB-

FCB

-RSB-

J.W.

Fickett

M.J.

Cinkosky

C.

Burks

P.E.

Hempfner

D.

Nelson

R.M.

Pecherer

P.

Sgro

D.M.Sorenson

R.D.

Sutherland

and

B.C.

Yantis

Human

Genome

Information

Resource

Technical

report

Obtained

through

personal

communication

-LSB-

-RSB-

H.C.

Forsdick

and

R.H.

Thomas

The

design

of

Diamond

A

distributed

multimedia

document

system

Technical

report

TR

number

Bolt

Beranek

and

Newman

Inc.

October

-LSB-

-RSB-

J.C.

Hamlin

and

John

Riedl

Visualize

Computer

visualization

of

graph

theory

March

Undergraduate

Research

Opportunities

Program

University

of

Minnesota

-LSB-

-RSB-

P.

Nises

and

J.

Wettby

Phonetalk

Technical

report

Royal

Institue

of

Technology

December

-LSB-

-RSB-

Richard

Phillps

MediaView

An

editable

multimedia

publishing

system

developed

with

an

object-oriented

toolkit

In

Proceedings

of

the

Summer

USENIX

Conference

pages

-LCB-

Nashville

TN

-LSB-

-RSB-

Richard

F.

Rashid

Threads

of

a

new

system

Unix

Review

-LRB-

-RRB-

-LCB-

August

-LSB-

-RSB-

S.

Sakata

Development

and

evaluation

of

an

In-house

multimedia

desktop

coneference

IEEE

journal

on

selected

areas

in

communcations

April

-LSB-

SFB

-RSB-

Mark

Stefik

Gregg

Foster

Daniel

G.

Bobrow

Kenneth

Kahn

Stan

Lanning

and

Lucy

Suchman

Beyond

the

chalkboard

Computer

support

for

collaboration

and

problem

solving

in

meetings

cacm

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

J.M.

Tazelaar

In-depth

groupware

Byte

magazine

December

-LSB-

-RSB-

Robert

Terek

and

Joseph

Pasquale

Experiences

with

audio

conferencing

using

the

X

window

system

UNIX

and

TCP/IP

In

Proceedings

of

the

Summer

USENIX

Conference

pages

-LCB-

Nashville

TN

-LSB-

-RSB-

Jamie

Thingelstaad

and

John

Riedl

Collaborative

physics

instruction

on

the

Macintosh

March

Undergraduate

Research

Opportunities

Program

University

of

Minnesota

-LSB-

-RSB-

D.B.

Terry

and

D.C.

Swinehart

Managing

stored

voice

in

the

Etherphone

system

ACM

Transactions

on

Computer

Systems

February

-LSB-

-RSB-

P.

Wegner

Dimensions

of

object-based

language

design

In

OOPSLA

Proceedings

October

continuous media
digital audio
multimedia