Real-Time

Parallel

Decoding

in

Software

Angelos

Bilas

Jason

Fritts

Jaswinder

Pal

Singh

fbilas@cs,

jefritts@ee,

jps@csg.princeton.edu

Department

of

Computer

Science

Department

of

Electrical

Engineering

Princeton

University

Princeton

NJ

Phone

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Abstract

The

growing

demand

for

high

quality

compressed

video

has

led

to

an

increasing

need

for

real-time

MPEG

decoding

at

greater

resolutions

and

picture

sizes

With

the

widespread

availability

of

small-scale

multiprocessors

a

parallel

software

implementation

provides

an

effective

solution

to

the

decoding

problem

We

present

a

parallel

decoder

for

the

MPEG

standard

implemented

on

a

shared

memory

multiprocessor

The

goal

of

this

work

is

to

provide

an

all-software

solution

for

real-time

high-quality

video

decoding

and

to

investigate

the

important

properties

of

this

application

as

they

pertain

to

multiprocessor

systems

Both

coarse

and

fine

grained

implementations

are

considered

for

parallelizing

the

decoder

The

coarsegrained

approach

exploits

parallelism

at

the

group

of

pictures

level

while

the

fine-grained

approach

parallelizes

within

pictures

at

the

slice

level

A

comparative

evaluation

of

these

methods

is

made

with

results

presented

in

terms

of

speedup

memory

requirements

load

balance

synchronization

time

and

temporal

and

spatial

locality

Introduction

Recent

advances

in

network

and

microprocessor

technology

have

placed

video

applications

within

our

reach

High

Definition

Television

-LRB-

HDTV

-RRB-

Broadcast

Satellite

Service

Cable

TV

distribution

on

optical

networks

Electronic

Cinema

Interactive

Storage

Media

Multimedia

Mailing

Networked

Database

Services

corporate

Internet

training

and

conferencing

Remote

Video

Surveillance

and

others

are

now

becoming

practical

applications

The

huge

amount

of

data

needed

to

make

video

available

in

all

these

cases

has

led

to

the

adoption

of

the

and

standards

for

motion

video

compression

and

decompression

These

standards

greatly

reduce

the

bandwidth

and

storage

space

required

Consequently

and

are

already

being

used

in

many

video

applications

and

their

adoption

continues

to

grow

rapidly

One

factor

limiting

widespread

use

of

the

MPEG

standard

is

its

computational

complexity

Video

encoding

and

decoding

under

MPEG

is

expensive

too

costly

for

uniprocessors

to

achieve

real-time

performance

in

software

for

displays

of

interesting

resolution

and

size

The

computational

demands

grow

as

users

desire

higher

quality

video

Encoding

is

more

expensive

than

decoding

but

can

often

be

done

offline

and

hence

may

not

need

to

be

done

in

realtime

Decoding

however

demands

real-time

performance

Fortunately

real-time

decoding

is

within

the

capabilities

of

bus-based

multiprocessors

which

are

rapidly

becoming

commodity

systems

Almost

all

major

computer

system

vendors

including

PC

vendors

now

provide

such

configurations

directly

and

desktop

multiprocessors

may

not

be

far

away

While

specialpurpose

signal-processing

hardware

can

be

used

to

accelerate

decoding

-LSB-

-RSB-

the

ability

to

do

it

in

software

offers

many

advantages

it

provides

greater

flexibility

for

accommodating

new

algorithms

and

enhancements

as

they

evolve

does

not

require

additional

expensive

hardware

and

runs

on

general-purpose

systems

that

can

also

be

used

for

other

purposes

In

this

paper

we

examine

the

extent

to

which

increasingly

popular

cache-coherent

bus-based

shared

memory

multiprocessors

can

be

used

to

speed

up

software

MPEG

decoding

-LRB-

we

also

present

some

initial

results

on

a

cache-coherent

machine

with

physically

distributed

memory

-RRB-

We

present

two

parallel

implementations

of

the

decoder

provided

by

the

MPEG

Software

Simulations

Group

-LSB-

-RSB-

The

first

version

exploits

very

coarse-grained

parallelism

across

groups

of

pictures

in

the

video

sequence

while

the

second

exploits

fine-grained

parallelism

within

each

picture

We

evaluate

their

performance

and

resource

requirements

for

different

picture

sizes

and

numbers

of

processors

on

a

Silicon

Graphics

Challenge

multiprocessor

that

though

being

a

fairly

expensive

multiprocessor

uses

the

same

parallel

Other

sequential

software

MPEG

encoders-decoders

-LRB-

codecs

-RRB-

are

publicly

available

-LSB-

-RSB-

algorithms

and

techniques

as

would

be

used

by

less

expensive

desktop

servers

Detailed

measurements

with

various

performance

monitoring

tools

are

used

to

understand

the

role

of

potential

bottlenecks

to

good

performance

Finally

we

use

multiprocessor

simulation

to

characterize

the

spatial

and

temporal

data

locality

properties

of

the

parallel

versions

to

understand

how

they

will

interact

with

alternative

memory

system

architectures

and

to

understand

how

to

design

multiprocessors

and

their

memory

systems

to

obtain

better

parallel

performance

for

this

class

of

application

We

first

present

an

overview

of

MPEG

in

Section

focusing

on

those

aspects

that

are

most

relevant

to

parallelization

Following

this

Section

describes

the

testbed

that

was

used

for

implementing

and

benchmarking

the

different

algorithms

In

Section

we

present

the

general

methodology

for

parallelizing

the

decoder

Section

describes

the

parallel

implementations

along

with

the

results

In

the

same

section

we

also

study

the

locality

properties

through

simulation

Section

discusses

related

work

and

we

discuss

conclusions

and

future

work

in

Section

MPEG

Overview

The

MPEG

coding

standard

defines

a

lossy

compression

technique

which

takes

advantage

of

spatial

and

temporal

correlation

to

achieve

high

compression

ratios

In

exploiting

spatial

correlation

compression

is

achieved

by

finding

the

similarities

within

each

picture

and

using

those

similarities

to

eliminate

redundancy

in

the

picture

Spatial

correlation

alone

however

provides

only

moderate

compression

so

temporal

correlation

must

also

be

exploited

In

elminating

temporal

redundancy

successive

pictures

within

the

video

sequence

are

examined

for

similarities

These

similarities

are

then

used

to

remove

redundancies

across

pictures

Using

both

spatial

and

temporal

correlation

the

MPEG

standard

provides

high

degrees

of

compression

on

video

sequences

Though

both

and

use

this

compression

technique

from

a

computational

perspective

is

more

interesting

due

to

its

greater

versatility

While

most

video

streams

follow

the

constrained

parameters

bit

stream

format

which

limits

the

the

maximum

bitrate

to

Mb/s

and

picture

size

to

pixels

can

actually

be

applied

to

a

wide

range

of

input

formats

However

is

much

more

generally

defined

providing

a

coding

syntax

that

is

a

superset

of

the

syntax

Some

of

the

additions

to

include

support

for

interlaced

video

sequences

and

a

scalable

syntax

which

allows

for

layered

coding

of

video

streams

A

lossy

compression

scheme

is

one

in

which

data

are

lost

Only

partial

recovery

of

the

original

unencoded

data

is

possible

In

interlaced

video

sequences

each

picture

is

either

a

frame

or

a

field

where

two

interlaced

fields

comprise

a

frame

A

progressive

video

sequence

has

no

concept

of

fields

and

therefore

each

picture

is

a

frame

Consequently

in

interlaced

video

pictures

that

are

coded

as

fields

have

half

the

number

of

While

this

examination

of

the

parallelism

in

MPEG

has

not

currently

taken

full

advantage

of

these

extensions

we

chose

the

decoder

with

the

intent

of

exploring

these

avenues

in

the

future

The

MPEG

standard

specifies

the

coded

representation

of

picture

information

for

digital

storage

media

and

digital

video

communication

Since

video

compression

can

be

used

in

many

different

applications

and

each

application

places

a

unique

set

of

constraints

upon

the

video

stream

the

MPEG

committee

created

standards

that

are

flexible

enough

to

meet

the

demands

of

most

applications

provides

the

greatest

versatility

supporting

constant

or

variable

bitrate

transmission

random

access

channel

hopping

scalable

decoding

bitstream

editing

and

special

play

functions

such

as

fast

forward

playback

fast

reverse

playback

slow

motion

pause

and

still

pictures

-LRB-

offers

only

a

subset

of

these

functions

-RRB-

The

complete

MPEG

standards

are

quite

detailed

and

only

a

minimal

overview

may

be

given

here

More

comprehensive

descriptions

may

be

found

in

-LSB-

-RSB-

Layered

structure

of

an

MPEG

stream

An

important

aspect

of

the

versatility

of

MPEG

is

its

layered

structure

-LSB-

-RSB-

The

hierarchy

of

layers

in

an

MPEG

bitstream

is

arranged

in

the

following

order

Sequence

Group

of

Pictures

-LRB-

GOP

-RRB-

Picture

Slice

Macroblock

and

Block

-LRB-

see

Figure

-RRB-

The

different

parts

of

the

stream

are

marked

with

unique

byte

aligned

codes

called

startcodes

These

startcodes

are

used

both

to

identify

certain

parts

of

the

stream

and

to

allow

random

access

into

the

video

stream

This

random

access

ability

is

vital

to

parallelization

The

highest

level

in

the

layering

is

the

sequence

level

A

sequence

is

made

up

of

groups

of

pictures

-LRB-

GOPs

-RRB-

Each

GOP

is

a

grouping

of

a

number

of

adjacent

pictures

The

purpose

in

creating

such

an

identifiable

grouping

is

to

provide

a

point

of

random

access

into

the

video

stream

for

play

control

functions

-LRB-

fast

forward

reverse

etc.

-RRB-

In

the

GOP

is

a

required

layer

of

the

structure

but

in

it

was

made

optional

because

it

was

found

that

the

sequence

layer

can

serve

in

the

same

capacity

Within

each

GOP

are

a

number

of

pictures

In

each

picture

corresponds

to

a

frame

but

in

interlaced

video

is

supported

so

each

picture

corresponds

to

either

a

frame

-LRB-

for

progressive

or

interlaced

video

streams

-RRB-

or

a

field

-LRB-

for

interlaced

video

streams

-RRB-

in

the

original

stream

Pictures

are

further

subdivided

into

slices

each

of

which

defines

a

fragment

of

a

row

in

the

picture

Slices

comprise

a

series

of

macroblocks

which

are

pixel

groups

containing

the

luminance

and

chrominance

data

for

those

pixels

in

the

decoded

picture

Macroblocks

are

divided

into

blocks

-LRB-

to

depending

upon

format

-RRB-

A

block

is

an

pixel

group

that

describes

the

luminance

or

chrominance

for

that

group

of

pixels

Blocks

are

the

basic

unit

of

data

at

which

the

decoder

processes

the

horizontal

lines

as

progressive

pictures

-LRB-

frames

-RRB-

and

must

be

displayed

at

twice

the

display

rate

Luminance

is

the

monochrome

representation

of

the

signal

and

chrominance

provides

the

color

information

for

the

video

Video

stream

Sequence

Picture

Slice

Macroblock

Block

Group

of

Pictures

Figure

High

level

bitstream

organization

in

MPEG

P

B

B

P

B

B

P

B

B

Encoder

Input

Decoder

Output

PBB

I

P

Prediction

BBPBBI

Encoder

Output/Decoder

Input

I

B

B

P

B

B

P

Prediction

Bidirectional

Figure

Example

of

picture

structure

Notice

that

while

the

encoder

input

and

decoder

output

both

have

pictures

in

the

correct

display

order

the

encoded

stream

has

them

out

of

order

such

that

each

I-Picture

comes

before

the

P-Picture

which

references

it

-LRB-

i.e.

picture

comes

before

pictures

-RRB-

and

each

I

and

P-Picture

comes

before

the

B-Pictures

that

reference

it

-LRB-

i.e.

pictures

and

come

before

pictures

-RRB-

encoded

video

stream

Macroblocks

and

blocks

do

not

have

startcodes

associated

with

them

their

boundaries

are

discovered

implicitly

while

decoding

Another

important

aspect

of

the

structure

of

MPEG

is

the

picture

type

Pictures

in

MPEG

are

encoded

into

one

of

three

types

as

shown

in

Figure

All

picture

types

use

spatial

correlation

but

not

all

use

temporal

correlation

The

first

picture

type

the

intra

coded

picture

-LRB-

I-Picture

-RRB-

uses

only

spatial

correlation

Since

their

decoding

is

independent

of

other

pictures

I-Pictures

provide

access

points

into

the

coded

stream

where

decoding

can

begin

However

using

just

spatial

correlation

they

achieve

only

moderate

compression

The

second

type

of

picture

the

predictive

coded

picture

-LRB-

P-Picture

-RRB-

is

coded

more

efficiently

by

also

using

temporal

redundancies

from

a

past

I

or

P-Picture

These

P-Pictures

are

then

used

for

reference

in

further

prediction

The

final

picture

type

the

bidirectionallypredictive

coded

picture

-LRB-

B-Picture

-RRB-

uses

temporal

redundancies

from

both

past

and

future

reference

pictures

and

consequently

achieves

the

highest

degree

of

compression

B-Pictures

are

never

used

as

references

for

prediction

The

organization

of

the

three

picture

types

in

a

stream

is

very

flexible

The

only

stipulation

upon

their

ordering

is

that

any

picture

that

depends

upon

temporal

redundancies

from

a

past

or

future

picture

must

be

decoded

after

its

reference

pictures

As

a

result

the

order

of

the

pictures

in

the

input

stream

to

the

decoder

-LRB-

i.e.

the

output

of

the

encoder

-RRB-

is

not

the

order

in

which

the

pictures

must

be

displayed

An

example

of

this

reordering

in

the

encoded

stream

is

actually

supports

a

fourth

picture

type

the

DC

coded

picture

-LRB-

D-Picture

-RRB-

type

However

this

type

is

little

used

and

was

eliminated

from

shown

in

Figure

Encoding

As

described

above

the

block

is

the

basic

unit

of

data

processing

For

each

block

of

data

in

the

video

sequence

the

encoder

performs

five

steps

to

produce

an

encoded

block

motion

estimation

discrete

cosine

transform

-LRB-

DCT

-RRB-

quantization

and

run-length

and

Huffman

coding

-LRB-

see

Figure

-RRB-

In

the

first

stage

motion

estimation

the

encoder

tries

to

take

advantage

of

temporal

redundancies

among

pictures

Motion

estimation

required

only

for

P

and

B-Pictures

is

performed

using

the

relevant

reference

pictures

During

motion

estimation

each

macroblock

is

moved

within

a

variable-size

search

window

around

its

position

in

the

reference

picture

until

the

position

of

greatest

similarity

is

found

This

displacement

is

saved

as

a

motion

vector

For

each

block

in

the

macroblock

the

difference

from

the

corresponding

block

at

the

position

indicated

by

the

motion

vector

in

the

reference

picture

is

computed

This

gives

the

prediction

error

for

each

element

in

that

block

The

next

four

stages

of

the

encoder

take

advantage

of

spatial

correlation

in

compressing

the

video

sequence

Following

motion

estimation

the

encoder

takes

either

the

original

block

data

-LRB-

for

I-Pictures

-RRB-

or

the

prediction

error

information

-LRB-

for

P

or

B-Pictures

-RRB-

and

performs

a

discrete

cosine

transform

-LRB-

DCT

-RRB-

on

the

block

to

find

its

frequency

spectrum

Typically

in

video

sequences

the

higher

frequency

components

are

less

prominent

as

they

are

less

noticeable

to

the

human

eye

so

much

of

the

transformed

block

has

Motion

estimation

is

only

performed

on

a

macroblock

basis

All

blocks

within

that

macroblock

use

the

same

set

of

motion

vectors

small

values

The

next

step

quantization

zeroes

out

many

of

these

smaller

values

reducing

the

number

of

distinct

frequency

values

This

quantization

step

is

the

lossy

step

of

the

encoding

process

and

the

degree

of

quantization

is

specified

using

both

quantization

matrices

and

a

quantization

coefficient

Uncompressed

Encoder

Decoder

Run

length

encoding

Quantization

Motion

Estimation

Huffman

encoding

Inverse

Quantization

Run

length

decoding

Huffman

decoing

Motion

Compensation

DCT

Inverse

DCT

Compressed

Video

Video

Uncompressed

Compressed

VideoVideo

Figure

The

MPEG

model

After

quantization

run-length

and

Huffman

coding

are

performed

Quantized

blocks

have

numerous

zeros

and

can

be

efficiently

encoded

using

run-length

coding

Only

non-zero

elements

and

the

number

of

zeros

between

them

are

recorded

saving

much

space

in

the

encoded

result

Finally

the

non-zero

elements

are

Huffman

coded

a

process

in

which

the

elements

with

the

most

common

values

are

given

the

shortest

codes

The

combination

of

run-length

coding

and

Huffman

coding

minimizes

the

amount

of

data

needed

to

describe

the

quantized

block

The

result

of

performing

these

five

encoding

stages

on

all

blocks

in

a

video

sequence

is

an

MPEG

encoded

video

stream

A

stream

may

be

encoded

once

and

then

transmitted

across

a

transmission

media

and/or

stored

as

needed

However

decoding

is

necessary

every

time

the

stream

is

viewed

Decoding

Decoding

is

a

completely

deterministic

process

Given

an

MPEG

stream

there

is

only

one

way

to

decode

it

into

a

sequence

of

pictures

The

decoding

process

for

an

MPEG

encoded

stream

performs

the

five

encoding

stages

in

reverse

order

First

Huffman

and

run-length

decoding

are

applied

to

generate

the

quantized

block

Then

inverse

quantization

is

performed

to

get

the

block

's

frequency

spectrum

From

this

the

inverse

discrete

cosine

transform

-LRB-

IDCT

-RRB-

is

taken

to

obtain

the

block

's

spatial

data

-LRB-

for

I-Pictures

-RRB-

or

prediction

error

data

-LRB-

for

P

or

B-Pictures

-RRB-

Then

if

necessary

motion

compensation

may

be

used

to

generate

the

final

macroblock

data

from

the

prediction

error

motion

vectors

and

reference

pictures

Since

the

order

of

the

pictures

in

the

encoded

stream

is

not

necessarily

the

display

order

it

is

the

job

of

the

decoder

to

display

the

pictures

in

proper

order

-LRB-

see

Figure

-RRB-

However

before

displaying

a

picture

more

processing

-LRB-

e.g.

dithering

-RRB-

may

be

necessary

A

major

difference

between

the

encoding

and

decoding

processes

is

that

in

the

former

much

time

is

spent

in

motion

estimation

as

it

is

necessary

to

search

for

the

most

similar

macroblock

in

the

reference

picture

-LRB-

s

-RRB-

whereas

in

the

latter

motion

vectors

are

already

available

which

makes

motion

compensation

much

cheaper

than

motion

estimation

System

Environment

This

section

describes

the

hardware

and

software

environment

that

was

used

in

the

implementation

as

well

as

the

testing

methodology

we

used

to

benchmark

the

different

algorithms

The

Multiprocessor

platform

The

SGI

Challenge

multiprocessor

is

a

cache-coherent

bus-based

centralized

shared

memory

multiprocessor

The

machine

we

use

has

processors

connected

by

a

bitwide

bus

with

peak

bandwidth

of

GB/sec

Each

processor

is

a

MIPS

with

peak

performance

of

MFlops

Each

node

has

first

level

data

and

instruction

caches

of

each

-LRB-

direct-mapped

-RRB-

and

a

unified

second

level

cache

of

-LRB-

setassociative

-RRB-

The

system

has

of

main

memory

that

is

interleaved

out

of

which

we

could

use

up

to

for

our

program

The

system

can

support

up

to

I/O

buses

each

peak

The

operating

system

is

IRIX

and

it

incorporates

substantial

functionality

from

UNIX

System

V

Release

Since

the

machine

supports

a

shared

address

space

programming

abstraction

shared

data

can

simply

be

allocated

as

such

and

then

referenced

directly

by

any

processor

Our

parallel

programs

are

written

in

C

augmented

with

the

parmacs

parallel

programming

macros

from

Argonne

National

Laboratory

-LSB-

-RSB-

Porting

the

program

to

other

shared

address

space

architectures

is

easily

achieved

by

using

the

proper

version

of

the

parmacs

system

for

the

architecture

under

consideration

The

macros

are

also

supported

by

our

multiprocessor

simulation

environment

so

we

can

measure

various

characteristics

of

the

parallel

programs

Performance

analysis

tools

To

understand

performance

characteristics

and

bottlenecks

we

used

the

pixie

and

prof

tools

pixie

instruments

the

code

and

counts

basic

blocks

The

ideal

execution

time

of

each

process

in

a

program

is

then

calculated

given

the

cycle

time

of

the

system

and

a

one

cycle

delay

for

every

memory

reference

prof

on

the

other

hand

uses

program

counter

sampling

-LRB-

pc-sampling

-RRB-

to

determine

where

time

is

spent

and

reports

actual

execution

time

including

memory

stall

time

However

the

timing

results

reported

by

pixie

and

prof

include

synchronization

ti

me

which

must

be

computed

separately

by

instrumentin

g

the

source

code

The

results

o

f

pixie

and

prof

may

then

be

combined

to

determine

the

time

spent

in

the

memory

subsystem

Stream

Resolution

GOP

size

Picture

size

K

K

Table

Description

of

test

streams

Test

st

reams

We

tried

to

be

as

consistent

as

possible

in

choosing

the

input

se

quences

Most

public

domain

sequences

are

small

ha

ve

random

characteristics

and

do

not

explore

the

parameter

space

in

any

systematic

way

We

therefore

created

our

own

set

of

test

streams

Starting

with

a

small

public

domain

stream

a

moving

view

of

a

flower

garden

with

pictures

and

resolution

of

pixels

-LRB-

flowg.mpg

from

Stanford

-RRB-

we

created

larger

streams

by

repeating

a

number

of

pictures

in

a

continuous

video

sequence

and

scaling

each

picture

using

interpolation

Each

resulting

stream

is

composed

of

a

total

of

pictures

has

a

pictures/sec

display

rate

and

or

Mbits/sec

bit

rate

The

I/P

picture

distance

is

thus

there

are

B-Pictures

between

any

two

consecutive

reference

I

or

P-Pictures

Table

shows

the

characteristics

of

the

streams

We

vary

only

two

parameters

in

our

test

streams

the

resolution

and

the

number

of

pictures

per

GOP

These

are

important

because

they

define

the

amount

of

processing

required

to

decode

a

picture

as

well

as

the

memory

requirements

of

the

system

As

seen

in

Table

we

use

four

different

resolutions

-LRB-

-RRB-

and

four

different

numbers

of

pictures

per

GOP

-LRB-

-RRB-

for

a

total

of

streams

The

public

domain

encoder

-LSB-

-RSB-

we

used

to

create

the

streams

creates

one

slice

for

each

row

of

a

picture

Thus

the

four

different

picture

sizes

have

and

slices

respectively

from

the

smallest

to

the

largest

Similarly

most

public

domain

video

sequences

we

found

also

have

a

small

number

of

slices

per

picture

-LRB-

usually

one

per

row

-RRB-

One

other

parameter

of

video

streams

that

is

of

great

importance

is

the

bit

rate

This

is

because

the

bit

rate

provides

a

measure

of

both

the

degree

of

compression

and

the

relative

quality

of

the

video

All

of

the

results

presented

in

this

paper

used

the

aforementioned

video

streams

which

assumed

a

fixed

In

terminology

all

the

streams

have

a

main

profile

and

a

high

level

The

last

t

wo

streams

are

more

commonly

found

with

pictures

sizes

of

and

We

used

the

uncommon

sizes

to

maintain

consistent

picture

size

ratios

bit

rate

of

Mb/s

for

the

and

streams

and

Mb/s

for

the

streams

But

since

bit

rates

can

vary

considerably

according

to

the

desired

degree

of

compression

or

video

quality

we

also

felt

it

important

to

examine

the

effect

of

different

bit

rates

on

parallelism

Using

streams

of

widely

varying

bit

rates

we

found

that

the

decoding

times

for

streams

of

a

given

picture

size

are

typically

within

of

the

time

measured

for

our

test

streams

However

this

decoding

time

differential

is

seen

to

a

proportionate

degree

with

an

increasing

number

of

processors

The

overall

result

is

that

the

speedups

are

consistent

So

even

though

the

absolute

execution

time

deviates

slightly

for

video

streams

of

different

bit

rates

there

is

no

noticeable

impact

on

parallel

performance

Exploiting

Parallelism

The

amount

of

work

associated

with

decoding

different

pictures

and

even

with

different

parts

of

the

same

picture

is

variable

and

unpredictable

so

maintaining

a

balanced

workload

requires

that

we

use

some

form

of

dynamic

tasking

mechanism

Static

assignment

of

tasks

to

processes

is

also

difficult

because

tasks

are

not

known

ahead

of

time

but

are

created

as

the

input

is

read

in

parallel

with

the

actual

computation

We

present

two

different

methods

for

exploiting

parallelism

In

both

methods

the

incoming

stream

is

decomposed

into

tasks

that

are

put

in

task

queues

and

can

be

processed

in

parallel

The

difference

lies

in

the

nature

and

granularity

of

the

tasks

which

affects

the

performance

and

characteristics

of

a

parallel

implementation

The

possible

choices

for

a

task

in

MPEG

are

sequence

group

of

pictures

-LRB-

GOP

-RRB-

picture

slice

macroblock

and

block

Given

the

encoding

scheme

in

MPEG

only

a

GOP

and

a

slice

are

reasonable

choices

as

we

shall

see

The

first

type

of

parallelism

is

across

pictures

Since

P

and

B-Pictures

depend

on

other

nearby

pictures

assigning

ad

jacent

pictures

to

different

processors

leads

to

many

serializing

dependencies

and

associated

synchronization

and

communication

among

processors

A

better

solution

is

to

parallelize

across

completely

independent

GOPs

Parallelizing

at

either

the

sequence

or

GOP

level

might

work

however

the

sequence

level

may

lead

to

tasks

which

are

too

large

and

create

load

imbalance

Therefore

the

GOP

level

is

a

more

reasonable

choice

At

the

GOP

level

tasks

can

still

be

very

coarse-grained

but

since

GOPs

are

relatively

independent

there

is

essentially

no

inherent

communication

in

the

parallel

algorithm

except

in

accessing

shared

task

queues

This

forms

the

first

approach

which

we

call

the

GOP-level

implementation

In

the

second

type

of

parallelism

parallelism

within

a

picture

the

only

plausible

approach

is

to

use

slices

Recall

that

in

the

GOP

level

is

optional

When

the

GOP

level

is

used

sequences

are

typically

large

but

when

it

is

not

used

sequence

sizes

are

usually

smaller

Hence

when

the

GOP

level

does

not

exist

the

sequence

level

may

be

used

for

parallelization

as

tasks

s

ince

t

hey

a

re

marked

with

startc

odes

in

the

in

put

stream

The

pro

blem

with

macroblocks

and

blocks

is

that

they

do

not

ha

ve

startcodes

to

identi

fy

them

without

actually

doin

g

the

decoding

itself

-LRB-

if

they

did

we

could

used

them

as

tasks

as

well

-RRB-

Parallelizing

at

these

levels

would

make

it

necessary

for

one

process

to

perform

the

decoding

of

the

stream

detect

the

boundaries

of

each

macroblock

-LRB-

including

its

motion

vectors

-RRB-

while

processing

the

stream

and

assign

the

macroblock

or

its

blocks

to

other

processors

for

further

processing

While

this

approach

may

be

viable

it

places

a

large

load

on

one

processor

and

would

only

pay

o

in

cases

where

there

is

a

signi

ficant

problem

with

load

imbalances

at

the

slice

level

Because

slices

are

groups

of

ad

jacent

macroblocks

they

increase

task

granularity

compared

to

macroblocks

potentially

causing

load

imbalance

but

help

reduce

communication

when

reconstructing

blocks

from

motion

vectors

and

other

nearby

blocks

Our

other

parallel

implementation

which

we

call

the

slice

level

implementation

defines

the

task

unit

to

be

a

slice

We

shall

discuss

both

these

parallel

versions

and

their

tradeoffs

further

in

subse

quent

sections

Experimental

Results

Parallelism

a

t

group

o

f

pi

ctures

le

vel

The

first

approach

taken

in

parallelizing

the

decoder

tries

to

exploit

parallelism

at

the

GOP

level

In

this

implementation

we

define

the

task

unit

to

be

a

GOP

Recall

that

a

GOP

consists

of

any

number

of

pictures

By

definition

a

GOP

must

contain

at

least

one

I-Picture

Also

the

first

picture

-LRB-

in

display

order

-RRB-

in

a

GOP

must

be

an

I-Picture

or

a

B-Picture

and

the

last

picture

in

a

GOP

must

be

an

I-Picture

or

a

P-Picture

If

the

first

picture

is

an

I-Picture

or

a

B

Picture

that

does

not

depend

on

the

pictures

of

the

previous

GOP

then

the

GOP

is

defined

as

a

closed

GOP

and

it

can

be

decoded

independently

Since

consecutive

GOPs

can

be

decoded

by

different

processors

GOPs

need

to

be

closed

Although

the

assumption

that

all

GOPs

in

the

stream

are

closed

is

not

necessarily

true

of

the

streams

generated

by

encoders

it

is

in

fact

not

very

restrictive

One

way

to

overcome

it

even

when

the

GOPs

in

the

input

stream

are

not

closed

is

by

taking

advantage

of

the

fact

that

the

stream

contains

startcodes

for

pictures

and

identifies

their

type

with

a

type

field

This

parallel

design

does

not

require

that

tasks

be

GOPs

as

defined

in

the

input

stream

but

rather

any

closed

set

of

pictures

that

can

be

decoded

independently

The

scan

process

could

scan

the

stream

and

construct

closed

tasks

This

would

be

somewhat

complicated

by

the

fact

that

each

of

these

tasks

would

have

to

contain

pointers

to

appropriate

headers

so

that

the

worker

processes

could

find

configuration

information

specific

to

the

pictures

that

it

would

decode

For

simplicity

we

assume

that

the

GOPs

in

the

input

stream

are

themselves

closed

and

can

be

processed

independently

Figure

shows

the

architecture

of

the

parallel

decoder

We

dedicate

one

process

the

scan

process

to

read

the

stream

from

the

disk

-LRB-

or

network

or

other

source

-RRB-

and

identify

the

tasks

While

reading

the

stream

into

memory

it

scans

the

stream

for

startcodes

that

mark

the

beginning

of

each

GOP

All

but

one

of

the

other

processes

are

worker

processes

which

dequeue

tasks

from

the

task

queue

and

decode

the

corresponding

GOP

Each

worker

processor

operates

in

a

loop

in

which

it

removes

the

next

task

from

the

queue

decodes

the

corresponding

GOP

and

places

the

decoded

pictures

in

a

display

queue

The

worker

processes

assume

that

the

data

for

the

GOPs

in

the

task

queue

are

already

in

memory

The

last

process

is

assigned

as

the

display

process

It

is

responsible

for

displaying

the

pictures

which

may

have

been

processed

and

inserted

in

the

display

queue

out

of

order

in

the

correct

order

It

is

also

responsible

for

dithering

the

pictures

In

the

measurements

we

present

we

do

not

include

dithering

time

since

it

is

not

a

necessary

part

of

the

decoding

The

dithering

cost

can

vary

greatly

depending

on

the

characteristics

of

the

display

device

The

scan

process

and

the

display

process

do

not

take

part

in

the

decoding

process

Another

approach

would

be

to

have

them

participate

when

free

but

this

would

complicate

the

system

Moreover

as

the

requirements

for

better

image

quality

and

consequently

picture

resolution

and

bit

rate

increase

these

processors

will

be

busy

most

of

the

time

anyway

The

speed

at

which

the

scan

process

is

placing

pictures

in

the

task

queue

is

shown

in

Table

Here

we

assume

-LRB-

quite

reasonably

-RRB-

that

the

scan

process

can

be

fed

with

data

at

the

required

bit

rate

Doing

this

under

a

variety

of

conditions

is

a

topic

of

current

research

in

the

areas

of

networking

and

I

O.

Picture

size

File

size

-LRB-

Mbytes

-RRB-

Number

of

pictures

1120 1120 1120

Scan

time

-LRB-

sec

-RRB-

Scan

rate

-LRB-

pics/sec

-RRB-

Table

Scan

rate

in

the

scan

process

The

motivation

for

the

GOP

level

design

is

twofold

First

tasks

are

large

and

independent

so

accesses

to

the

task

queue

are

infrequent

and

there

is

little

synchronization

or

inherent

communication

among

the

worker

processes

Second

the

resulting

system

is

easy

to

understand

and

implement

Picture

size

Max

pictures/sec

Table

Maximum

number

of

pictures/sec

decoded

for

each

picture

size

Disk

Slice

Slice

Slice

Slice

Slice

Slice

Slice

Slice

Slice

Slice

Slice

Slice

Pic

Pic

Pic

Pic

a

b

c

e

Slice

Task

Queue

Task

Queue

Display

Queue

WorkerWorkerWorker

Scan

Server

Display

Server

Display

GOP

a

GOP

GOP

GOP

b

c

d

GOP

Task

Queue

Figur

e

Ar

chi

tecture

o

f

the

para

llel

decoder

Results

I

n

our

mea

surements

on

the

Challenge

we

tried

to

capture

the

beha

vior

of

the

decoder

in

terms

of

speedups

memor

y

re

quirements

load

balance

and

the

components

of

e

xecution

time

including

memory

overhead

We

omit

the

results

obtained

for

the

smallest

resolution

-LRB-

-RRB-

in

all

cases

due

to

space

limitations

Per

for

mance

an

d

s

peedup

We

measure

speedup

as

the

ratio

of

the

number

of

pictures

per

second

that

P

wor

ker

processors

-LRB-

P

total

processors

-RRB-

can

decode

to

the

number

of

pictures

per

second

that

are

decoded

by

one

worker

processor

-LRB-

total

processors

-RRB-

This

is

di

fferent

than

the

speedup

obtained

over

a

uniprocessor

system

which

would

multiplex

the

scan

and

display

processes

with

the

worker

process

in

the

uniprocessor

baseline

and

hence

likely

in

flate

the

speedups

The

results

in

Figure

show

that

the

speedup

is

almost

linear

in

all

cases

Table

gives

the

maximum

number

of

pictures

per

second

decoded

for

each

picture

resolution

using

worker

processes

Load

imbalance

To

capture

load

imbalance

we

measured

the

minimum

maximum

and

average

com

puting

times

among

the

worker

processes

Figure

shows

that

when

the

number

of

pictures

per

GOP

is

small

the

minimum

and

maximum

times

are

very

close

to

the

average

This

means

that

all

the

worker

processes

spend

approximately

the

same

amount

of

time

computing

As

the

number

of

pictures

per

GOP

increases

the

load

imbalances

become

more

apparent

because

the

number

of

tasks

decreases

and

the

time

to

process

a

task

increases

with

GOP

si

ze

Thus

if

one

or

more

processors

process

even

one

more

task

than

the

rest

they

will

appear

to

have

computed

signi

ficantly

more

In

reality

even

this

is

just

an

artifact

of

the

relatively

short

input

stream

and

load

imbalance

among

workers

is

not

likely

to

be

a

problem

for

real

streams

that

contain

many

GOPs

Memor

y

system

and

sync

hroni

zation

over

heads

The

time

spent

executing

instructions

stalled

in

memory

and

waiting

at

synchronization

points

was

measured

by

pixie

prof

and

source

level

instrumentation

as

discussed

earlier

The

pixie

prof

times

in

Figure

show

that

in

all

the

cases

-LRB-

with

an

average

of

-RRB-

of

the

time

is

spent

stalled

in

memory

We

shall

study

cache

miss

rates

and

memory

system

interactions

through

simulation

later

Synchronization

time

among

the

worker

processes

is

minimal

They

only

need

to

synchronize

when

accessing

common

resources

The

time

spent

on

locks

was

measured

to

be

negligible

compared

to

the

processing

time

in

each

slave

-LRB-

MBytes

-RRB-

Figure

Actual

memory

requirements

for

the

GOP

approach

Memory

requirements

The

maximum

memory

required

by

the

system

depends

on

the

number

of

processors

and

the

picture

size

Each

processor

needs

to

keep

up

to

three

decoded

pictures

in

memory

-LRB-

two

reference

pictures

and

the

one

currently

being

decoded

-RRB-

The

speed

of

the

scan

server

affects

the

memory

requirements

as

well

It

allocates

memory

to

store

the

data

it

reads

from

disk

potentially

increasing

memory

requirements

when

it

outpaces

the

decoding

In

Figure

we

plot

the

maximum

amount

of

memory

used

by

the

decoder

for

each

test

stream

We

see

that

the

memory

used

by

the

system

grows

with

the

size

of

the

GOP

the

size

of

the

picture

and

the

number

of

processors

used

In

many

applications

this

is

not

a

problem

because

the

size

of

a

GOP

is

relatively

small

For

instance

applications

that

require

random

access

fast-forward

playback

or

fast-reverse

playback

favor

the

use

of

relatively

short

GOPs

To

predict

memory

requirements

for

different

Fi

gu

re

Spee

dups

f

or

t

he

G

OP

approa

ch

exec

frames/GOP

Figure

Load

ba

la

nce

for

the

GOP

approach

The

x

axis

is

the

number

of

wor

ker

processors

used

and

the

y

axis

the

minimum

average

and

maximum

execution

times

among

the

worker

processes

-LRB-

normali

zed

to

the

average

-RRB-

streams

and

system

con

figurations

we

derived

an

analytical

model

that

gives

the

memory

re

quirements

of

the

program

as

a

function

of

execution

time

taking

into

account

the

speeds

of

the

scan

server

and

the

decoding

process

Figure

shows

the

predicted

requirements

in

memory

for

three

cases

as

a

function

of

time

The

third

case

-LRB-

pictures

GOP

processors

-RRB-

can

not

be

run

on

the

system

because

the

memory

requirements

exceed

the

available

memory

Despite

the

dynamic

allocation

and

deallocation

of

memory

as

tasks

are

created

and

completed

the

model

has

been

verified

to

be

very

close

to

the

actual

behavior

of

the

system

In

addition

to

the

large

memory

requirements

this

method

also

has

the

problem

that

it

has

large

random

access

latency

for

play

functions

For

example

should

the

user

fast

forward

to

a

later

section

of

the

video

sequence

decoding

must

begin

anew

at

that

point

with

each

processor

grabbing

one

GOP

Because

only

one

processor

processes

a

GOP

the

speed

at

which

the

video

begins

to

display

at

that

point

is

dependent

upon

one

processor

not

all

the

processors

As

a

result

the

GOP

parallel

method

is

better

suited

to

continuous

play

Sl

ice

le

vel

paralleli

s

m

Addressing

these

problems

led

us

to

consider

another

approach

to

parallelizing

the

decoder

In

our

second

parallel

decoder

implementation

we

define

the

task

unit

to

be

a

slice

As

defined

by

the

standard

a

slice

in

MP

EG

is

a

series

of

an

arbitrary

number

of

macroblocks

within

one

row

of

the

picture

Each

slice

must

contain

at

least

one

macroblock

and

consecutive

slices

may

not

overlap

Slices

occur

in

the

bitstream

in

the

order

in

which

they

are

encountered

starting

at

the

upper-left

of

the

picture

and

proceeding

by

raster-scan

order

from

left

to

right

and

top

to

bottom

However

slices

do

not

need

to

maintain

the

same

structure

from

picture

to

picture

there

may

be

different

numbers

of

slices

and/or

different

slice

sizes

H

A

B

C

D

E

F

G

Figure

The

general

slice

structure

in

a

picture

In

the

most

general

case

it

is

not

necessary

for

slices

to

cover

the

entire

picture

-LRB-

Figure

-RRB-

Areas

not

enclosed

in

a

slice

are

not

encoded

However

in

all

the

profiles

defined

so

far

by

the

standard

a

r

es

tr

icte

d

s

lice

str

ucture

is

used

in

which

every

macroblock

in

time

Ideal

timeActual

Ideal

timeActual

Ideal

timeActual

time

Fi

gure

Ide

ala

ndac

tual

time

forthe

GO

Pa

pproach

The

x

axis

isthe

num

ber

ofprocessors

and

the

yaxisthe

ideal

-LRB-

blac

kbars

-RRB-

and

actual

-LRB-

white

bars

-RRB-

time

asgi

ven

bypixie

and

prof

respectively

summed

over

all

workerpro

cesses

The

results

arenormali

zed

tothe

ideal

time

Mbytes

t

-LRB-

sec

-RRB-

mem

-LRB-

x

-RRB-

scan

-LRB-

x

-RRB-

frames

-LRB-

x

-RRB-

100 200 300 400

Mbytes

t

-LRB-

sec

-RRB-

mem

-LRB-

x

-RRB-

scan

-LRB-

x

-RRB-

frames

-LRB-

x

-RRB-

50 100 150 200

Mbytes

t

-LRB-

sec

-RRB-

mem

-LRB-

x

-RRB-

scan

-LRB-

x

-RRB-

frames

-LRB-

x

-RRB-

Figure

Predicted

memory

re

quirements

forthree

cases

mem

-LRB-

x

-RRB-

is

memory

usage

scan

-LRB-

x

-RRB-

isthe

portion

thatcorresp

onds

tothe

scan

process

and

frames

-LRB-

x

-RRB-

thememory

allocated

for

pictures

Note

that

mem

-LRB-

x

-RRB-

scan

-LRB-

x

-RRB-

frames

-LRB-

x

-RRB-

the

picture

is

enclosed

ina

slice.The

architecture

ofthe

decoder

isbasically

the

sameas

inthe

first

approach

However

because

ofthe

needof

theprocessors

toaccess

picture

header

informationwhile

decoding

slices

and

the

need

to

synchronize

atpicture

boundaries

we

found

that

a

task

queuefacilitates

a

simpler

implementation

The

first

levelof

the

task

queue

holds

pictures

while

thesecondlev

el

holds

the

slices

within

those

pictures

ascan

beseen

in

Figure

The

scan

process

while

scanningthe

input

stream

adds

a

task

tothe

picture

taskqueue

for

each

picture

startcode

found

and

then

foreac

h

slice

startcode

found

within

that

picture

it

addsa

task

tothe

slice

task

queue

associated

with

thatpicture

When

decoding

a

picture

worker

processesremo

ve

tasks

from

the

slice

queue

assocated

with

thatpicture

When

the

current

slice

task

queue

emptiesand

the

slices

are

alldecoded

the

picture

is

placed

inthe

display

queue

and

thenext

task

from

thepicturetask

queue

isaccessed

The

slice

queue

associatedwith

that

picture

then

becomes

thenew

task

queue.In

our

first

implementation

-LRB-

simple

slice

version

-RRB-

pro

cessors

synchronize

atthe

end

ofevery

picture

so

parallelism

isonly

exploited

within

apicture

Also

noattempt

is

made

to

preserve

locality

across

the

slicesfrom

different

pictures

that

are

assigned

tothe

samepro

cessor.Two

important

differences

from

theGOP-level

ap-proac

h

is

that

the

memory

requirements

are

muchlo

wer

and

theclosed

GOP

assumption

is

notnecessary

Since

all

theprocessors

inthe

system

work

on

thesame

picture

which

isinshared

memory

atmost

threepictures

in

allneed

tobe

inmemory

ata

time

-LRB-

versusthree

pictures

perprocessor

as

required

bythe

GOPv

ersion

-RRB-

The

other

benefit

ofthe

slice

version

is

that

itdoesnot

have

therandom

access

latency

problem

forplaycon

trol

functions

When

aplay

control

function

causespla

y

tobegin

from

a

new

position

in

the

video

stream

all

worker

processors

not

just

one

immediately

begindeco

ding

thenew

GOP

slice

by

slice

in

parallel.The

disadvantages

of

the

slice

approach

aresyn-c

hronization

and

inherent

interprocess

communica-tion

Processes

communicate

asthey

access

the

samemacroblo

cks

from

the

reference

pictures

particularly

ifthose

macroblocks

-LRB-

slices

-RRB-

were

assigned

to

and

written

byother

processesin

the

re

ference

pict

ure.As

note

dabove

the

worker

processes

in

this

i

m

p

lementation

synchroni

ze

at

the

endofeachpicture

The

number

of

slicesperpicture

isnotde

fined

by

the

standard

butmosttestse

quences

we

found

usedonly

one

sliceperrowofmacroblocks

Thuseachpicture

usually

containsasmall

numberofslices

-LRB-

the

vertical

resolution

dividedby

the

verticalsize

ofa

macroblock

-RRB-

This

hasanimportant

impacton

theload

balance

andtheperformance

ofthisapproach

For

e

xample

a

picture

has

slices

If

we

todecodesuch

apicture

twoworkerswill

get

three

sliceswhiletheother

twelveonlygettwoand

will

beidlewhilethefirst

threearedecodingtheirfinal

slice

thiscreatesaseriousproblem

inspeedups

Particularly

since

ithappens

inev

erypicture

Thereisanimprovement

inexecutiontime

as

processorsareadded

onlywhentheload

isdivided

equallybetweenallthe

processors.T

oremedy

this

weimproved

theimplementationto

synchronize

theworkersonly

aftercertainpicturet

ypes

not

aftereverypicture

-LRB-

improvedsliceversion

-RRB-

The

key

observationisthatall

B-pictures

inaseriesuse

thesame

referencepictures

andarenotthemselvesused

as

referencepictures

Thus

sincethenextpicture

does

notdependonthe

picturecurrentlybeingdeco

ded

availableworkerscan

begindecoding

thenext

picture

aftercompleting

alltheirtaskson

thecurren

t

picture.Synchronization

isneededonly

atthe

end

ofanIorP-picture

Thisdoesnotexploitthe

maximum

concurrency

butthatwouldrequirecomplex

synchronizationatthe

slicelevel.In

the

followingparagraphs

wepresentandexplainthe

behavior

ofthesetwofine

grainedimplementa-tions

ResultsResults

forthesliceimplementation

arepresentedin

theform

ofspeedups

synchronization

time

-LRB-

loadim

balance

-RRB-

andidealversus

actualtime

Compared

totheresults

forthe

GOP

approach

memoryrequire-men

tsare

verylowandpractically

independentof

then

umber

ofprocessorsandthe

GOPsize.Wepresentsync

hronization

timeresultsfor

loadimbalanceratherthan

max-min-average

results

overtheexecutionsincew

orkerprocesses

synchronize

atpictureboundaries.Since

the

effectivenessofthe

sliceapproachdoes

notdep

end

onthenumberofpictures

inaGOP

weonlyv

arythe

sizeofthepictures

andkeeptheGOPsizeconstan

t

case

Perform

anceandspeedup

Speedupsaremea-sured

in

thesamewayas

intheGOPapproach.Figure

showstheperformance

ofbothsliceimple-men

tations

Weseethatifthe

processessynchronize

atev

erypicture

thenspeedups

arenearlylinearonly

forlarge

pictures

-LRB-

thatcontainmany

slices

-RRB-

Thekneesin

thesimple

version

especially

happenwhentheinteger

ratioofthenumberof

slices

inapictureoverthe

numberofprocessors

is

reduced

caseeachpicture

version

Frames/secforthe

sliceapproachcon

soperformance

does

n'tincreaseformore

processors.The

improved

versiongreatly

reducesthisimbal-ance

Thenumber

ofslicesthat

canbeprocessedislarger

depending

ontheI/Pdistance

ofthestream.This

implementation

exposesenough

slicelevelcon-currency

for

thenumbersofprocessors

usedandac

hievesvery

goodspeedupsforall

pictureresolutions

Framesize

version

Impro

vedversion

GOP

version

T

Maximum

numberofframes/sec

decodedforeac

hpicture

size

themaximumnumber

offramespersecond

decoded

foreachpictureresolution

Fromfromthis

tablewe

seethattheimproved

sliceversionhascomparable

performance

totheGOP

version.Itisslo

werdueto

theincreasedoverhead

inmanagingthefiner

tasks

-LRB-

reading

pictureheaders

multipletimes

etc.

-RRB-

andthe

additionalsynchronization

timeneededat

pictureboundaries

Synchronization

overhead

Figure

verageratio

ofsynchronizationwait

timetoexecutiontime

forallworkers

asafunction

ofthenumberofw

orkerprocesses

Itclearlyshows

thattheimprovedv

ersionperforms

better.Thetimes

reportedincludeb

othtimespent

inthetaskqueue

andatsynchroniza-tion

points

though

theformerisvery

smallcomparedto

thesynchronization

time.Thus

althoughtaskgran

ularityis

muchsmallerforthis

version

usingacen

tralizedtask

queuedoesnotconstitute

aproblemfor

thesliceapproach

either.The

knees

thatwerementioned

beforeareagainvisible

inFigure

onlynowthey

arereversed.Theratio

ofsynchronization

toexecution

timegenerallyincreases

orremains

constant

However

itdoesdropwhenev

erthe

ratioofslicesperpicture

tothenumberof

theworker

processesdecreases

sync/exec

Theaverage

-LRB-

synctime/exec

time

-RRB-

of

a

llworkerprocesses

versusthenum

ber

ofprocessorsintheslicemetho

d.

Themoreslices

#procs

thelessthesynctime.Forcertainnumbersofprocessorsthisdiagramre

flects

thatmoreprocessorsstayidlelongerinthesimpleversion

In

theimprovedversionsynchroni

zation

doesnotincreaseasmuchwiththenumberofprocessors

Memorysystem

Usingprof

andp

ixie

wefoundtheactualtimetobeonaveragewithin

oftheideal

whichmeansthatcachemissescostverylittleinthisapproachaswell.Thus

both

GOP

levelandslicelevelapproachesgivegoodspeedups

thoughthelatterusesmuchle

ssmemory

While

thememoryre

quirements

oftheformerincreaselinearlywithGOPsize

pictureresolutionandnumberofpro

cessors

therequirementsofthelatterdependonlyonpictureresolution

Ad

ditionally

theGOPlevelapproachhaslongrandomaccesslatencyforplaycontrol

functions.Ontheotherhand,theslicelevelversionhassomewhathighersyn-chro

nizationandcommunicationneeds

whichreducesitsspeedup.Weshallnowexamine

thememorysysteminteractionsoftheseversionsmorecloselytodeterminetheexp

ectedscalingwhenusinglargermachinesandlargerpicturesizes

andtoseehowdi

fferentcacheorganizationsimpactperformance

LocalitypropertiesTounde

rstandthetemporalandspatialdatalocal-ity

weperformedsoftwaresimulation

softhemulti-processorexecutionfortheprogram

Thesimulationsweredoneusin

gtheTangoLiteexecution-drivenref-erencegeneratorcoupledtoamemorysystem

sim-ulator

-LSB-

-RSB-

Thesimulatormodelsacache-coherentmultiprocessorwithonele

velofcacheperprocessorandisflexibleinsettingarchitecturalparameters.In

theresultswepresentmissrates

-LRB-

ratioofmissestomemoryreferences

-RRB-

nottimin

gnumbers

becausetheimpactofsometypesofcachemissesonperformancecanbehid

denbyusingrelaxedmemoryconsistencymodelsandotherlatencytolerancetechni

ques.Forspatiallocality

fullyassociative

cache.Weseethatthe

missratehalveswheneverthecachelinesizedoubles

whichindicatesthatthepro

gramhasverygoodspatiallocality.TheresultsarefortheGOPversion

size

-LRB-

bytes

-RRB-

Read

miss

Figure

Read

missrateversuslinesizeforaneight-pro

cessorexecution

fullyassociativecache

size

-LRB-

KB

-RRB-

Read

miss

rateGOP

version

size

-LRB-

KB

-RRB-

Simple

slice

version

Figure

Miss

rateversuscachesize.Left

GOPv

anda

cacheline.Right

Simple

sliceversion

for

heline.sliceversionhas

similarbehavior.The

sizeandscaling

ofaprogram

sworkingsets

-LRB-

i.e.its

temporallocality

-RRB-

areimportanttounderstandingho

wdatatraffic

andperformancewillscaletolargerproblems

andmachines

andfordeterminingwhatcac

hesizeswill

benecessaryforgoodperformance.W

emeasurethe

workingsetsoftheprogrambyplotting

theread

missrateversusthecachesize

-LRB-

per-pro

cessor

-RRB-

used

inthesimulations.SincetheGOPapproac

hdoes

n't

haveanysharingamongtheworkerpro

cesses

-LRB-

which

alldosimilarwork

-RRB-

weassumeaone

processorexecution

andone-way

two-wayandfully

associative

F

ortheslicelevel

size

-LRB-

KB

-RRB-

missesGOP

version

size

-LRB-

KB

-RRB-

slice

version

Ratio

ofreadcapacitymissesoverreadcold

missesversus

cachesize.Left

processorand

Right

Simpleslice

eightworkerprocessors.Theresultsforasingleworkerprocessorwillbeessenti

allythesameasfortheGOPapproach

becausetheslicelevelapproachincursinher

entinterprocessorcommunication

andbecausewithparallelismwithinapicture

theaccesspatternschangeasthenumberofprocessorschanges.Thesimulationres

ultsindicatethatthemissrateforlargecachesizesisdominatedbycoldmissesra

therthancommunicationmisseseveninthiscase.Thenumberoftruesharingmisses

issmallincomparison

andfalsesharingnegligible.Asforcapacitymisses

wefi

nd

-LRB-

-RRB-

-LRB-

one-waya

ssociative

-RRB-

uggeststhattheworkingsetsarerelativelysmall.Toverifythisweplottheratio

ofthereadcapacitytothereadcoldmissesversusthecachesize

gureshowsthatthenumberofcapacitymissesissmallcomparedtothecoldmisses.T

husthecapacitymissratesandtrafficdonotconstituteabottleneckandincreasi

ngthecachesizefurther

beyondtheworkingsetsize

doesnotsignificantlyimpr

oveperformance.Theresultsalsoshowthattheworkingsetsizedoesnotchangewit

hthepicturesizeorthenumberofprocessors,evenfortheslice-levelversion

su

ggestingthatitisdeterminedbythedatausedforthereconstructionofasinglema

croblockorsetofmacroblocks

whichisindependentoftheseparameters.Thework

ingsetsarethereforeexpectedtofitinthecachesofmodernmultiprocessorsandc

apacitymissesarenotlikelytobeaproblemevenforverylargepicturesineithera

pproach.WecanexpecttocontinuetoobtaingoodparallelperformanceonMPEGdeco

dingatleastonsmalltomoderatescalemultiprocessors

RelatedWorkWearenota

wareofanyworksofarthathasdemonstratedsuccessfulparallelizationofMPEG-2

decodingonsharedmemorymultiprocessors,orexam-inedtheperformancebottlen

ecksandcharacteristics.TheworkonparallelMPEG-2hasfocusedonmessage-pass

ingsystems

andmostlyontheencodingprocesswithitsmuchgreatercomputationa

lcosts.Reportedworkhasnotanalyzedthebottlenecksortheimpor-tantdataloca

litycharacteristics

-LSB-

-RSB-

presentsaparallelMPEG-2encoderforlargescalemult

iprocessors.Parallelismisexploitedattheblockandmacroblocklevel.Theyrep

Swith330processors.Theirteststreamshaverelativelylowresolution

-LRB-

maximum

-RRB-

andthereislittleperformanceanalysistoenableextrapolation

-LSB-

-RSB-

de

scribesaparalleldecoderthatexploitspar-allelizationattheGOPlevelinames

sage-passingenvironment

-LSB-

-RSB-

presentsanMP

Theyconcludethatfortheirapproach

thebestparallelschemeshouldbebasedontheslices.Parallelizingatthemacrob

locklevelisfoundtobeimpracticalforthegeneralcase.Real-timeencodingisac

hievedforrelativelysmallpicturesizes.Workhasalsobeendoneindesigninghar

dwareorcombinedhardware-softwarecodecsthatachievereal-timeperformance

-LSB-

-RSB-

describesasoftwaresolutionontheMultimediaVideoMultiprocessor

-LRB-

-RRB-

andreportsreal-timeresultsforsmallpictureencoding-decoding

Theyal

somentionthattheproposedsolutionachievesrealtimedecodingforlargerpictu

resaswell

ConclusionsWehaveinvestigatedthe

behavioroftwoparallelimplementationsoftheMPEG-2decodingprocessonshared

memorymultiprocessors.ParallelizationwasperformedattheGOPlevelandslice

level

respectively

usingtheparmacsparallelenvironment.Bothmethodsdemon

stratedverygoodspeedups.TheresultswepresentedwereobtainedusingaSGIChal

lengesystem

butthetwoimplementationsareportableacrossalargenumberofpla

tforms.ParallelismacrossGOPsreducesthesynchroniza-tionrequiredamongpro

cessors

buthaslargerandomaccesslatencyforplaycontrolfunctionsandcanlea

dtoextremememoryrequirementsthatincreaselinearlywiththeGOPsize

-LRB-

numbero

fpicturespergroup

-RRB-

pictureresolution

andnumberofprocessors.Ontheotherh

and

parallelizingattheslicelevelincreasessynchronizationandcommunicati

onsomewhatbutes-sentiallyeliminatestherandomaccesslatencyproblemanddra

maticallyreducesmemoryrequirements.Thememoryrequirementsforthisapproac

hdonotincreasewithGOPsizeornumberofprocessors

butonlywithpictureresolu

tion.Wealsousedasimulatortoinvestigatethecachebehaviorofthedecodingpro

cess

andfoundexcellentspatialandtemporallocality.Theresultsshowverygoo

dparallelspeedupsforbothversions,andthatwecanachieverealtimedecod-ingf

orreasonablesizedpictures

-LRB-

-RRB-

onsmall-scalesharedmemorymul

tiprocessors.Forlargerpictures

-LRB-

e.g.

-RRB-

closetoreal-timeperfor-ma

ncemaybeachievablewithhighendsystemsusingthelatestprocessors

andperhap

sfurtheroptimizationoftheserialuniprocessorcode

-LRB-

wehavenottriedtooptimi

zethecodefromtheSoftwareSimulationGroupotherthanthroughthecompiler

-RRB-

Al

so

sincecommunicationissmall

andsinceourresultsshowthatcapacitymissesa

renotaproblemandworkingsetsdonotgrowwithpicturesize

moreandmoreprocess

orscanbebroughttobearaslargerresolutionsareused

evenintheslicelevelver

sion.Thetrendinmicroprocessorstowardtheuseoflongcachelinesalsohelpsspe

edupdecoding

etomoderate-scalecache-coherentmachinesthathavephysicallydistributedma

inmemory.Willafullydynamicscheme

withnoattempttopreservelocalityintask

assignment

workwellascommunicationcostsgetrelativelyhigher?Forexample

inourslicelevelimplementationwemakenoattempttoensurethattheprocessorde

codingagivensliceisalsoassignedslicesfromlaterframeswhichreferencethat

slice.Andwilldataplacementacrossphysicallydistributedmemoriesbeimporta

ntandmanageable?Wearecurrentlyinvestigatingtheseissuesoncache

coherent

distributedmemorymachines.PreliminaryresultsontheStanfordDASHmultiproc

essor

-LSB-

-RSB-

showthatboththeGOPandtheimprovedsliceversionscaleaslongasthere

areenoughtaskstokeeptheworkerprocessesbusy.However

fortheseprogramswri

ttenwithoutanyattentiontodatadistributionspeedupsarenotasgoodasonacent

ralizedsharedmemorysystem.Forinstance

theimprovedsl

-LRB-

oneclusterinDASH

-RRB-

andtheGOPversionspeedsupalittleless.Dat

aplacementonadistributedsharedmemorysystemaffectsthenumberofremotemiss

eseachprocessexperiences

andconsequentlytheperformanceofthesystem.Usin

gahardwareperformancemonitor

weobservedthelatency

-LRB-

notcontention

-RRB-

ofalot

ofremotemissesasbeingthemajorimpedimenttospeeduponDASH

-LRB-

synchronization

waittimeandhenceloadimbalanceisnotaproblemforeithertheGOPorimprovedsli

ceversions

-RRB-

sosmartdataplacementacrossmainmemoriesmaybenecessarytoimpr

ovespeedups.Thisisdifficultfortheslicelevelscheme

butcanbedonefortheGO

Plevelbyusingataskqueueperprocessor

havingaprocessorbeassignedthetasks

correspondingtoGOPsthatareloadedintoitslocalmemory

-LRB-

GOPsmaybeloadedinro

und-robinorderamongmemories

-RRB-

andthenhavethemstealtasksfromotherqueuesf

orloadbalancing.Oursimulationresultsforthelowcommunicationmissrateandt

hesmallworkingsetssuggestthatthisschemeshouldworkwellonmoderate-scalem

achines

butthisbearsmoredetailedinvestigation

FutureWorkAsmentioned

intheMPEGOverview,therearemanyextensionsoverMPEG-1thatwereaddedwhencre

Foremostamongtheseweresup-portforinterlacedvideoandascalab

lesyntaxwhichallowslayeredcodingofvideosequences.Inourinitialwork

wepr

imarilyexploredparallelizationasitappliedtothegeneralMPEGstandard

-LRB-

both

-RRB-

Whilethescalablesyntaxisnotoftenusedintoday

sapplicat

ions

thisisprimarilyduetothedifficultyinhandlingthelargeamountofdatath

ataccompaniestheupperlevelofthescalablevideosequence.Thiswilllikelycha

ngeastechnologygrowstomeettheseneeds.Unlikethescalablesyntax

interlace

Hence

itwillbenecessarytoexploreth

eparallelizationofboththeseextensionstoprovideacompletemultiprocessors

olutiontotheproblemofreal-timeMPEG-2decoding

llengesystemusedinourexperimentsisnotaninexpensivecommoditysystem.Howe

ver,itgaveustheabilitytoinvesti-gatedifferentaspectsofMPEG-2decodingwi

thasignificantnumberofprocessors.Giventhatprocessortechnologyhasalread

ymovedtothenextgenerationofprocessors

-LRB-

UltraSparc

etc.

-RRB-

small

scaleSMPsystemsthatarebecomingcommoditysystems

willoutperformtheChalle

ngesystemweused.ThisworkshowsthatanallsoftwaresolutiontotheMPEG-2decod

ingproblemisbothviableandinteresting.Toverifyhowwellthesenewsystems

-LRB-

es

peciallythoseprocessorswithextendedinstructionssetsforgraphicssupport

e.g.UltraSparc

etc.

-RRB-

willdo

weplantoportandrunthealgorithmspresented

Exploringdifferentarchitecturesisanot

herfuturedirection.Workonsoftwaresharedmemorysystemshasshownthatmedium

scalesystemsthatsupportasharedmemoryabstractioninsoftwaremayprovideuse

rswithaneasytoprogramandhighlycosteffectivesolution.Investigatingthebe

havioroftheparalleldecoderunderthesesystems

forexamplesharedvirtualmem

oryinwhichcoherenceismaintainedatpagegranularity

isaninterestingproble

erstandingthestructureofthedecoder.WeareindebtedtoNECandparticularlyto

JamesPhilbinforgenerouslyprovidingtheChallengesystemweusedforthemeasur

ementsaswellastoKasinathAnupindiandHenryCejtinfortheirhelpinmanagingth

ediskspaceandthecputimeweused.WewouldalsoliketothankLiviuIftodeforhish

elpinusingtheChallengesystem.References

-LSB-

-RSB-

ShahriarM.Akramullah

IshfaqA

hmad

andMingL.Liou

ing

JournalofParallelandDistributedComputing

Nov.

-LRB-

-RRB-

-LSB-

-RSB-

ArgonneNationalLaboratory,9700SouthCallAvenue,Argonne,IL60439-48

ParmacsParallelProgrammingSystem

-LSB-

-RSB-

V.Bhaskaran

andK.Konstantinid

es

ImageandVideoCompressionStandards

AlgorithmsandArchitectures

Kluw

-LSB-

-RSB-

KouroshGharachorloo

DanLenoski

JamesLaudon

PhillipGibbons

AnoopGupt

a

andJohnHennessy

MemoryConsistencyandEventOrderinginScalableShared-M

emoryMultipro-cessors

May

-LSB-

-RSB-

StephenGoldschmidt

andHelenD

avis

TangoIntroductionandTutorial

ComputerSystemsLaboratory

Stanford

University

Stanford

-LSB-

-RSB-

AndyC.Hung

Portabl

eVideoResearchGroup

-LRB-

PRVG

-RRB-

Stan-fordUniversity

-LSB-

-RSB-

M.K.Kwong

P.T.P.Tang

andB.Lin

AReal-TimeMPEGSoftwareDecoderUsingaPortableMessa

ge-PassingLibrary

MathematicsandComputerScienceDivision,ArgonneNa-tio

-LSB-

-RSB-

W.Lee

J.Golston

R.J.Gove

andY.Ki

m

Real-timeMPEGVideoCodeconaSingle-ChipMultiprocessor

Proceedingsoft

heSPIE

DigitalVideoCompressiononPersonalComputers

Al-gorithmsandTechno

-LSB-

-RSB-

DanielLenoski

JamesLaudon

Kourosh

Ghara-chorloo

Wolf-DietrichWeber

AnoopGupta

JohnHennessy

MarkHorowitz

andMonicaLam

TheStanfordDashMultiprocessor

IEEECom-puter

Vol

No.

-LSB-

-RSB-

K.Shen

L.A.Rowe

andE.J.Delp

AparallelImplemen

FasterThanReal-Time

ProceedingsoftheSPIEconfe

renceonDigitalVideoCompression

Algo-rithmsandTechnologies

0February1995,SanJose,California,pp.407-418

-LSB-

-RSB-

PlateauResearchGroup

sGuide

ComputerScienceDivision

Univers

ityofCalifornia

Berkeley

-LSB-

-RSB-

KetalPatel

BrianC.Smith

andLawrenceA.Rowe

PerformanceofaSoftwareMPEGVideoDecoder

ComputerScienceDivision-EECS

UniversityofCalifornia

Berkeley

-LSB-

-RSB-

MPEGSoftwareSimulationGroup

MPEG

Encoder/Decoder

-LSB-

-RSB-

Y.Yu

andD.Anastassiou

Soft

P

roceedingsoftheSPIE,ConferenceonDigitalVideoCompressiononPer-sonalComp

uters

AlgorithmsandTechnologies,SanJose,CA,Feb.6-10

Vol

pp.

-LSB-

-RSB-

GenericCod-ingofMovingPicturesand

AssociatedAudio

RecommendationH

Seoul

No

v.

-LSB-

-RSB-

S.Eckart

andC.Fogg

ISO/IECMPEG

SoftwareVideoCodec

Procee

dingsoftheSPIEconferenceonDigitalVideoCompression

AlgorithmsandTechnol

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O