Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

MATS

BRORSSON

and

PER

STENSTR?M

Department

of

Computer

Engineering

Lund

University

P.O.

Box

LUND

Sweden

email

matsbror@dit.lth.se

tel

fax

December

Abstract

In

a

shared

memory

model

memory

accesses

remain

to

be

a

major

obstacle

in

the

design

of

large-scale

multiprocessors

Therefore

a

large

number

of

memory

management

schemes

with

varying

cost/performance

have

been

reported

in

the

literature

The

choice

as

to

pick

a

certain

scheme

is

guided

by

the

sharing

behaviour

of

the

anticipated

application

suite

for

the

machine

Since

the

designer

must

evaluate

the

performance

of

a

large

number

of

design

choices

for

a

large

number

of

applications

there

is

a

need

for

an

evaluation

methodology

that

makes

it

possible

to

effectively

scan

the

design

space

In

this

paper

we

present

such

a

qualitative

methodology

that

captures

and

analyses

the

sharing

behaviour

of

parallel

programs

in

relation

to

shared

memory

management

schemes

that

have

been

studied

The

key

contribution

is

the

identification

of

four

conceptual

parameters

Spatial

granularity

Degree

of

sharing

the

Access

mode

and

the

Temporal

granularity

These

parameters

enable

us

to

visualise

sharing

behaviour

in

an

easy-tounderstand

fashion

In

order

to

demonstrate

the

strength

of

the

methodology

we

analyse

the

sharing

behaviour

of

four

scientific

applications

three

of

which

are

taken

from

the

SPLASH

benchmark

suite

from

Stanford

University

We

use

the

qualitative

data

to

judge

what

memory

management

policy

is

most

appropriate

Introduction

The

shared

memory

programming

model

for

MIMD

multiprocessors

is

generally

believed

to

have

a

number

of

advantages

over

e.g.

the

message

passing

model

-LSB-

-RSB-

Among

them

can

be

noted

ease

of

programming

and

process

migration

However

in

order

to

achieve

high

performance

a

major

obstacle

is

to

support

this

model

with

effective

shared

memory

access

Several

multiprocessor

architectures

can

support

shared

memory

but

we

will

in

the

following

consider

an

architecture

with

a

local/remote

shared

memory

organisation

-LRB-

see

Figure

-RRB-

The

shared

memory

is

distributed

over

a

number

of

processing

elements

which

communicate

via

an

interconnection

network

Each

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

processing

element

also

contains

a

cache

memory

This

type

of

architecture

is

a

Non-Uniform

Memory

Access

-LRB-

NUMA

-RRB-

architecture

in

which

local

memory

is

less

costly

to

access

than

remote

memory

This

non-uniformity

in

access

times

makes

it

very

important

to

place

and

migrate

shared

data

well

in

the

distributed

shared

memory

We

will

for

the

rest

of

this

paper

equal

the

word

process

with

processing

element

-LRB-

or

processor

-RRB-

since

sharing

within

a

processing

element

does

not

pose

any

problems

on

the

memory

system

Numerous

approaches

have

been

taken

in

order

to

speed

up

the

average

access

times

to

the

shared

memory

of

local/remote

architectures

-LRB-

see

e.g.

-LSB-

-RSB-

and

-LSB-

-RSB-

-RRB-

They

encompass

both

cache

coherence

maintenance

and

virtual

page

level

management

The

trend

in

these

policies

has

been

to

make

some

assumptions

on

program

behaviour

in

order

to

reduce

hardware

or

software

complexity

and

implementation

cost

while

still

retaining

good

performance

Their

performance

have

usually

been

evaluated

by

test

implementations

on

real

machines

or

simulators

and

then

by

running

a

representative

set

of

workload

When

designing

the

memory

system

for

a

local/remote

architecture

the

design

space

is

quite

large

and

in

order

to

choose

a

simple

but

still

effective

shared

memory

management

policy

one

needs

a

good

understanding

of

the

sharing

behaviour

of

the

applications

that

are

to

be

supported

The

sharing

behaviour

is

absolutely

crucial

to

most

policies

and

if

not

taken

into

account

can

lead

to

a

design

that

performs

poorly

The

usual

way

of

evaluating

memory

policies

by

making

test

implementations

is

very

time

consuming

and

costly

to

perform

even

if

they

are

done

by

means

of

simulations

In

this

paper

we

define

some

concepts

for

reasoning

about

sharing

behaviour

in

relation

to

the

performance

of

shared

memory

management

We

have

found

that

there

are

four

critical

parameters

for

describing

the

sharing

behaviour

the

Spatial

granularity

the

Degree

of

sharing

the

Access

mode

and

the

Temporal

granularity

These

parameters

are

defined

in

Section

These

conceptual

parameters

are

used

in

a

new

methodology

to

study

the

sharing

behaviour

of

real

life

applications

for

the

purpose

of

qualitatively

evaluate

the

relative

performance

of

various

shared

memory

management

policies

The

methodology

includes

an

analysis

tool

which

is

fed

by

memory

references

from

an

effective

RISC-based

shared

memory

multiprocessor

Based

on

the

stream

of

memory

references

which

is

analysed

the

sharing

behaviour

is

visualised

in

three

types

of

diagrams

using

the

concepts

of

the

parameters

mentioned

above

the

Exclusive/Read

Only

profile

the

Sharing

profile

and

the

Mobility

matrix

LM

P

C

LM

P

C

LM

P

C

LM

P

C

NETWORK

Figure

The

NUMA

shared

memory

multiprocessor

Processor

Cache

Memory

Local

memory

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

We

have

then

used

this

methodology

for

a

study

of

the

sharing

behaviour

of

four

scientific

applications

three

of

which

are

taken

from

the

SPLASH

parallel

benchmark

suite

from

Stanford

University

-LSB-

-RSB-

The

rest

of

the

paper

is

outlined

as

follows

First

in

Section

some

shared

memory

management

policies

are

presented

and

compared

regarding

implementation

cost

We

then

discuss

the

critical

parameters

of

sharing

behaviour

and

its

relation

to

memory

management

policies

in

Section

Section

presents

the

methodology

used

for

capturing

and

visualising

the

sharing

behaviour

Section

presents

the

application

programs

which

have

been

analysed

and

the

result

of

the

analysis

is

shown

in

Section

The

paper

is

concluded

after

a

section

on

related

work

Shared

memory

management

policies

The

address

space

of

a

program

can

be

divided

into

three

main

parts

Instructions

Private

data

and

Shared

data

If

we

disregard

process

migration

issues

instructions

and

private

data

are

trivially

managed

without

the

concern

of

maintaining

consistency

We

will

thus

in

the

rest

of

the

paper

focus

on

the

shared

data

portion

of

a

program?s

address

space

Several

varieties

of

shared

memory

management

policies

with

varying

requirements

on

hardware

and

software

complexity

have

been

proposed

in

the

literature

-LRB-

see

e.g.

-LSB-

-RSB-

and

-LSB-

-RSB-

-RRB-

They

can

principally

be

classified

according

to

two

criteria

the

block

size

they

operate

on

and

whether

they

support

replication

of

blocks

or

not

Common

for

them

all

is

that

their

performance

is

highly

dependent

upon

the

sharing

behaviour

of

the

workload

programs

If

replication

is

not

supported

the

policy

migrates

the

block

to

where

it

is

mostly

needed

according

to

its

algorithm

When

blocks

are

replicated

there

is

a

need

for

maintaining

coherence

between

the

copies

Coherence

is

usually

maintained

either

by

write-invalidate

where

all

copies

are

invalidated

upon

a

write

operation

to

the

block

or

by

write-update

where

the

copies

are

updated

with

the

written

value

We

will

in

this

paper

only

consider

the

write-invalidate

coherence

mechanism

In

order

to

span

the

design

space

we

give

a

few

examples

of

shared

memory

management

policies

of

different

hardware

and

software

complexity

The

number

of

processors

-LRB-

and

thus

the

number

of

caches

in

our

model

-RRB-

is

denoted

by

N.

Cache

block

management

Full-map

directory

cache

coherence

schemes

The

cache

coherence

scheme

for

the

DASH

multiprocessor

is

an

example

of

a

fullmap

directory

scheme

which

has

been

built

and

evaluated

-LSB-

-RSB-

A

directory

on

each

processing

element

stores

the

identity

of

the

caches

holding

a

copy

of

a

memory

block

Replication

is

supported

since

there

can

be

as

many

copies

of

a

memory

block

as

there

are

caches

The

overhead

in

storing

the

state

information

is

Q

-LRB-

-RRB-

Limited

directory

cache

coherence

schemes

Agarwal

et

al.

proposed

a

limited

directory

scheme

which

allows

a

restricted

number

of

copies

of

a

given

memory

block

-LSB-

-RSB-

The

required

state

information

is

reduced

to

Q

-LRB-

NlogN

-RRB-

The

replication

is

limited

which

can

lead

to

performance

degradation

if

the

degree

of

sharing

is

larger

than

the

number

of

allowed

copies

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

One-pointer

limited

directory

cache

coherence

schemes

A

special

case

of

the

limited

directory

scheme

above

is

when

only

one

copy

is

allowed

-LSB-

-RSB-

This

is

a

very

cost

effective

scheme

in

terms

of

hardware

but

requires

that

processors

access

data

in

a

migratory

fashion

in

order

to

work

well

-LSB-

-RSB-

Software-based

cache

coherence

schemes

Cheong

and

Veidenbaum

have

proposed

a

software-based

cache

coherence

scheme

in

which

the

programmer

-LRB-

or

the

compiler

-RRB-

inserts

coherence

operations

in

the

code

-LSB-

-RSB-

Such

operations

include

selective

cache

invalidation

and

cache-on

and

off

Replication

is

limitedly

supported

This

is

the

least

complex

scheme

to

implement

in

terms

of

hardware

but

it

puts

the

most

demand

on

the

programmer

-LRB-

compiler

-RRB-

and

the

sharing

behaviour

of

the

programs

Page

level

memory

management

Page

migration

policies

Scheurich

and

Dubois

have

proposed

a

page

migration

mechanism

based

on

hardware

counters

for

each

physical

page

in

the

system

-LSB-

-RSB-

A

page

migrates

when

there

is

imbalance

between

local

and

remote

references

Policies

like

this

can

be

shown

to

be

competitive

-LSB-

-RSB-

but

require

substantial

support

in

hardware

Page

replication

policies

LaRowe

and

Ellis

has

implemented

and

compared

a

number

of

page

replication

strategies

in

conjunction

with

migration

schemes

-LSB-

-RSB-

In

their

schemes

the

operating

system

is

responsible

for

making

the

decision

upon

when

and

which

page

to

replicate

and

no

extra

hardware

but

the

usual

for

virtual

memory

management

is

required

The

design

space

presents

a

large

variety

of

shared

memory

management

policies

to

use

in

order

to

reduce

the

average

latency

times

to

the

shared

memory

The

goal

is

to

use

the

least

complex

policy

while

still

maintain

good

performance

The

choice

is

dependent

on

the

sharing

behaviour

of

the

anticipated

programs

Some

programs

as

will

be

shown

later

do

not

have

the

potential

at

all

to

exploit

page

level

migration/replication

for

instance

For

other

programs

it

is

highly

probable

that

a

page

migration/replication

policy

will

work

well

but

the

actual

choice

is

a

function

of

implementation

parameters

Software

based

policies

will

work

for

programs

that

do

not

exhibit

much

dynamic

changes

in

the

sharing

behaviour

while

the

opposite

requires

more

aggressive

implementations

In

the

next

section

we

will

review

the

critical

parameters

for

characterizing

programs

according

to

their

sharing

behaviour

These

parameters

are

used

to

visualise

the

behaviour

in

order

to

restrict

the

design

space

of

shared

memory

policies

to

ones

that

really

have

potential

to

good

performance

Critical

parameters

for

shared

memory

management

We

will

here

discuss

the

critical

parameters

of

a

program

in

order

to

qualitatively

reason

about

the

performance

of

the

shared

memory

management

policies

presented

above

We

will

consider

the

costs

involved

in

supporting

the

policies

and

relate

those

to

the

program

sharing

behaviour

Cost

in

this

context

is

defined

to

be

the

time

needed

to

perform

a

policy

operation

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

The

shared

address

space

is

partitioned

in

blocks

of

size

S.

Consider

the

memory

references

issued

by

all

processors

to

a

block

for

a

time

period

T.

Four

critical

parameters

have

been

found

regarding

the

sharing

of

a

block

The

Degree

of

sharing

is

determined

by

the

number

of

processing

elements

issuing

references

to

the

block

during

T

The

Access

mode

can

either

be

Read

Only

or

Read/Write

depending

on

which

operations

are

made

on

the

block

The

Temporal

granularity

is

the

largest

T

during

which

the

block

is

accessed

with

the

same

degree

of

sharing

and

access

mode

Likewise

the

Spatial

granularity

is

the

largest

S

during

a

time

period

for

which

the

block

is

accessed

with

the

same

degree

of

sharing

and

access

mode

First

consider

the

spatial

granularity

A

small

spatial

granularity

suggests

that

we

can

employ

a

cache

based

policy

while

a

large

spatial

granularity

implies

that

a

page

level

mechanism

would

also

work

well

depending

on

the

degree

of

sharing

the

access

mode

and

the

temporal

granularity

A

large

spatial

granularity

is

desired

in

order

to

take

advantage

of

prefetching

effects

and

for

minimising

state

information

Next

look

at

the

degree

of

sharing

Four

classes

of

memory

blocks

according

to

their

degree

of

sharing

are

defined

Exclusive

Blocks

which

are

exclusively

referenced

by

one

processing

element

Majority

Shared

More

than

K

of

the

references

to

these

blocks

originate

from

one

processing

element

K

Shared

by

few

Only

a

small

number

-LRB-

F

F

-RRB-

processing

elements

make

references

to

these

blocks

Shared

by

many

These

blocks

are

actively

referenced

by

several

processors

-LRB-

number

of

processors

F

-RRB-

If

a

majority

of

the

accesses

to

the

shared

memory

are

to

Exclusive

blocks

migration

based

policies

have

a

potential

to

perform

well

For

cache

coherence

policies

the

decision

mechanism

is

simple

as

soon

as

a

processor

makes

an

access

to

a

block

it

is

migrated

to

the

corresponding

cache

For

page

level

policies

we

can

adopt

more

sophisticated

decision

algorithms

possibly

implemented

in

software

to

determine

when

an

imbalance

between

the

references

from

different

processing

elements

should

cause

a

page

migration

In

our

terminology

this

means

that

Majority

Shared

pages

have

a

potential

to

benefit

from

page

migration

policies

If

a

particular

migration

policy

will

work

well

or

not

is

dependent

on

the

costs

associated

with

block

-LRB-

cache

line

or

page

-RRB-

migration

When

the

temporal

granularity

is

larger

by

some

factor

than

the

cost

of

migrating

a

block

the

migration

policy

will

work

This

factor

and

the

cost

can

however

in

practical

situations

be

hard

to

determine

For

a

higher

degree

of

sharing

-LRB-

Shared

by

few

or

Shared

by

many

-RRB-

we

may

consider

replication

policies

Consider

the

write-invalidate

coherence

mechanism

We

must

then

require

a

Read

Only

access

mode

for

sufficient

temporal

granularity

in

order

to

make

replication

policies

work

As

soon

as

a

write

operation

occurs

the

other

copies

are

invalidated

and

we

end

up

with

an

Exclusive

block

for

some

time

However

if

the

temporal

granularity

is

large

enough

to

defend

the

cost

of

replicating

data

we

can

employ

a

replication

algorithm

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

A

large

spatial

granularity

S

and

a

large

temporal

granularity

T

is

desired

for

both

replication

and

migration

policies

This

releases

the

requirements

on

the

shared

memory

management

system

Figure

and

Figure

summarise

the

relationships

between

program

sharing

behaviour

and

applicable

shared

memory

management

policies

Figure

relates

to

page

level

policies

while

Figure

shows

the

corresponding

relationships

for

cache

coherence

policies

Consider

a

specific

block

in

memory

The

leaves

in

the

trees

in

the

figures

will

then

correspond

to

what

shared

memory

management

policy

is

appropriate

for

that

block

In

this

context

MCOST

and

RCOST

are

the

implementation

dependent

thresholds

for

which

migration

and

replication

of

the

block

is

beneficial

No

policy

and

No

cache

means

that

this

block

will

not

benefit

from

a

page

level

or

cache

coherence

policy

No

policy

Large

spatial

granularity

-LRB-

page

level

policies

-RRB-

Temporal

granularity

Exclusive

Majority

Shared

R/W

Read

Only

Page

replication

Figure

Relationship

between

sharing

behaviour

and

page

level

policies

Temporal

granularity

Access

mode

Degree

of

sharing

Page

migration

No

policy

No

policy

T

MCOST

T

RCOST

T

RCOST

T

MCOST

Small

spatial

granularity

-LRB-

cache

line

policies

-RRB-

Degree

of

sharing

Access

mode

Exclusive

R/W

Read

Only

R/W

Read

Only

Access

mode

Temporal

granularity

Temporal

granularity

full

map

directory

Temporal

granularity

Figure

Relationship

between

sharing

behaviour

and

cache

coherence

policies

No

cache

No

cache

No

cache

No

cache

No

cache

Limited

directory

one-pointer

Limited

directory

T

MCOST

T

RCOST

T

MCOST

T

RCOST

T

RCOST

T

RCOST

Shared

by

few

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

The

common

case

regarding

accesses

to

blocks

will

be

the

determining

factor

when

deciding

which

policy

is

the

best

for

a

particular

program

Similarly

the

majority

of

the

anticipated

programs

will

together

with

a

function

of

architectural

and

implementation

parameters

such

as

latency

times

and

network

bandwidth

determine

the

memory

policy

chosen

for

the

multiprocessor

The

next

section

describes

how

we

can

visualise

the

sharing

behaviour

using

the

conceptual

parameters

defined

in

this

section

Capturing

and

visualising

sharing

behaviour

As

mentioned

in

the

previous

section

there

are

four

key

parameters

of

sharing

behaviour

related

to

shared

memory

management

policies

Spatial

granularity

the

Degree

of

sharing

the

Access

mode

and

the

Temporal

granularity

This

section

describes

a

methodology

by

which

to

capture

and

visualise

these

critical

parameters

in

an

application

program

Analysis

of

reference

streams

In

order

to

analyse

the

sharing

behaviour

of

a

program

we

take

a

reference

stream

merged

from

all

processing

elements

and

put

it

into

an

analysis

tool

Each

memory

reference

is

assumed

to

be

accompanied

with

its

address

the

identification

of

the

processor

issuing

the

reference

type

of

reference

-LRB-

Read

Write

-RRB-

and

the

global

time

No

assumption

is

made

on

the

generation

of

the

reference

streams

as

long

as

the

required

information

is

provided

with

each

memory

reference

However

instead

of

taking

any

of

the

existing

multiprocessor

traces

available

we

are

using

a

functional

simulator

of

a

shared

memory

multiprocessor

-LSB-

-RSB-

as

our

memory

reference

generator

This

removes

the

need

for

storing

large

traces

on

disk

and

gives

us

more

flexibility

However

the

most

important

factor

was

the

ability

to

use

a

modern

processor

technology

in

an

environment

we

can

fully

control

from

source

code

down

to

memory

references

The

structure

of

this

system

is

shown

in

Figure

The

memory

reference

generator

consists

of

a

number

of

simulated

SPARC

processors

ideally

connected

to

a

common

shared

memory

Time

is

measured

in

processor

cycles

and

during

one

processor

cycle

each

processor

issues

exactly

one

memory

reference

An

instruction

cycle

thus

takes

from

one

to

three

SPARC

is

a

registered

trademark

of

SPARC

International

Inc.

P

P

P

P

Memory

Data

gathering

Figure

Memory

reference

generator

and

analysis

tool

and

analysis

Memory

reference

generator

Analysis

tool

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

processor

cycles

one

for

instruction

fetch

and

if

it

is

a

load/store

operation

one

ore

two

additional

accesses

depending

on

the

length

of

the

operand

A

round-robin

algorithm

is

used

at

the

memory

to

service

these

references

An

important

feature

of

this

reference

generator

is

its

effectiveness

which

allows

us

to

execute

complete

applications

which

generate

several

hundreds

of

million

memory

references

in

reasonable

time

It

can

be

argued

that

the

uniform

memory

access

time

affects

the

analysis

of

the

memory

references

We

have

made

measurements

where

the

unity

memory

latency

has

been

replaced

by

an

average

value

and

the

actual

latency

is

determined

with

a

random

generator

for

each

memory

reference

The

sharing

behaviour

was

not

found

to

be

different

with

the

non-uniform

latencies

compared

to

the

unity

latency

The

analysis

tool

monitors

the

references

sent

to

the

memory

References

to

the

shared

portion

of

the

memory

are

identified

and

sent

as

input

to

the

analysis

tool

For

the

purpose

of

the

analysis

the

shared

memory

is

divided

into

byte

sized

lines

which

is

presumed

to

be

the

smallest

block

size

on

which

to

perform

shared

memory

management

With

each

memory

line

two

counters

for

each

processing

element

are

associated

One

for

read

references

and

one

for

write

references

For

each

shared

memory

reference

exactly

one

counter

is

incremented

the

one

which

corresponds

to

a

read

or

write

operation

from

the

issuing

processing

element

At

certain

time

intervals

-LRB-

measured

in

processor

cycles

-RRB-

the

counters

for

each

line

are

sampled

and

analysed

in

order

to

determine

the

class

of

the

line

for

the

past

interval

according

to

the

four

classes

presented

in

Section

We

call

these

intervals

sampling

intervals

Each

class

is

separated

into

two

groups

reflecting

the

access

mode

-LRB-

Read

Only

Read/Write

-RRB-

A

block

is

regarded

as

Majority

Shared

if

more

than

K

of

the

references

are

issued

by

a

single

processing

element

Similarly

a

block

is

Shared

by

few

if

F

F

processors

have

accessed

the

block

during

the

sampling

interval

In

order

to

classify

memory

blocks

larger

than

a

line

adjacent

lines

a

grouped

to

form

larger

blocks

The

counters

of

lines

for

instance

are

combined

to

yield

information

on

byte

blocks

After

all

blocks

of

different

sizes

have

been

classified

the

corresponding

counters

are

zeroed

and

a

new

sampling

interval

is

started

The

mobility

of

a

block

between

classes

is

also

determined

by

keeping

track

of

the

class

the

block

was

classified

at

the

last

sampling

point

For

each

block

size

there

is

a

counter

for

each

class

counting

the

number

of

shared

memory

references

that

have

been

made

to

blocks

of

this

size

and

class

When

the

program

has

finished

executing

these

counters

are

divided

with

the

total

number

of

shared

memory

references

resulting

in

the

fraction

of

memory

references

made

to

blocks

of

this

size

and

class

This

data

is

then

shown

in

diagrams

for

varying

sampling

intervals

and

block

sizes

as

described

in

Section

The

next

section

discusses

an

example

program

used

to

illustrate

the

methodology

for

capturing

and

visualising

program

sharing

behaviour

Example

program

Figure

shows

a

fragment

of

a

program

designed

to

reference

an

array

so

that

the

array

is

divided

into

migratory

objects

A

migratory

object

can

possibly

be

accessed

by

all

processors

but

at

any

given

time

only

by

a

single

processor

-LSB-

-RSB-

During

the

execution

of

the

program

the

object

will

thus

migrate

from

one

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

processing

element

to

another

given

a

shared

memory

management

policy

based

on

migration

-LSB-

-RSB-

The

program

is

written

in

the

Single-Program-Multiple-Data

-LRB-

SPMD

-RRB-

model

-LSB-

-RSB-

in

which

all

processors

execute

the

same

code

The

variable

myID

contains

the

identity

of

the

current

processor

which

is

used

to

partition

the

arrays

between

the

processors

In

the

program

there

are

three

shared

data

structures

The

arrays

glob

a

and

glob

b

and

a

barrier

synchronisation

variable

Only

references

to

these

are

considered

by

the

analysis

tool

The

example

program

is

designed

to

have

the

following

access

patterns

During

one

iteration

of

the

inner

loop

each

processor

makes

two

reads

to

array

glob

b

and

two

writes

to

array

glob

a

-LRB-

see

Figure

-RRB-

Array

glob

b

is

thus

read

only

The

write

operations

to

glob

a

are

within

each

processors

exclusive

portion

of

the

array

Each

other

iteration

of

the

outer

loop

also

accesses

glob

b

exclusively

but

each

other

iteration

one

of

the

read

operations

are

to

the

neighbouring

processing

elements

portion

of

glob

b

which

means

that

for

this

iteration

two

processing

elements

will

share

this

part

of

the

array

The

portions

of

the

arrays

migrate

times

with

a

barrier

synchronisation

between

each

migration

to

ensure

that

there

is

no

skew

between

the

processors

which

all

execute

the

same

code

The

example

program

has

been

executed

on

processors

with

an

array

consisting

of

entries

of

two

bytes

each

Each

processor

thus

references

a

portion

of

bytes

each

during

an

iteration

An

iteration

is

measured

to

take

around

processor

cycles

In

summary

we

expect

for

all

block

sizes

and

for

as

long

sampling

intervals

as

possible

that

about

half

of

the

shared

references

have

access

mode

Read

Only

We

also

expect

that

for

blocks

smaller

than

bytes

-LRB-

the

program?s

spatial

granularity

-RRB-

and

for

sampling

intervals

less

than

-LRB-

the

programs

temporal

granularity

-RRB-

an

overwhelming

majority

of

Read/Write

references

will

be

to

Exclusive

blocks

n

length/nprocs

Each

processors

gets

n

elements

of

the

array

-LRB-

myID

-RRB-

k

myID

for

-LRB-

i

i

i

-RRB-

-LCB-

if

-LRB-

-LRB-

k

-RRB-

nprocs

-RRB-

-LCB-

offset

k

n

Determine

wrap-around

offset

-RCB-

else

-LCB-

offset

n

offset

to

the

neighbouring

processor

-RCB-

for

-LRB-

j

k

n

j

-LRB-

k

-RRB-

n

j

-RRB-

-LCB-

if

-LRB-

-LRB-

i

-RRB-

-RRB-

-LCB-

glob

a

-LSB-

j

-RSB-

glob

b

-LSB-

j

-RSB-

glob

b

-LSB-

j

-RSB-

will

be

shared

by

glob

a

-LSB-

j

-RSB-

glob

b

-LSB-

j

offset

-RSB-

but

still

read

only

-RCB-

else

-LCB-

glob

a

-LSB-

j

-RSB-

glob

b

-LSB-

j

-RSB-

Both

glob

a

-LSB-

j

-RSB-

and

glob

b

-LSB-

j

-RSB-

glob

a

-LSB-

j

-RSB-

glob

b

-LSB-

j

-RSB-

are

exclusively

accessed

-RCB-

-RCB-

BARRIER

-LRB-

glob

BA

nprocs

-RRB-

Wait

for

all

other

processors

k

-LRB-

k

-RRB-

nprocs

Migrate

my

part

of

the

array

to

next

processor

-RCB-

Figure

Example

program

fragment

exemplifying

exclusive

references

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

Similarly

about

half

of

the

references

in

Read

Only

access

mode

should

be

to

Exclusive

blocks

and

the

other

half

to

Shared

by

few

blocks

The

next

section

will

show

that

this

is

really

the

case

based

on

the

execution

of

the

example

program

and

analysis

of

its

shared

memory

references

Data

presentation

In

this

section

we

will

illustrate

the

visualisation

of

sharing

that

is

used

in

the

rest

of

the

paper

The

introduction

of

the

diagrams

are

made

in

the

order

of

stepwise

refinement

First

we

show

a

diagram

which

will

give

a

first

idea

of

whether

a

replication

or

migration

memory

policy

will

work

at

all

Then

we

will

look

at

a

particular

sampling

interval

to

get

more

information

of

the

sharing

behaviour

of

that

particular

temporal

granularity

Finally

we

will

study

the

dynamic

behaviour

of

blocks

of

a

particular

size

by

looking

at

the

mobility

between

classes

Figure

shows

how

the

portion

of

references

to

Exclusive

and

Read

Only

blocks

changes

with

varying

block

sizes

and

sampling

intervals

for

the

example

program

This

is

called

the

Exclusive/Read

Only

profile

As

expected

we

see

that

at

least

of

the

references

are

to

Exclusive/Read

Only

blocks

We

also

see

that

as

long

as

the

sampling

interval

is

less

than

the

expected

temporal

granularity

-LRB-

processor

cycles

-RRB-

and

the

block

size

is

less

than

the

expected

spatial

granularity

-LRB-

bytes

-RRB-

the

fraction

of

references

to

Exclusive/Read

Only

blocks

is

overwhelming

-LRB-

-RRB-

The

Exclusive/Read

Only

profile

gives

us

a

hint

that

simple

memory

management

policies

will

work

well

for

at

least

half

of

the

references

It

also

indicates

that

replication

or

migration

policies

that

use

block

sizes

less

than

bytes

and

can

take

advantage

of

a

temporal

granularity

less

than

processor

cycles

will

work

very

well

More

detailed

information

on

the

sharing

is

given

by

Figure

which

shows

the

sharing

profile

of

the

example

program

for

a

sampling

interval

of

processor

cycles

Read

Only

and

Read/Write

access

Figure

References

to

Exclusive

and

Read

Only

blocks

for

varying

sampling

intervals

and

block

sizes

the

Exclusive/Read

Only

profile

migr

Portion

of

References

to

Exclusive

and

Read

Only

Blocks

Block

Size

Sampling

Interval

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

modes

are

denoted

RO

and

RW

respectively

in

the

diagrams

In

this

diagram

we

can

see

that

percent

of

the

references

are

Read

Only

as

expected

from

the

example

program

We

can

also

see

that

about

half

of

these

are

to

Exclusive

blocks

and

the

other

half

to

Shared

by

few

blocks

As

long

as

the

block

size

is

less

than

bytes

the

rest

of

the

references

are

also

as

expected

to

Exclusive

Read/Write

blocks

For

block

sizes

of

bytes

-LRB-

the

spatial

granularity

-RRB-

there

is

for

both

Read

Only

blocks

and

for

Read/Write

blocks

a

significant

amount

of

references

to

Majority

Shared

blocks

which

is

due

to

the

sharing

that

occurs

when

a

block

migrates

to

another

processor

Remember

that

the

iteration

time

is

about

processor

cycles

The

migration

is

done

either

on

the

first

half

of

a

sampling

interval

in

which

case

the

next

processor

in

turn

will

issue

most

of

the

references

to

the

block

during

this

sampling

interval

or

during

the

latter

half

of

the

period

in

which

case

the

later

processor

accesses

it

the

most

In

either

case

More

than

of

the

references

are

likely

to

come

from

one

processing

element

to

this

block

and

it

will

subsequently

be

classified

as

Majority

Shared

For

larger

blocks

we

can

see

that

first

there

are

just

a

few

processors

sharing

the

blocks

and

for

even

larger

blocks

many

processors

share

the

block

This

is

just

according

to

how

this

example

program

was

written

With

a

block

size

of

bytes

there

has

to

be

at

least

two

processors

accessing

it

actively

Similarly

for

kilobyte

blocks

there

has

to

be

at

least

processors

sharing

the

block

which

is

more

than

a

few

-LRB-

-RRB-

A

small

portion

of

the

shared

references

-LRB-

-RRB-

are

always

to

Read/Write

Shared

by

many

blocks

These

are

references

to

the

barrier

synchronisations

which

are

accessed

by

all

processors

As

was

noted

above

the

sharing

profile

shows

a

great

deal

of

references

to

Majority

Shared

blocks

of

bytes

According

to

Figure

this

suggests

that

a

page

migration

policy

has

a

potential

to

perform

well

However

we

still

lack

some

information

on

the

dynamics

between

sampling

points

For

this

purpose

we

introduce

the

Mobility

Matrix

-LRB-

see

Figure

-RRB-

Figure

Classification

of

shared

memory

blocks

for

the

example

program

the

Sharing

Profile

RO

Read

Only

RW

Read/Write

migr

Sampling

Interval

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

Block

sizes

Portion

of

shared

references

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

The

mobility

matrix

shows

the

transitions

between

classes

at

sampling

points

A

bar

represents

the

portion

of

total

number

of

transition

from

one

class

to

another

class

A

bar

in

the

diagonal

indicates

the

amount

of

blocks

which

at

a

sampling

point

are

classified

in

the

same

block

as

they

were

the

last

time

In

Figure

we

first

note

that

all

transitions

are

within

each

access

mode

Read

Only

blocks

are

again

classified

as

Read

Only

blocks

-LRB-

but

perhaps

in

a

different

class

-RRB-

and

the

same

for

Read/Write

blocks

We

also

note

that

the

Shared

by

many

blocks

always

stay

in

that

class

for

the

whole

measuring

interval

For

Read

Only

blocks

we

can

note

a

rather

high

mobility

between

Exclusive

Majority

Shared

and

Shared

by

few

classifications

For

Read/Write

blocks

the

transitions

are

mainly

between

the

Exclusive

and

Majority

Shared

classes

This

gives

a

stronger

intuition

that

a

migration

policy

would

work

well

In

summary

the

Exclusive/Read

Only

profile

the

Sharing

profile

and

the

Mobility

matrix

give

a

good

intuition

of

the

inherent

sharing

in

programs

for

shared

memory

multiprocessors

By

a

quick

glance

on

the

Exclusive/Read

Only

profile

one

gets

an

understanding

on

the

spatial

and

the

temporal

granularity

which

are

perhaps

the

most

crucial

parameters

if

a

shared

memory

management

policy

will

work

at

all

For

more

detailed

information

one

looks

in

the

sharing

profile

and

in

the

mobility

matrix

in

which

all

classes

are

represented

The

next

section

describes

the

application

programs

used

in

our

study

and

Section

presents

the

sharing

behaviour

of

these

programs

Application

programs

In

order

to

exemplify

our

methodology

we

have

analysed

the

inherent

sharing

behaviour

of

four

parallel

scientific

applications

All

but

one

of

the

applications

are

taken

from

the

SPLASH

benchmark

suite

from

Stanford

-LSB-

-RSB-

The

programs

are

written

in

C

using

the

parmacs

macros

from

Argonne

National

Laboratories

to

express

parallelism

-LSB-

-RSB-

The

programs

have

been

compiled

with

the

gcc

compiler

using

the

optimization

flag

This

means

that

besides

code

restructuring

to

get

around

delayed

branches

as

many

variables

as

possible

are

put

in

registers

This

stresses

the

shared

memory

behaviour

more

since

the

portion

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

migr

Sampling

Interval

Block

Size

From

class

Figure

A

mobility

matrix

of

the

example

program

RO

Read

Only

RW

Read/Write

To

class

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

of

shared

memory

references

gets

higher

-LSB-

-RSB-

Shared

data

structures

are

marked

volatile

to

prevent

the

compiler

from

putting

these

variables

in

registers

The

model

of

parallelism

used

is

the

Single-Program-Multiple-Data

-LRB-

SPMD

-RRB-

model

of

execution

-LSB-

-RSB-

which

specifies

one

process

per

processing

element

All

processors

execute

the

same

program

and

the

work

is

dynamically

or

statically

partitioned

to

the

processes

Synchronisation

between

processes

are

made

through

barriers

and

between

barriers

critical

sections

can

be

protected

with

monitors

The

programs

generally

start

with

some

initialisation

performed

by

one

master

process

The

master

then

creates

as

many

slave

processes

as

there

are

processing

elements

The

processing

elements

execute

the

same

code

until

the

algorithm

has

finished

Some

post-processing

may

then

be

done

by

the

master

process

Table

summarises

the

programs

and

the

data

sets

The

programs

were

executed

using

processing

elements

Only

shared

references

of

the

parallel

section

of

the

programs

were

monitored

and

analysed

The

initialisation

and

post-processing

are

expected

to

be

insignificant

in

full-scale

runs

Figure

shows

the

general

reference

composition

of

the

applications

We

have

included

the

simple

example

program

from

Section

migr

as

a

comparison

First

note

that

the

portion

of

instructions

is

very

large

This

is

a

consequence

of

the

RISC-based

processor

used

in

the

memory

reference

generator

Data

references

are

only

made

through

explicit

load

or

store

instructions

and

most

private

variables

can

be

stored

in

registers

The

portion

of

shared

references

ranges

from

for

Water

to

as

most

a

little

more

than

for

Grid

which

may

seem

little

but

given

that

instructions

and

private

data

can

be

put

in

local

memory

and

will

have

a

very

high

cache

hit

ratio

the

average

memory

latency

time

might

very

well

be

dominated

by

the

shared

memory

references

Sharing

behaviour

of

the

benchmark

programs

This

section

presents

the

sharing

behaviour

of

the

applications

according

to

our

methodology

and

discusses

its

influence

on

shared

memory

management

Due

to

space

limitations

we

can

only

present

data

for

a

few

aspects

To

get

a

full

understanding

of

the

sharing

behaviour

requires

more

diagrams

than

we

have

presented

here

Table

Benchmark

programs

Program

Description

Data

set

Grid

An

iterative

solution

of

Laplace?s

equation

on

a

two-dimensional

grid

x

matrix

iterations

A

simulation

of

the

forces

exerted

on

a

body

in

a

flow

of

extremely

low

density

particles

steps

test

geometry

Water

A

simulation

of

the

forces

and

potentials

in

a

system

of

water

molecules

in

the

liquid

state

molecules

time

steps

Cholesky

Cholesky

factorization

of

a

sparse

matrix

matrix

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

The

sharing

behaviour

of

each

program

is

expressed

using

the

Exclusive/Read

Only

profile

the

Sharing

profile

and

the

Mobility

matrix

as

described

in

Section

For

each

program

the

sharing

profile

and

the

mobility

matrix

has

been

chosen

so

as

to

illustrate

a

particular

behaviour

with

respect

to

some

shared

memory

management

policy

in

mind

Grid

Figure

-LRB-

a

-RRB-

shows

the

Exclusive/Read

Only

profile

of

the

grid

program

It

indicates

a

spatial

granularity

of

around

bytes

and

a

temporal

granularity

of

processor

cycles

for

which

more

than

of

the

references

are

either

to

Exclusive

or

Read

Only

blocks

This

looks

promising

for

a

shared

memory

management

purposes

Let

us

take

a

closer

look

at

the

Sharing

profile

for

the

purpose

of

cache

coherence

strategies

Since

typical

latencies

for

coherence

operations

are

on

the

order

of

processor

cycles

-LSB-

-RSB-

we

need

the

temporal

granularity

to

be

sufficiently

larger

to

take

advantage

of

a

cache

coherence

strategy

Figure

-LRB-

b

-RRB-

shows

the

Sharing

profile

for

a

sampling

interval

of

processor

cycles

We

see

here

that

most

of

the

references

in

the

Exclusive/Read

Only

profile

are

to

Read

Only

blocks

Less

than

are

to

Exclusive

Read

Write

blocks

Seeing

that

this

behaviour

does

not

change

significantly

we

get

the

discouraging

result

that

only

around

of

the

references

to

shared

memory

will

be

helped

by

a

cache

coherence

algorithm

Since

the

majority

of

the

Read

Only

blocks

are

Exclusive

Majority

Shared

or

Shared

by

few

we

deduce

that

a

limited

directory

scheme

will

be

sufficient

to

satisfy

the

replication

needed

in

this

application

To

what

extent

the

references

to

the

Read

Only

blocks

will

benefit

from

a

cache

coherence

scheme

depends

on

whether

they

stay

Read

Only

or

not

Figure

-LRB-

c

-RRB-

which

shows

the

Mobility

matrix

for

byte

blocks

indicates

that

this

is

not

the

case

Most

of

the

changes

are

between

the

Read

Only

and

the

Read/Write

Exclusive

classes

This

is

however

not

destructive

for

the

cache

coherence

scheme

The

conclusion

made

is

that

a

limited

directory

scheme

has

the

potential

to

work

well

for

the

grid

program

However

only

some

of

the

references

will

take

great

advantage

of

it

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

cholesky

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

Shared

writes

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

Shared

reads

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAA

AAAAPrivate

writes

Private

reads

Instructions

Figure

General

reference

composition

of

the

application

programs

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

grid

32 192 192 1000

Sampling

Interval

Block

Size

-LRB-

c

-RRB-

Mobility

Matrix

Figure

Sharing

behaviour

of

grid

grid

32 192 192 1000

Sampling

Interval

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

Block

sizes

Portion

of

shared

references

-LRB-

b

-RRB-

Sharing

Profile

grid

32 192 192 1000

Portion

of

References

to

Exclusive

and

Read

Only

Blocks

Block

Size

Sampling

Interval

-LRB-

a

-RRB-

Exlusive/Read

Only

To

class

From

class

Profile

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

Sampling

Interval

Block

Size

-LRB-

c

-RRB-

Mobility

Matrix

Figure

Sharing

behaviour

of

Sampling

Interval

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

Block

sizes

Portion

of

shared

references

-LRB-

b

-RRB-

Sharing

Profile

Portion

of

References

to

Exclusive

and

Read

Only

Blocks

Block

Size

Sampling

Interval

-LRB-

a

-RRB-

Exlusive/Read

Only

From

class

To

class

Profile

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

The

Exclusive/Read

Only

profile

of

can

be

seen

in

Figure

-LRB-

a

-RRB-

No

clear

temporal

or

spatial

granularity

can

be

seen

but

for

small

sampling

intervals

and

for

small

sized

blocks

the

amount

of

references

to

Exclusive

and

Read

Only

block

is

large

For

page

level

block

sizes

there

are

almost

no

references

at

all

to

blocks

of

these

classes

which

indicates

that

page

level

migration

and/or

replication

will

not

work

for

However

block

of

cache

line

sizes

have

a

large

amount

of

Exclusive

or

Read

Only

references

Figure

-LRB-

b

-RRB-

shows

the

Sharing

profile

for

a

sampling

interval

of

processor

cycles

which

is

relevant

for

cache

coherence

policies

Looking

at

a

block

size

of

bytes

there

are

almost

no

Read

Only

references

at

all

but

still

a

large

-LRB-

-RRB-

amount

of

references

to

Exclusive

blocks

Another

of

the

references

are

to

Shared

by

few

blocks

The

Mobility

matrix

-LRB-

see

Figure

-LRB-

c

-RRB-

-RRB-

shows

that

the

mobility

is

rather

low

The

blocks

stay

Exclusive

with

a

small

number

of

transitions

between

Read/Write

and

Read

Only

Exclusive

In

summary

A

limited

directory

cache

coherence

scheme

seem

promising

also

for

but

page

level

migration

or

replication

will

not

give

a

performance

gain

Water

The

Exclusive/Read

Only

profile

-LRB-

see

Figure

-LRB-

a

-RRB-

-RRB-

exhibit

a

high

degree

of

references

to

Exclusive

and

Read

Only

blocks

for

a

wide

range

of

block

sizes

and

sampling

intervals

For

small

block

sizes

and

small

sampling

intervals

it

is

extremely

high

suggesting

that

cache

coherence

policies

have

a

high

potential

of

doing

well

Also

page

level

policies

should

be

capable

of

working

well

which

we

will

investigate

next

Figure

-LRB-

b

-RRB-

shows

the

Sharing

profile

for

a

sampling

interval

of

processor

cycles

-LRB-

relevant

for

page

level

mechanisms

-RRB-

Consider

a

page

size

of

bytes

-LRB-

which

Li

-LSB-

-RSB-

has

found

to

be

working

well

for

page

replication/migration

-RRB-

Almost

half

of

the

references

are

to

Read

Only

blocks

A

large

percentage

of

the

Read/Write

references

are

to

Exclusive

or

Majority

Shared

blocks

This

gives

an

indication

that

both

page

level

replication

and

migration

will

work

well

This

intuition

is

strengthened

by

the

Mobility

matrix

-LRB-

see

Figure

-LRB-

c

-RRB-

-RRB-

The

Read

Only

blocks

mostly

stay

that

way

even

if

there

are

a

few

transitions

between

Read

Only

blocks

and

Read/Write

Exclusive

blocks

There

are

quite

many

transitions

between

Read/Write

Exclusive

blocks

and

Read/Write

Majority

Shared

indicating

that

these

blocks

are

migratory

with

a

migration

frequency

between

and

processor

cycles

The

cost

of

page

migration

will

determine

its

success

for

the

migratory

blocks

The

Exclusive/Read

Only

profile

most

surely

guarantees

a

high

advantage

of

using

a

cache

coherence

strategy

and

we

have

a

high

confidence

of

that

page

level

migration

and

replication

will

be

beneficial

Cholesky

Figure

-LRB-

a

-RRB-

shows

the

Exclusive/Read

Only

profile

of

Cholesky

Also

in

this

case

there

is

a

large

amount

of

references

to

Exclusive

and

Read

Only

blocks

We

can

be

almost

certain

of

a

good

performance

of

a

cache

coherence

policy

The

policy

chosen

is

dependent

on

the

degree

of

sharing

and

some

other

factors

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

Water

Sampling

Interval

Block

Size

Figure

Sharing

behaviour

of

Water

-LRB-

c

-RRB-

Mobility

Matrix

Water

Sampling

Interval

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

Block

sizes

Portion

of

shared

references

-LRB-

b

-RRB-

Sharing

Profile

Water

Portion

of

References

to

Exclusive

and

Read

Only

Blocks

Block

Size

Sampling

Interval

-LRB-

a

-RRB-

Exlusive/Read

Only

To

class

From

class

Profile

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

cholesky

Sampling

Interval

Block

Size

Figure

Sharing

behaviour

of

Cholesky

-LRB-

c

-RRB-

Mobility

Matrix

cholesky

Sampling

Interval

RO-Exclusive

RO-Majority

Shared

RO-Shared

by

few

RO-Shared

by

many

RW-Exclusive

RW-Majority

Shared

RW-Shared

by

few

RW-Shared

by

many

Block

sizes

Portion

of

shared

references

-LRB-

b

-RRB-

Sharing

Profile

cholesky

Portion

of

References

to

Exclusive

and

Read

Only

Blocks

Block

Size

Sampling

Interval

-LRB-

a

-RRB-

Exlusive/Read

Only

To

class

From

class

Profile

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

which

can

be

seen

in

the

appropriate

sharing

profiles

We

will

here

look

at

possibilities

for

page

level

migration/replication

The

sharing

profile

for

a

sampling

interval

of

processor

cycles

can

be

seen

in

Figure

-LRB-

b

-RRB-

Consider

a

page

size

of

bytes

First

we

note

that

almost

half

of

the

references

are

to

Read

Only

blocks

Then

we

see

that

a

considerable

amount

of

references

-LRB-

-RRB-

are

to

Read/Write

Exclusive

or

Majority

Shared

blocks

Both

these

facts

suggest

that

page

level

migration

and

replication

has

a

potential

of

good

performance

The

transitions

between

the

classes

for

blocks

of

bytes

are

very

low

-LRB-

see

Figure

-LRB-

c

-RRB-

-RRB-

At

all

sampling

points

most

Exclusive

blocks

-LRB-

both

Read

Only

and

Read/Write

-RRB-

keep

their

classification

indicating

a

rather

static

behaviour

This

reinforces

the

conclusion

from

looking

at

the

sharing

profile

that

page

level

migration

and

replication

will

perform

well

Discussion

We

have

in

sections

to

presented

some

aspects

on

the

sharing

behaviour

of

the

benchmark

applications

used

in

our

study

To

get

the

full

picture

one

needs

to

look

more

closely

on

several

sharing

profiles

which

we

have

to

omit

due

to

space

limitations

However

with

these

applications

as

a

starting

point

we

can

conclude

that

a

limited

directory

scheme

will

work

well

for

most

applications

-LRB-

with

the

possible

exception

of

grid

for

which

only

some

of

the

references

would

benefit

from

any

cache

coherence

policy

-RRB-

We

can

also

conclude

that

page

level

replication

will

work

well

for

most

applications

due

to

the

high

degree

of

Read

Only

blocks

of

page

size

The

performance

gain

from

page

migration

are

more

questionable

but

well

worth

investigating

Guidance

on

the

requirements

on

such

implementations

can

be

given

from

the

Sharing

profiles

and

the

Mobility

matrices

These

results

are

well

in

accordance

with

a

performance

study

made

using

some

of

these

applications

In

-LSB-

-RSB-

Water

and

Cholesky

was

used

to

investigate

fundamental

architectural

differences

in

NUMA

and

COMA

architectures

One

of

the

issues

investigated

was

the

effect

of

page

migration

-LRB-

however

not

page

replication

-RRB-

on

the

node

hit

ratio

As

we

have

stated

as

possible

here

there

was

a

significant

improvement

of

node

hit

ratio

for

Cholesky

and

almost

for

Water

did

improve

but

not

very

much

which

most

likely

is

due

to

a

very

successful

cache

coherence

management

Another

study

-LSB-

-RSB-

investigated

the

cache

invalidation

patterns

for

among

other

programs

and

Water

As

we

would

expect

from

our

measurements

the

number

of

copies

invalidated

at

a

write

operation

are

almost

always

only

one

This

means

that

at

most

two

processing

elements

have

copies

of

a

block

In

our

measurements

we

got

a

smaller

fraction

of

references

to

Exclusive

and

Shared

by

few

blocks

than

in

-LSB-

-RSB-

for

the

program

This

is

due

to

the

frequent

invalidation

misses

in

-LRB-

one

every

fifth

reference

-RRB-

The

next

section

will

review

some

of

the

related

work

before

we

conclude

the

paper

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

Related

work

While

most

studies

on

program

behaviour

of

parallel

programs

on

shared

memory

multiprocessors

have

concentrated

on

processor

scheduling

and

load

balancing

some

work

has

been

done

on

the

data

sharing

in

the

system

Most

of

these

studies

have

been

made

for

the

same

reason

as

this

that

the

sharing

behaviour

plays

an

important

role

for

the

performance

of

shared

memory

management

policies

One

of

the

first

studies

made

was

done

by

Darema-Rogers

et

al.

-LSB-

-RSB-

They

used

traces

of

three

scientific

programs

executed

on

an

eight-processor

simulator

Their

main

result

is

that

even

though

the

largest

portion

of

the

address

space

consists

of

shared

data

only

a

small

portion

of

the

actual

memory

references

are

to

the

shared

data

They

also

find

that

the

shared

data

references

come

in

bursts

However

they

do

not

address

the

degree

of

sharing

or

access

mode

nor

do

they

use

the

notion

of

spatial

granularity

at

all

Eggers

and

Katz

-LSB-

-RSB-

defined

a

write

run

to

be

a

sequence

of

write

references

to

a

shared

address

by

a

single

processor

It

is

initiated

by

the

first

write

operation

from

a

processor

and

terminated

by

the

first

access

of

another

process

-LRB-

read

or

write

-RRB-

They

performed

some

measurement

with

simulations

of

to

processors

The

write

run

model

was

developed

specifically

to

capture

sharing

in

order

to

compare

writeinvalidate

and

write-update

cache

coherence

policies

and

is

thus

not

suited

to

portray

the

general

sharing

behaviour

which

is

the

scope

of

this

paper

A

more

general

study

was

done

by

Agarwal

and

Gupta

-LSB-

-RSB-

which

studied

memory

reference

characteristics

of

some

applications

executed

under

MACH

on

a

processor

VAX

computer

They

use

the

notions

of

spatial

and

temporal

granularity

under

other

names

but

do

not

address

the

degree

of

sharing

more

than

looking

into

the

frequency

of

references

to

a

block

from

different

processors

They

do

however

include

operating

system

references

and

address

the

impact

of

process

migration

on

the

cache

coherence

protocol

Johnson

Baylor

and

Rathi

-LSB-

-RSB-

studied

the

memory

reference

behaviour

of

three

scientific

applications

on

a

simulated

multiprocessor

The

intent

was

to

study

the

inherent

sharing

in

the

applications

by

measuring

the

ownership

cycle

times

of

memory

blocks

ranging

from

to

bytes

The

ownership

cycle

time

relates

to

temporal

granularity

but

unfortunately

their

study

does

not

use

realistic

ownership

times

in

order

to

make

powerful

statements

on

shared

memory

management

policies

They

group

the

blocks

in

groups

according

to

ownership

cycle

times

ranging

from

processor

cycles

to

This

should

be

compared

with

actual

latency

times

in

real

systems

-LSB-

-RSB-

and

our

sampling

intervals

which

range

from

to

processor

cycles

An

old

processor

technology

or

a

non-optimizing

compiler

-LRB-

or

a

combination

thereof

-RRB-

on

their

part

is

a

probable

cause

for

the

substantial

difference

in

dynamic

reference

composition

in

their

study

compared

to

ours

A

little

more

than

half

of

the

references

in

their

measurements

were

instructions

which

should

be

compared

with

our

figure

around

Gallivan

et

al.

stresses

the

importance

of

studying

the

memory

reference

behaviour

in

order

to

make

statements

on

shared

memory

system

performance

-LSB-

-RSB-

However

they

do

not

use

real

workloads

and

the

type

of

multiprocessor

used

in

the

study

-LRB-

An

eight-processor

Alliant

-RRB-

does

not

exploit

much

sharing

They

do

however

use

the

notions

of

spatial

and

temporal

granularity

Lastly

Weber

and

Gupta

-LSB-

-RSB-

studied

the

cache

invalidation

patterns

of

scientific/engineering

applications

-LRB-

some

of

which

are

from

SPLASH

-RRB-

on

a

simulated

system

ranging

from

to

processors

The

link

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

cache

invalidations

to

high

level

data

objects

which

they

have

classified

in

five

classes

in

order

to

be

able

to

extrapolate

the

results

to

larger

machines

The

results

are

very

much

linked

to

the

particular

cache

design

including

the

cache

coherence

protocol

used

and

is

not

suitable

for

talking

about

general

sharing

behaviour

In

summary

apart

from

-LSB-

-RSB-

most

studies

which

have

been

made

are

closely

linked

to

a

particular

architectural

feature

that

are

to

be

studied

or

evaluated

Also

most

of

the

studies

were

made

for

a

small

number

of

processing

elements

We

have

found

that

the

sharing

behaviour

can

differ

significantly

between

a

few

number

of

processors

and

The

difference

between

and

processors

is

however

not

as

significant

There

is

a

fundamental

lack

in

the

study

of

the

critical

parameters

degree

of

sharing

access

mode

and

temporal

and

spatial

granularity

which

are

crucial

for

a

full

understanding

of

the

sharing

behaviour

of

a

program

However

there

seems

to

be

a

consensus

on

that

the

memory

reference

behaviour

in

general

and

the

sharing

behaviour

in

particular

are

of

crucial

importance

in

order

to

achieve

high

performance

memory

systems

Conclusions

The

sharing

behaviour

is

perhaps

the

most

important

consideration

when

designing

a

shared

memory

management

system

for

a

multiprocessor

Several

researchers

have

paid

attention

to

this

fact

-LSB-

-RSB-

but

there

has

been

a

lack

of

concepts

and

a

methodology

for

studying

the

sharing

behaviour

in

relation

to

shared

memory

management

policies

We

have

in

this

paper

presented

concepts

for

reasoning

about

sharing

behaviour

for

the

purpose

to

qualitatively

compare

different

memory

management

policies

on

both

cache

block

and

virtual

page

levels

We

have

also

presented

a

methodology

by

which

we

can

effectively

capture

the

sharing

behaviour

of

large

applications

The

sharing

behaviour

is

visualised

in

the

Exclusive/Read

Only

profile

the

Sharing

profile

and

in

the

Mobility

matrix

We

have

analysed

the

sharing

behaviour

of

four

scientific

programs

according

to

our

methodology

and

presented

the

results

The

methodology

has

been

shown

to

be

useful

in

visualising

the

sharing

behaviour

of

a

sample

of

benchmark

applications

Using

this

methodology

we

have

been

able

to

narrow

the

field

of

applicable

shared

memory

management

policies

for

these

programs

As

shown

in

Section

our

results

are

well

in

accordance

with

other

measurements

for

different

purposes

but

using

the

same

programs

This

gives

us

confidence

in

the

correctness

of

the

analysis

tool

and

reinforces

the

advantage

of

using

this

methodology

to

get

information

on

the

sharing

behaviour

of

parallel

programs

prior

to

design

a

shared

memory

management

system

Acknowledgements

The

authors

would

like

to

thank

Jonas

Skeppstedt

for

valuable

comments

on

the

manuscript

This

work

was

supported

by

the

Swedish

National

Board

for

Industrial

and

Technical

Development

-LRB-

Nutek

-RRB-

under

the

contract

number

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

Bibliography

-LSB-

-RSB-

Agarwal

A.

and

Gupta

A.

Memory-Reference

Characteristics

of

Multiprocessor

Applications

under

MACH

Proceedings

of

the

ACM

SIGMETRICS

Conference

pages

-LSB-

-RSB-

Agarwal

A.

Simoni

R.

Hennessy

J.

and

Horowitz

M.

An

Evaluation

of

Directory

Schemes

for

Cache

Coherence

Proceedings

of

the

Symposium

on

Computer

Architecture

pages

-LSB-

-RSB-

Bennet

J.

K.

Carter

J.

B.

and

Zwaenepoel

W.

Adaptive

Software

Cache

Management

for

Distributed

Shared

Memory

Architectures

Proceedings

of

the

Symposium

on

Computer

Architecture

pages

May

-LSB-

-RSB-

Black

D.

L.

Gupta

A.

and

Weber

W

D.

Competitive

Management

of

Distributed

Shared

Memory

Proceedings

of

Compcon

-LSB-

-RSB-

Boyle

J.

Butler

R.

Disz

T.

Glickfield

B.

Lusk

E.

Overbeek

R.

Patterson

J.

and

Stevens

R.

Portable

Programs

for

Parallel

Processors

Holt

Rinehart

and

Winston

inc.

-LSB-

-RSB-

Brorsson

M.

Dahlgren

F.

Nilsson

H.

and

Stenstr?m

P.

A

Multiprocessor

Simulator

Technical

Report

Department

of

Computer

Engineering

Lund

University

P.O.

Box

Lund

Sweden

In

preparation

-LSB-

-RSB-

Chaiken

D.

Fields

C.

Kurihara

K.

and

Agarwal

A.

Directory-Based

Cache

Coherence

in

Large-Scale

Multiprocessors

IEEE

Computer

June

-LSB-

-RSB-

Cheong

H.

and

Veidenbaum

A.

V.

Compiler-Directed

Cache

Management

in

Multiprocessors

IEEE

Computer

June

-LSB-

-RSB-

Darema

F.

George

D.

A.

Norton

V.

A.

and

Pfister

G.

F.

A

single-program-multiple-data

computational

model

for

EPEX/FORTRAN

Parallel

Computing

-LRB-

-RRB-

April

-LSB-

-RSB-

Darema-Rogers

F.

Pfister

G.

F.

and

So

K.

Memory

Access

Patterns

of

Parallel

Scientific

Programs

Proceedings

of

the

ACM

SIGMETRICS

Conference

pages

May

-LSB-

-RSB-

Eggers

S.

J.

and

Katz

R.

A

Characterization

of

Sharing

in

Parallel

Programs

and

its

Application

to

Coherency

Protocol

Evaluation

Proceedings

of

the

Symposium

on

Computer

Architecture

pages

-LSB-

-RSB-

Gallivan

K.

Gannon

D.

Jalby

W.

Malony

A.

and

Wijshoff

H.

Experimentally

Characterizing

the

Behavior

of

Multiprocessor

Memory

System

A

Case

Study

IEEE

Transactions

on

Software

Engineering

-LRB-

-RRB-

February

-LSB-

-RSB-

Johnson-Baylor

S.

K.

and

Rathi

B.

D.

A

Study

of

the

Memory

Reference

Behaviour

of

Engineering/Scientific

Applications

in

Parallel

Processors

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

-LSB-

-RSB-

LaRowe

R.

P.

and

Ellis

C.

S.

Experimental

Comparison

of

Memory

Management

Policies

for

NUMA

Multiprocessors

ACM

Transactions

on

Computer

Systems

-LRB-

-RRB-

November

Visualising

Sharing

Behaviour

in

relation

to

Shared

Memory

Management

-LSB-

-RSB-

LeBlanc

T.

J.

Marsh

B.

D.

and

Scott

M.

L.

Memory

Management

for

Large-Scale

NUMA

Multiprocessors

Technical

Report

TR

Computer

Science

Department

University

of

Rochester

Rochester

NY

USA

March

-LSB-

-RSB-

Lenoski

D.

Laudon

J.

Gharachorloo

K.

Gupta

A.

and

Hennessy

J.

The

Directory-Based

Cache

Coherence

Protocol

for

the

DASH

Multiprocessor

Proceedings

of

the

Symposium

on

Computer

Architecture

pages

-LSB-

-RSB-

Lenoski

D.

Laudon

J.

Joe

T.

Nakahira

D.

Stevens

L.

Gupta

A.

and

Hennessy

J.

The

DASH

Prototype

Implementation

and

Performance

Proceedings

of

the

Symposium

on

Computer

Architecture

pages

-LSB-

-RSB-

Li

K.

IVY

A

shared

Virtual

Memory

System

for

Parallel

Computing

Proceedings

of

International

Conference

on

Parallel

Processing

pages

-LSB-

-RSB-

Nilsson

H.

and

Stenstr?m

P.

The

Scalable

Tree

Protocol

A

Cache

Coherence

Approach

for

Large-Scale

Multiprocessors

Technical

Report

Department

of

Computer

Engineering

Lund

University

P.O.

Box

Lund

Sweden

October

-LSB-

-RSB-

Scheurich

C.

and

Dubois

M.

Dynamic

Page

Migration

in

Multiprocessors

with

Distributed

Global

Memory

Proceedings

of

the

International

Conference

on

Distributed

Computer

Systems

pages

-LSB-

-RSB-

Singh

J.

P.

Weber

W.-D

and

Gupta

A.

SPLASH

Stanford

Parallel

Applications

for

Shared

Memory

Technical

Report

Computer

Systems

Laboratory

Stanford

University

-LSB-

-RSB-

Stenstr?m

P.

Joe

T.

and

Gupta

A.

Comparative

Performance

Evaluation

of

Cache-Coherent

NUMA

and

COMA

Architectures

Proceedings

of

the

Symposium

on

Computer

Architecture

pages

-LSB-

-RSB-

Torrellas

J.

Lam

M.

S.

and

Hennessy

J.

L.

Shared

Data

Placement

Optimizations

to

Reduce

Multiprocessor

Cache

Miss

Rates

Proceedings

of

the

conference

on

Parallel

Processing

pages

-LSB-

-RSB-

Weber

W.-D

and

Gupta

A.

Analysis

of

Cache

Invalidation

Patterns

in

Multiprocessors

Proceeding

of

the

International

Conference

on

Arcitectural

Support

for

Programming

Languages

and

Operating

Systems

pages

April

parallel applications
performance evaluation
program behaviour
shared memory management
shared memory multiprocessors


The Minimum Feature Set Problem Kevin S. Van Horn and Tony Martinez Computer Science Department Brigham Young University Provo , UT This paper appeared in Neural Networks 7 -LRB- 1994 -RRB- , no. 3 , pp. 491 -LCB- 494 . Minimum Feature Set / 1 The Minimum Feature Set Problem Kevin S. Van Horn and Tony Martinez Computer Science Department Brigham Young University Provo , UT Acknowledgments : We wish to acknowledge the support of Wordperfect and Novell in this research , and thank the referees for their helpful comments . Send reprint requests to : Kevin S. Van Horn , 3361 TMCB , Computer Science Dept. , Brigham Young University , Provo , UT 84602 . Phone : -LRB- 801 -RRB- 378-6914 . Running title : Minimum Feature Set . Minimum Feature Set / 2 The Minimum Feature Set Problem Abstract One approach to improving the generalization power of a neural net is to try to minimize the number of non-zero weights used . We examine two issues relevant to this approach , for single-layer nets . First we bound the VC dimension of the set of linearthreshold functions that have non-zero weights for at most s of n inputs . Second , we show that the problem of minimizing the number of non-zero input weights used -LRB- without misclassifying training examples -RRB- is both NP-hard and difficult to approximate . Minimum Feature Set / 3 1 Introduction Consider the task of learning from examples an -LRB- unknown -RRB- linear threshold function f of n real-valued features . Theoretical results -LRB- Blumer , Ehrenfeucht , Haussler & Warmuth , 1989 -RRB- give an upper bound on the number of examples needed for learning that is linear in the VC dimension of the hypothesis space used , which in this case is n +1 . If f depends on only a small subset of the features , this argues for removing the superfluous features ; but often it is not known beforehand on which specific features f depends . Thus there has long been interest in learning algorithms which minimize the number of features actually used -LRB- i.e. the number of non-zero input weights -RRB- , while still correctly classifying all the training examples . -LRB- Note that Littlestone -LRB- 1988 -RRB- gives an alternative approach to the problem of many unnecessary features , for a restricted class of linear threshold functions with Boolean inputs . -RRB- In section 2 we quantify the benefits of minimizing the number of features used , by giving an upper bound on the VC dimension of the set of linear threshold functions with n inputs and at most s non-zero input weights . For s o n this bound is much less than n + 1 . This VC dimension bound improves on the bound that follows from the -LRB- more general -RRB- results of Baum and Haussler -LRB- 1989 -RRB- . In section 3 we examine the time-complexity of this feature-minimization problem , which we call MINIMUM FEATURE SET -LRB- MIN FS -RRB- . We show that MIN FS is both NP-hard and difficult to approximate . In particular , we show the following : ffl No polynomial-time algorithm can approximate MIN FS to within a constant factor , unless P = NP . ffl No polynomial-time algorithm can approximate MIN FS to within an o -LRB- log m -RRB- factor , where m is the number of training examples , unless NP ? DTIME -LSB- nlog log n -RSB- . 2 Benefits of minimization 2.1 Preliminaries We begin with a quick review of some notions used in the PAC -LRB- probably approximately correct -RRB- model of learning . The definitions and theorems of this subsection are from Blumer et al. -LRB- 1989 -RRB- . Definition 1 A hypothesis space H over a set X is a set of f0 ; 1g-valued functions with domain X . We say that a hypothesis h 2 H is consistent with a set E of training examples if h -LRB- x -RRB- = y for all examples -LRB- x ; y -RRB- 2 E . We say that a learning algorithm uses H consistently if it returns a hypothesis h 2 H that is consistent with the set of training examples given it , whenever such an h exists . Minimum Feature Set / 4 Definition 2 Let H be a hypothesis space over a set X. For any finite I ? X we define ? H -LRB- I -RRB- = fh \ I : h 2 Hg ; where we regard the elements of H as subsets of X. For any integer m , 1 < = m < = jXj , we define ? H -LRB- m -RRB- = max I?X ; jIj = m j?H -LRB- I -RRB- j : ? H -LRB- m -RRB- is just the maximum number of ways a set of m elements of X can be dichotomized by elements of H. Note that ? H -LRB- m -RRB- < = 2m , since an m-element set has only 2m distinct subsets . Definition 3 Let H be a hypothesis space over a set X . The Vapnik-Chervonenkis dimension of H , denoted VCdim -LRB- H -RRB- , is the largest integer m s.t. ? H -LRB- m -RRB- = 2m . I.e. , VCdim -LRB- H -RRB- is the cardinality of the largest subset of X which can be arbitrarily dichotomized by elements of H . In general , VCdim -LRB- H -RRB- and ? H -LRB- m -RRB- are related as follows : Theorem 1 If VCdim -LRB- H -RRB- = d and m > = d > = 1 , then ? H -LRB- m -RRB- < = -LRB- em =d -RRB- d , where e is the base of the natural logarithm . The VC dimension is important in computing upper bounds on a learning algorithm 's sample complexity -LRB- a measure of the number of examples needed for learning -RRB- . For learning algorithms that use a hypothesis space H consistently , these bounds are linear in VCdim -LRB- H -RRB- -LRB- Blumer et al. , 1989 -RRB- . 2.2 Effect of minimization on VC dimension Let H be the set of linear threshold functions on n real-valued features . We will identify h 2 H with its defining weight vector w , i.e. the vector w 2 Qn +1 s.t. h -LRB- x -RRB- = -LRB- 1 if wn +1 + Pni = 1 wixi > if wn +1 + Pni = 1 wixi < : -LRB- Q is the set of rational numbers . -RRB- We have VCdim -LRB- H -RRB- = n +1 -LRB- Blumer et al. , 1989 -RRB- . Definition 4 The cost of w 2 Qn +1 -LRB- or the corresponding h 2 H -RRB- is the number of nonzero weights other than the bias weight wn +1 , i.e. jfi : wi < > ; 1 < = i < = ngj . Definition 5 Let S ? N def = f1 ; : : : ; ng and 1 < = s < = n ; then we define ffl H -LSB- S -RSB- is the set of h 2 H which have nonzero weights -LRB- other than the bias -RRB- only for the features in S , and Minimum Feature Set / 5 ffl H -LSB- s -RSB- is the set of h 2 H whose cost is at most s. Suppose a learning algorithm A uses H consistently and also minimizes the cost of the hypothesis it returns . If s is the cost of the function being learned , then A also uses H -LSB- s -RSB- consistently , and we have an upper bound on sample complexity that is linear in VCdim -LRB- H -LSB- s -RSB- -RRB- . Even if A only does approximate minimization , the notion of an effective hypothesis space -LRB- Blumer et al. , 1989 -RRB- can be used to obtain sample-complexity bounds that use VCdim -LRB- H -LSB- s -RSB- -RRB- rather than VCdim -LRB- H -RRB- . Thus it is important to have good bounds on the former . Theorem 2 Let 1 < = s < n . Then VCdim -LRB- H -LSB- s -RSB- -RRB- < 2:71 -LRB- s + 1 -RRB- log2 -LRB- en = -LRB- s + 1 -RRB- -RRB- . Proof . We can assume that s + 1 < = n = 2 . If not , i.e. s + 1 > n = 2 , then n > = s + 1 > = -LRB- n + 1 -RRB- = 2 -LRB- since s and n are integers -RRB- , and writing lg for log2 we have 2:71 -LRB- s + 1 -RRB- lg ? en s + 1 ? > = 2:71 -LRB- s + 1 -RRB- lg e > = 2:71 2 -LRB- n + 1 -RRB- lg e > n + 1 : But since H -LSB- s -RSB- ? H we have VCdim -LRB- H -LSB- s -RSB- -RRB- < = VCdim -LRB- H -RRB- = n + 1 , which is less than our bound . We now compute ? H -LSB- s -RSB- -LRB- m -RRB- . The set of linear threshold functions of s inputs has VC dimension s + 1 . Thus VCdim -LRB- H -LSB- S -RSB- -RRB- = s + 1 for all S ? N , jSj = s , and by Theorem 1 this gives ? H -LSB- S -RSB- -LRB- m -RRB- < = -LRB- em = -LRB- s + 1 -RRB- -RRB- s +1 for all m > = s + 1 . Using this inequality we obtain ? H -LSB- s -RSB- -LRB- m -RRB- < = X jSj = s ? H -LSB- S -RSB- -LRB- m -RRB- < = n s ! -LRB- em = -LRB- s + 1 -RRB- -RRB- s +1 : By the assumption that s + 1 < = n = 2 we have ? ns +1 ? > ? ns ? . A variant of Stirling 's formula states that ? jk ? < = -LRB- ej = k -RRB- k , for all j and k -LRB- Cormen , Leiserson & Rivest , 1990 , p. 102 -RRB- . Combining these we obtain ? H -LSB- s -RSB- -LRB- m -RRB- < -LRB- en = -LRB- s + 1 -RRB- -RRB- s +1 -LRB- em = -LRB- s + 1 -RRB- -RRB- s +1 for all m > = s + 1 . Let x0 def = lg -LRB- en = -LRB- s + 1 -RRB- -RRB- and m0 def = 2:71 -LRB- s + 1 -RRB- x0 . To prove the theorem it suffices to show that ? H -LSB- s -RSB- -LRB- m0 -RRB- < 2m0 , by the definition of the VC dimension . This inequality will hold if m0 ? lg -LRB- ? H -LSB- s -RSB- -LRB- m0 -RRB- -RRB- > 0 . Dividing this inequality by s + 1 , and using the fact that m0 > = s + 1 , we find m0 ? lg -LRB- ? H -LSB- s -RSB- -LRB- m0 -RRB- -RRB- s + 1 > m0 s + 1 ? x0 ? lg ? em0 s + 1 ? = 1:71 x0 ? lg -LRB- 2:71 ex0 -RRB- Thus it suffices to show that f -LRB- x0 -RRB- > = 0 , where f -LRB- x -RRB- def = 1:71 x?lg -LRB- 2:71 ex -RRB- . We have f -LRB- x -RRB- = 1:71 ? -LRB- x ln 2 -RRB- ? 1 , hence f -LRB- x -RRB- is non-decreasing for all x > = xmin def = -LRB- 1:71 ln 2 -RRB- ? 1 . Minimum Feature Set / 6 Thus it suffices to show that f -LRB- x -RRB- > = for some xmin < = x < = x0 . Let us choose x = lg -LRB- 2e -RRB- . Since s + 1 < = n = 2 we have x0 > = lg -LRB- 2e -RRB- . We also have lg -LRB- 2e -RRB- = ln -LRB- 2e -RRB- = ln 2 > 1:71 ? 1 = ln 2 = xmin : Finally , by computation we find f -LRB- lg -LRB- 2e -RRB- -RRB- ss 7:55 ? 10 ? 3 > 0 . 2 The above VC dimension bound is a significant improvement over n +1 when s o n . It is also an improvement over the ? -LRB- s log -LRB- sn -RRB- -RRB- bound that can be obtained from Corollary 5 of Baum and Haussler -LRB- 1989 -RRB- , which is a more general result applicable to multi-layered neural nets . 3 The complexity of minimization We can formally state the minimization problem we have been discussing as follows : Definition 6 MINIMUM FEATURE SET -LRB- MIN FS -RRB- Instance : Integers m > = 1 , n > = 1 , and an m ? -LRB- n + 1 -RRB- matrix A of rational numbers s.t. ai ; n +1 2 f1 ; ? 1g for all i. Problem : Find a w 2 Qn +1 satisfying Aw > , of minimum cost . -LRB- Recall that the cost of w is the number of nonzero wi , i < > n + 1 . -RRB- In the above , m is the number of training examples and n is the number of features . Ai , the i-th row of the matrix A , is obtained from the i-th training example -LRB- xi ; yi -RRB- as follows : first , augment xi with 1 to obtain an -LRB- n + 1 -RRB- - element vector -LRB- xi -RRB- ; then , if it is a negative example -LRB- yi = 0 -RRB- , negate this vector . The requirement that Aw > is equivalent to Aiw > for all i , so this gives us -LRB- xi -RRB- ? w > for positive examples and -LRB- xi -RRB- ? w < for negative examples . The requirement that ai ; n +1 2 f1 ; ? 1g arises from the above construction , in which the example vectors were augmented with 1 to provide for the bias term ; thus ai ; n +1 = 1 for positive examples and ai ; n +1 = ? 1 for negative examples . We now give some standard definitions that will be needed in our investigation of the difficulty of solving or approximating MIN FS . Definition 7 A minimization problem M has the following form , for some pair of predicates P and Q and integer-valued cost function < = : Given x satisfying P -LRB- x -RRB- -LRB- a problem instance -RRB- , find some y satisfying Q -LRB- x ; y -RRB- -LRB- a solution for x -RRB- that minimizes < = -LRB- x ; y -RRB- . Definition 8 A polynomial-time approximation algorithm -LRB- PTAA -RRB- A for a minimization problem M is a polynomial-time algorithm which takes as input some instance x satisfying P -LRB- x -RRB- and outputs a y satisfying Q -LRB- x ; y -RRB- . We say that A approximates M to within a factor OE -LRB- x -RRB- if , additionally , < = -LRB- x ; y -RRB- < = OE -LRB- x -RRB- < = -LRB- x ; z -RRB- for any z such that Q -LRB- x ; z -RRB- . We say that M can be approximated to within a factor OE -LRB- x -RRB- if there is a PTAA that approximates it to within a factor OE -LRB- x -RRB- . Minimum Feature Set / 7 Definition 9 A cost-preserving polynomial transformation from minimization problem M to minimization problem M0 is a pair of functions -LRB- t1 ; t2 -RRB- with the following properties , where x is taken to be an instance of M : 1 . t1 maps instances of M to instances of M0 . t2 maps pairs -LRB- y ; x -RRB- , where y is a solution for t1 -LRB- x -RRB- , to solutions for x. 2 . t1 and t2 are both computable in polynomial time . 3 . If x has a solution of cost k , then t1 -LRB- x -RRB- has a solution of cost at most k. 4 . If y is a solution for t1 -LRB- x -RRB- of cost k , then t2 -LRB- y ; x -RRB- has cost at most k. Suppose there exist both a cost-preserving polynomial transformation from M to M0 and a PTAA that approximates M0 to within a factor OE -LRB- x0 -RRB- . Then there is a PTAA that approximates M to within a factor OE -LRB- t1 -LRB- x -RRB- -RRB- : compute x0 = t1 -LRB- x -RRB- , run the PTAA for M0 on x0 to get y , and output t2 -LRB- y ; x -RRB- . Thus any limits on the approximability of M give corresponding limits on the approximability of M0 . We will show that MIN FS is both NP-hard and difficult to approximate via a transformation from MIN SET COVER . Definition 10 MINIMUM SET COVER -LRB- MIN SC -RRB- Instance : A finite set S and a collection C of subsets of S. Problem : Find a cover of S -LRB- a set C ? C s.t. SC = S -RRB- of minimum cardinality . Theorem 3 There is a cost-preserving polynomial transformation from MIN SC to MIN FS in which instances -LRB- S ; C -RRB- of MIN SC are mapped to instances -LRB- m ; n ; A -RRB- of MIN FS with m = jSj + 1 . Proof . In the following we shall use the notation -LRB- ? -RRB- , where ? is a logical formula , for -LRB- if ? then 1 else 0 -RRB- . This notation is due to Graham , Knuth and Patashnik -LRB- 1989 -RRB- . Let -LRB- S ; C -RRB- be an instance of MIN SC . Define m = jSj +1 and n = jCj . Enumerate the elements of S as s1 , s2 , : : : , sm ? 1 and the elements of C as c1 , c2 , : : : , cn . Define t1 -LRB- S ; C -RRB- = -LRB- m ; n ; A -RRB- , where A is an m ? -LRB- n + 1 -RRB- matrix constructed from the column vectors of 1 's and 0 's corresponding to each cj , as follows : ... c1 c2 ? ? ? cn ? 1 ... ? ? ? ? ? ? 1 -LRB- Thus , aij = -LRB- si 2 cj -RRB- for i < m and j < = n. -RRB- Minimum Feature Set / 8 Note that the inequality Aw > is equivalent to the set of inequalities Aiw > 0 , 1 < = i < = m . This set of inequalities is in turn equivalent to X j < = n -LRB- si 2 cj -RRB- wj > wn +1 > ; 1 < = i < m -LRB- 1 -RRB- Suppose that -LRB- S ; C -RRB- has a solution of cost k , i.e. there exists some C ? C which is a cover of S , and jC j = k. Define the vector w by wj = -LRB- -LRB- cj 2 C -RRB- if j < = n :5 if j = n + 1 : Note that w has cost k and wn +1 > 0 . Using the fact that C is a cover of S we have also that , for 1 < = i < m , X j < = n -LRB- si 2 cj -RRB- wj = X j < = n -LRB- si 2 cj -RRB- -LRB- cj 2 C -RRB- > 1 > wn +1 ; so w satisfies -LRB- 1 -RRB- . Thus we have shown that t1 -LRB- S ; C -RRB- = -LRB- m ; n ; A -RRB- has a solution of cost k . We now consider the reverse mapping . For all w 2 Qn +1 , define t2 -LRB- w ; S ; C -RRB- = fcj : wj > ; 1 < = j < = ng : Suppose that w is a solution for t1 -LRB- S ; C -RRB- = -LRB- m ; n ; A -RRB- , of cost k. Let C = t2 -LRB- w ; S ; C -RRB- . It is clear from the definition of t2 that C ? C and jC j < = k . We have furthermore that Aw > and hence w satisfies the inequalities -LRB- 1 -RRB- . Thus for each i there is some j s.t. si 2 cj and wj > 0 . But wj > implies cj 2 C , so C is a cover for S. Thus we have shown that w is a solution for -LRB- S ; C -RRB- of cost at most k. 2 Corollary 4 MIN FS is NP-hard . In addition , we have the following approximability limits : 1 . For every constant c > = 1 , MIN FS can not be approximated to within a factor c , unless P = NP . 2 . MIN FS can not be approximated to within an o -LRB- log m -RRB- factor , unless NP ? DTIME -LSB- nlog log n -RSB- . Proof . MIN SC is NP-hard , hence by Theorem 3 , so is MIN FS . From Theorem 3 , if MIN FS can be approximated to within some constant factor c , so can MIN SC . But Bellare , Goldwasser , Lund and Russel -LRB- 1993 -RRB- have shown that MIN SC can not be approximated to within a constant factor unless P = NP . Suppose that MIN FS can be approximated to within a factor g -LRB- m ; n ; A -RRB- = f -LRB- m -RRB- , where f -LRB- m -RRB- = o -LRB- log m -RRB- . Then MIN SC can be approximated to within a factor g -LRB- t1 -LRB- S ; C -RRB- -RRB- = f -LRB- jSj + 1 -RRB- = o -LRB- log -LRB- jSj + 1 -RRB- -RRB- = o -LRB- log jSj -RRB- . But Bellare et al. have also Minimum Feature Set / 9 shown that MIN SC can not be approximated to within an o -LRB- log jSj -RRB- factor unless NP ? DTIME -LSB- nlog log n -RSB- . 2 Haussler -LRB- 1988 -RRB- looks at the problem of finding a pure conjunctive concept consistent with a set of examples , with the minimum number of conjuncts , and shows that it is NP-hard via a reduction from MIN SC . His reduction is similar to ours , and defines a cost-preserving polynomial transformation that produces jSj + 1 examples . Thus the conclusions of Corollary 4 also apply to Haussler 's problem . We note in closing that Corollary 4 should not be taken as evidence against the possibility of obtaining improved sample-complexity bounds by approximating MIN FS . Lemma 5.6 of Haussler -LRB- 1988 -RRB- and Theorem 3.2.1 of Blumer et al. -LRB- 1989 -RRB- can be used to establish improved upper bounds on sample complexity when the approximation factor grows slowly with the number of examples . References Baum , E. B. , & Haussler , D. -LRB- 1989 -RRB- . What size net gives valid generalizations ? Neural Computation , 1 , 151 -LCB- 160 . Bellare , M. , Goldwasser , S. , Lund , C. , & Russell , A. -LRB- 1993 -RRB- . Efficient probabilistically checkable proofs and applications to approximation . Proceedings of the 25th Annual ACM Symposium on Theory of Computing , 294 -LCB- 304 . Blumer , A. , Ehrenfeucht , A. , Haussler , D. , & Warmuth , M. K. -LRB- 1989 -RRB- . Learnability and the Vapnik-Chervonenkis dimension . Journal of the ACM , 36 , 929 -LCB- 965 . Cormen , T. H. , Leiserson , C. E. , & Rivest , R -LRB- 1990 -RRB- . Introduction to algorithms . Cambridge , MA : MIT Press . Graham , R. L. , Knuth , D. E. , & Patashnik , O. -LRB- 1989 -RRB- . Concrete mathematics : a foundation for computer science . Reading , MA : Addison-Wesley . Haussler , D. -LRB- 1988 -RRB- . Quantifying inductive bias : AI learning algorithms and Valiant 's learning framework . Artificial Intelligence , 36 , 177 -LCB- 221 . Littlestone , N. -LRB- 1988 -RRB- . Learning quickly when irrelevant attributes abound : a new linear-threshold algorithm . Machine Learning , 2 , 285 -LCB- 318 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O

approximation algorithm
complexity
learning
linear threshold
minimization