A

Theory

for

Memory-Based

Learning

Jyh-Han

Lin

and

Jeffrey

Scott

Vitter

Department

of

Computer

Science

Brown

University

Providence

Rhode

Island

November

A

Theory

for

Memory-Based

Learning

Jyh-Han

Lin

Motorola

Inc.

Communications

Systems

Lab

Applied

Research

Paging

and

Wireless

Data

Group

N.W.

Avenue

Boynton

Beach

FL

-LCB-

jyh-han

lin@pts.mot.com

Jeffrey

Scott

Vitter

Department

of

Computer

Science

Brown

University

Providence

R.

I.

-LCB-

jsv@cs.brown.edu

research

was

conducted

while

the

author

was

at

the

Department

of

Computer

Science

Brown

University

Abstract

A

memory-based

learning

system

is

an

extended

memorymanagement

system

that

decomposes

the

input

space

either

statically

or

dynamically

into

subregions

for

the

purpose

of

storing

and

retrieving

functional

information

The

main

generalization

techniques

employed

by

memory-based

learning

systems

are

the

nearest-neighbor

search

space

decomposition

techniques

and

clustering

Research

on

memory-based

learning

is

still

in

its

early

stage

In

particular

there

are

very

few

rigorous

theoretical

results

regarding

memory

requirement

sample

size

expected

performance

and

computational

complexity

In

this

paper

we

propose

a

model

for

memory-based

learning

and

use

it

to

analyze

several

methods

ffl-covering

hashing

clustering

tree-structured

clustering

and

receptive-fields

for

learning

smooth

functions

The

sample

size

and

system

complexity

are

derived

for

each

method

Our

model

is

built

upon

the

generalized

PAC

learning

model

of

Haussler

-LSB-

-RSB-

and

is

closely

related

to

the

method

of

vector

quantization

in

data

compression

Our

main

result

is

that

we

can

build

memory-based

learning

systems

using

new

clustering

algorithms

-LRB-

Lin

and

Vitter

-LSB-

-RSB-

-RRB-

to

PAC-learn

in

polynomial

time

using

only

polynomial

storage

in

typical

situations

MOTIVATION

In

this

paper

we

introduce

a

model

for

memory-based

learning

and

consider

the

problem

of

learning

smooth

functions

by

memory-based

learning

systems

A

memorybased

learning

system

is

an

extended

memory

management

system

that

decomposes

the

input

space

either

statically

or

dynamically

into

subregions

for

the

purpose

of

storing

and

retrieving

functional

information

for

some

smooth

function

The

main

generalization

techniques

employed

by

memory-based

learning

system

are

the

nearestneighbor

search

space

decomposition

techniques

and

clustering

Although

memorybased

learning

systems

are

not

as

powerful

as

neural

net

models

in

general

the

training

problem

for

memory-based

learning

systems

may

be

computationally

more

tractable

An

example

memory-based

learning

system

is

shown

in

Figure

The

encoder

maps

an

input

from

the

input

space

X

into

a

set

of

addresses

and

the

decoder

maps

the

set

of

activated

memory

locations

into

an

output

in

the

output

space

Y

The

look-up

table

for

memory-based

learning

systems

can

be

organized

as

hash

tables

trees

or

full-search

tables

The

formal

definitions

of

memory-based

learning

systems

will

be

given

in

Section

The

motivation

for

our

model

is

as

follows

In

the

human

motor

system

most

of

the

computations

done

are

entirely

subconscious

The

detailed

computations

of

what

each

muscle

must

do

in

order

to

coordinate

with

other

muscles

so

as

to

produce

the

desired

movement

are

left

to

low-level

subconscious

computing

centers

Considering

the

complexity

of

the

type

of

manipulation

tasks

routinely

performed

by

biological

organisms

it

seems

that

the

approach

of

controlling

robotic

manipulator

systems

by

a

mathematical

formalism

such

as

trigonometric

equations

is

inadequate

to

produce

truly

sophisticated

motor

behavior

To

remedy

this

situation

Albus

-LSB-

-RSB-

proposed

a

memory-driven

table-reference

motor

control

system

called

Cerebellar

Model

Articulation

Controller

-LRB-

CMAC

-RRB-

The

fact

that

for

n

input

variables

with

R

distinguishable

levels

there

are

Rn

possible

inputs

may

be

sufficient

to

discourage

this

line

of

research

However

Albus

observed

that

for

any

physical

manipulator

system

the

number

of

different

inputs

that

are

likely

to

be

encountered

-LRB-

and

thus

the

size

of

memory

that

is

actually

needed

-RRB-

is

much

smaller

than

Rn

He

also

noticed

for

similar

motor

behaviors

-LRB-

for

example

swinging

a

bat

or

a

golf

club

-RRB-

that

the

required

muscle

movements

are

similar

Albus

outlined

a

memory

management

technique

to

take

MOTIVATION

Table

look-up

Input

space

X

Output

space

Y

bg

Memory

Z

s

encoder

decoder

Figure

An

example

memory-based

learning

system

The

encoder

maps

an

input

from

the

input

space

X

into

a

set

of

addresses

and

the

decoder

maps

the

set

of

activated

memory

locations

into

an

output

in

the

output

space

Y

advantage

of

these

two

properties

and

make

the

memory-based

approach

to

learning

control

functions

more

practical

In

the

CMAC

system

each

input

x

from

an

input

space

X

is

assigned

by

a

mapping

to

a

set

-LRB-

x

-RRB-

of

locations

in

a

memory

V

Each

location

contains

a

vector

in

an

output

space

Y

The

output

f

-LRB-

x

-RRB-

is

computed

by

summing

the

values

-LRB-

weights

-RRB-

at

all

of

the

memory

locations

assigned

to

x

f

-LRB-

x

-RRB-

X

-LRB-

x

-RRB-

V

-LSB-

i

-RSB-

The

mapping

has

the

characteristic

that

similar

inputs

in

the

input

space

X

map

to

overlapping

sets

of

locations

in

the

memory

V

while

dissimilar

inputs

map

to

distinct

sets

of

locations

in

the

memory

V

The

amount

of

overlap

between

two

sets

of

locations

in

the

memory

V

is

related

to

the

generalized

Hamming

distance

between

two

corresponding

inputs

in

X

This

mapping

is

supposed

to

give

automatic

generalization

-LRB-

interpolation

-RRB-

between

inputs

in

X

that

is

similar

inputs

produce

similar

outputs

Clearly

this

scheme

may

require

the

size

of

memory

V

to

be

on

the

same

order

of

magnitude

as

the

total

number

of

possible

input

vectors

in

X

In

practice

this

is

hardly

feasible

For

this

reason

the

memory

V

is

considered

to

be

only

a

hypothetical

memory

each

location

in

V

is

mapped

using

a

hash

function

h

to

a

physical

memory

Z

of

practical

size

The

output

f

-LRB-

x

-RRB-

is

then

computed

by

summing

the

values

in

the

memory

Z

that

are

mapped

to

by

the

input

x

f

-LRB-

x

-RRB-

X

-LRB-

x

-RRB-

Z

-LSB-

h

-LRB-

i

-RRB-

-RSB-

X

-LRB-

x

-RRB-

Z

-LSB-

i

-RSB-

where

h

ffi

As

a

result

of

the

random

hashing

from

the

hypothetical

memory

V

to

the

physical

memory

Z

the

sets

of

memory

locations

mapped

to

by

dissimilar

inputs

in

input

space

X

have

a

low

but

nonzero

probability

of

overlapping

this

can

create

an

undesirable

generalization

between

dissimilar

inputs

The

resulting

system

will

produce

an

output

f

-LRB-

x

-RRB-

Y

for

any

input

x

in

the

input

space

X

Since

the

number

of

locations

in

the

real

memory

Z

will

typically

be

much

smaller

than

the

total

number

of

possible

inputs

it

is

unlikely

that

the

weights

in

Z

can

be

found

such

that

the

outputs

of

CMAC

system

are

correct

over

the

entire

input

space

On

the

other

hand

it

is

unlikely

that

all

possible

inputs

will

be

encountered

in

solving

a

particular

control

or

classification

problem

The

standard

CMAC

model

has

been

applied

to

the

real-time

control

of

robots

with

encouraging

success

-LRB-

Miller

-LSB-

-RSB-

Miller

Glanz

and

Kraft

-LSB-

-RSB-

-RRB-

A

comprehensive

coverage

of

the

CMAC

models

and

learning

algorithms

is

given

by

Dean

-LSB-

-RSB-

Research

on

the

CMAC

model

and

its

variants

is

still

in

its

early

stage

In

particular

there

are

very

few

rigorous

theoretical

results

available

Many

problems

remained

unanswered

among

them

the

following

In

the

current

experimental

study

learning

parameters

are

chosen

on

an

ad

hoc

basis

The

effects

of

the

scale

of

resolution

the

size

of

physical

memory

and

A

MEMORY-BASED

LEARNING

MODEL

the

size

of

the

training

database

-LRB-

examples

-RRB-

on

system

performance

are

largely

unknown

Given

a

class

F

of

functions

and

a

tolerable

relative

error

bound

what

are

the

sample

size

and

memory

size

required

to

approximate

functions

in

F

Given

a

sample

what

are

the

computational

complexities

of

training

That

is

how

much

time

does

it

require

to

determine

system

parameters

from

the

sample

In

Section

we

outline

a

theoretical

framework

for

answering

these

problems

Our

memory-based

learning

model

is

built

upon

the

generalized

PAC

learning

model

of

Haussler

-LSB-

-RSB-

and

is

closely

related

to

the

method

of

vector

quantization

in

data

compression

-LRB-

Gersho

-LSB-

-RSB-

Gray

-LSB-

-RSB-

Riskin

-LSB-

-RSB-

Gersho

and

Gray

-LSB-

-RSB-

-RRB-

Section

introduces

the

notion

of

quantization

number

which

is

intended

to

capture

the

optimal

memory

requirement

of

memory-based

learning

systems

for

a

given

error

bound

The

quantization

number

can

be

significantly

smaller

than

the

covering

number

in

practice

In

Section

we

use

our

model

to

analyze

several

methods

for

learning

smooth

functions

by

nearest-neighbor

systems

Our

main

result

is

that

we

can

build

memorybased

learning

systems

using

the

new

clustering

algorithms

of

Lin

and

Vitter

-LSB-

-RSB-

to

PAC-learn

in

polynomial

time

using

only

polynomial

storage

in

typical

situations

We

extend

our

analysis

to

tree-structured

and

higher-order

memory-based

learning

system

in

Section

and

respectively

We

conclude

with

some

possible

extensions

to

our

model

in

Section

A

MEMORY-BASED

LEARNING

MODEL

Let

T

be

a

complete

and

separable

metric

space

with

distance

metric

dT

We

denote

the

metric

space

by

-LRB-

T

dT

-RRB-

Let

H

-LRB-

T

-RRB-

denote

the

space

whose

points

are

the

compact

subset

of

T

The

diameter

of

a

set

A

H

-LRB-

T

-RRB-

denoted

as

diam

-LRB-

A

-RRB-

is

dT

-LRB-

-RRB-

The

distance

dT

-LRB-

t

A

-RRB-

from

a

point

t

to

a

set

A

H

-LRB-

T

-RRB-

is

defined

as

dT

-LRB-

t

x

-RRB-

For

any

ffl

an

ffl-cover

for

A

is

a

finite

set

U

T

such

that

for

all

t

A

there

is

a

u

U

such

that

dT

-LRB-

t

u

-RRB-

ffl

If

A

has

a

finite

ffl-cover

for

every

ffl

then

A

is

totally

bounded

Let

N

-LRB-

A

ffl

dT

-RRB-

denote

the

size

of

the

smallest

ffl-cover

for

A

We

refer

to

N

-LRB-

A

ffl

dT

-RRB-

as

the

covering

number

MEMORY-BASED

LEARNING

SYSTEMS

In

this

paper

we

let

X

ae

k

be

the

input

space

and

Y

ae

be

the

output

space

and

let

dX

and

dY

be

the

Euclidean

metrics

In

typical

applications

X

and

Y

are

usually

hypercubes

or

hyperrectangles

Let

MX

diam

-LRB-

X

-RRB-

and

MY

diam

-LRB-

Y

-RRB-

For

a

positive

integer

s

let

Ns

denote

the

set

sg

Let

Nrs

be

the

collection

of

all

r-element

subsets

-LRB-

r-subsets

-RRB-

of

Ns

Let

U

usg

and

B

be

a

subset

of

U

then

index

-LRB-

B

-RRB-

denotes

the

set

of

indices

of

elements

in

B.

MEMORY-BASED

LEARNING

SYSTEMS

Definition

A

generic

memory

based

learning

system

G

realizes

a

class

of

functions

from

the

input

space

-LRB-

X

dX

-RRB-

to

the

output

space

-LRB-

Y

dY

-RRB-

Each

function

g

realizable

by

G

can

be

specified

by

a

sequence

of

memory

contents

Z

zsi

where

s

is

a

positive

integer

and

a

pair

of

functions

h

i

is

the

encoder

which

is

a

mapping

from

X

to

and

is

the

decoder

which

is

a

mapping

from

to

Y

We

can

write

g

as

the

composition

ffi

We

denote

Z

-LRB-

i

-RRB-

zi

We

may

regard

Ns

as

the

address

-LRB-

or

neuronal

-RRB-

space

and

as

the

collection

of

sets

of

activated

addresses

-LRB-

or

neurons

-RRB-

We

will

often

study

parameterized

classes

of

memory-based

learning

systems

Let

C

G

be

a

complexity

function

of

memory-based

learning

systems

which

maps

a

system

g

G

to

a

positive

real

number

The

most

straightforward

complexity

measure

is

the

size

of

memory

which

we

will

use

in

this

paper

However

for

some

applications

other

complexity

measures

may

be

more

appropriate

For

example

in

real-time

applications

we

may

be

more

concerned

with

the

speed

of

encoding

and

decoding

In

remote-control

applications

the

sensor/encoder

and

effector/decoder

may

not

be

at

the

same

location

and

the

sensor

has

to

send

control

signals

-LRB-

addresses

-RRB-

to

the

effector

via

communication

channels

In

such

a

scenario

communication

complexity

may

be

a

more

important

issue

We

let

Gs

denote

the

class

of

memory-based

learning

systems

of

complexity

at

most

s

that

is

Gs

fg

j

C

-LRB-

g

-RRB-

sg

We

are

interested

in

the

following

two

types

of

memory-based

learning

systems

full-search

systems

and

tree-structured

systems

In

a

full-search

system

each

memory

location

corresponds

to

a

region

in

the

input

space

and

contains

a

representative

vector

-LRB-

key

-RRB-

and

a

functional

value

the

encoder

maps

an

input

to

the

memory

locations

corresponding

to

regions

that

include

the

input

point

Examples

of

full-search

A

MEMORY-BASED

LEARNING

MODEL

systems

include

Voronoi

systems

and

receptive-field

systems

Definition

The

class

G

-LSB-

s

rGrs

of

-LRB-

generalized

-RRB-

Voronoi

systems

of

order

r

is

defined

as

follows

Let

U

usgy

and

B

be

an

r-subset

of

U

then

Vor

-LRB-

B

r

-RRB-

denotes

the

Voronoi

region

of

order

r

for

B

The

encoder

of

a

Voronoi

system

of

order

r

and

size

s

is

a

mapping

from

X

to

Nrs

and

maps

x

X

to

index

-LRB-

B

-RRB-

if

and

only

if

x

Vor

-LRB-

B

r

-RRB-

The

decoder

is

a

mapping

from

X

Nrs

and

is

defined

as

g

-LRB-

x

-RRB-

r

X

-LRB-

x

-RRB-

Z

-LRB-

i

-RRB-

We

shall

refer

to

the

first-order

Voronoi

systems

simply

as

Voronoi

systems

Definition

The

class

G

-LSB-

s

of

receptive-field

systems

is

defined

as

follows

Let

R

Rsg

be

a

collection

of

polyhedral

sets

-LRB-

regions

-RRB-

such

that

SR

Ri

X

The

encoder

maps

an

input

x

to

the

set

-LRB-

x

-RRB-

of

indices

of

regions

that

contain

x.

Note

that

the

regions

are

allowed

to

be

overlapped

The

maximum

degree

of

overlap

is

the

order

of

the

system

The

decoder

is

a

mapping

from

X

Nrs

and

is

defined

as

g

-LRB-

x

-RRB-

X

-LRB-

x

-RRB-

Z

-LRB-

i

-RRB-

Notable

examples

of

receptive-field

systems

include

the

CMAC

model

and

Moody

's

multi-resolution

hierarchies

-LRB-

Moody

-LSB-

-RSB-

-RRB-

In

a

tree-structured

system

the

encoder

partitions

the

input

space

into

a

hierarchy

of

regions

An

input

is

mapped

to

the

memory

location

corresponding

to

the

region

represented

by

a

leaf

The

computational

advantage

of

tree-structured

systems

over

full-search

systems

in

sequential

models

of

computation

is

that

the

mapping

from

an

input

to

a

memory

location

can

be

done

quickly

by

tree

traversal

Definition

The

class

G

-LSB-

s

of

tree-structured

systems

is

defined

as

follows

The

encoder

of

a

tree-structured

systems

of

size

s

partitions

the

input

space

into

a

hierarchy

of

regions

specified

by

a

tree

with

s

nodes

Each

internal

node

has

a

number

of

branches

each

of

which

is

associated

with

a

key

Given

an

input

starting

at

the

root

node

the

encoder

compares

the

input

with

each

key

and

follows

the

branch

associated

with

the

key

nearest

to

the

input

the

search

proceeds

this

way

until

a

leaf

is

reached

The

search

path

is

output

by

the

encoder

as

the

address

for

that

input

The

decoder

takes

a

search

path

and

outputs

the

value

in

the

leaf

THE

MEMORY-BASED

LEARNING

PROBLEM

Examples

of

tree-structured

systems

include

learning

systems

based

upon

quadtrees

and

k-d

trees

such

as

SAB-trees

of

Moore

-LSB-

-RSB-

THE

MEMORY-BASED

LEARNING

PROBLEM

Informally

given

a

probability

measure

P

over

X

Y

the

goal

of

learning

in

this

model

is

to

approximate

P

by

a

memory-based

learning

system

g

G

of

reasonable

complexity

The

expected

error

of

the

hypothesis

g

with

respect

to

P

is

denoted

by

erP

-LRB-

g

-RRB-

E

-LSB-

dY

-LRB-

g

-LRB-

x

-RRB-

y

-RRB-

-RSB-

Z

X?Y

dY

-LRB-

g

-LRB-

x

-RRB-

y

-RRB-

dP

-LRB-

x

y

-RRB-

where

hx

yi

is

the

random

vector

corresponding

to

P

The

formal

PAC

memorybased

learning

model

is

defined

below

Definition

A

memory-based

learning

problem

B

is

specified

by

a

class

G

of

memorybased

learning

systems

and

a

class

P

of

probability

measures

over

X

Y

where

X

k

and

Y

We

say

that

B

is

learnable

if

for

any

ffi

and

ffl

the

following

holds

There

exists

a

-LRB-

possibly

randomized

-RRB-

algorithm

L

such

that

if

L

is

given

as

input

a

random

sample

sequence

h

-LRB-

xi

yi

-RRB-

i

of

polynomial

size

m

-LRB-

k

-RRB-

then

with

probability

at

least

ffi

L

will

output

a

memory-based

learning

system

L

-LRB-

-RRB-

G

that

satisfies

erP

-LRB-

L

-LRB-

-RRB-

-RRB-

fflMY

If

L

runs

in

polynomial

time

then

we

say

that

B

is

polynomial-time

learnable

SMOOTH

FUNCTIONS

Without

any

restriction

on

the

class

P

of

probability

measures

over

X

Y

learning

is

not

likely

to

be

feasible

in

terms

of

memory

requirement

sample

size

and

computational

complexity

In

this

paper

we

restrict

P

to

be

generated

by

some

smooth

function

f

and

some

probability

measure

PX

over

X

that

is

the

sample

point

is

of

the

form

-LRB-

x

f

-LRB-

x

-RRB-

-RRB-

Further

justification

for

the

smoothness

assumption

is

given

by

Poggio

and

Girosi

-LSB-

-RSB-

Definition

A

function

f

from

X

into

Y

is

called

a

Lipschitz

function

if

and

only

if

for

some

K

we

have

dY

-LRB-

f

-LRB-

x

-RRB-

f

-LRB-

-RRB-

-RRB-

KdX

-LRB-

x

-RRB-

VORONOI

ENCODERS

AND

QUANTIZATION

NUMBERS

for

all

x

X.

Let

kfkL

denote

the

smallest

such

K

A

class

of

functions

F

from

X

into

Y

is

called

Lipschitz

functions

if

and

only

if

for

some

K

we

have

sup

kfkL

K

Let

kFkL

denote

the

smallest

such

K

We

call

K

the

Lipschitz

bound

The

Lipschitz

bound

does

not

have

to

hold

everywhere

it

suffices

for

our

purpose

if

it

holds

with

probability

one

For

example

the

class

of

piece-wise

Lipschitz

functions

satisfies

this

relaxed

condition

Haussler

-LSB-

-RSB-

has

relaxed

the

Lipschitz

condition

further

Definition

For

each

f

F

and

real

ffl

-LRB-

f

ffl

-RRB-

is

the

real-valued

function

on

X

defined

by

-LRB-

f

ffl

x

-RRB-

supfdY

-LRB-

f

-LRB-

x

-RRB-

f

-LRB-

-RRB-

-RRB-

g

where

the

supremum

is

taken

over

all

X

for

which

dX

-LRB-

x

-RRB-

ffl

Let

PX

be

a

probability

measure

over

X

We

say

that

the

F

is

uniformly

Lipschitz

on

the

average

with

respect

to

PX

if

for

all

ffl

and

all

f

F

there

exists

some

K

such

that

E

-LSB-

-LRB-

f

ffl

K

x

-RRB-

-RSB-

ffl

Let

kFkPX

L

be

the

smallest

such

K.

For

a

class

PX

of

probability

measures

over

X

we

define

kFkPX

L

kFkPX

L

VORONOI

ENCODERS

AND

QUANTIZA

TION

NUMBERS

The

class

G

-LSB-

s

of

Voronoi

systems

-LRB-

nearest-neighbor

systems

-RRB-

is

defined

as

follows

We

can

specify

each

g

Gs

by

a

set

U

usg

of

size

s.

Let

V

or

-LRB-

uj

-RRB-

denote

the

Voronoi

region

for

the

point

uj

The

encoder

of

g

is

a

mapping

from

X

to

Ns

and

maps

x

X

to

j

if

and

only

if

x

V

or

-LRB-

uj

-RRB-

Let

Z

zsg

ae

Y

The

decoder

of

g

is

a

mapping

from

Ns

to

Y

defined

by

-LRB-

j

-RRB-

zj

In

other

words

the

system

maps

an

input

x

to

its

nearest

neighbor

in

U

and

then

outputs

the

value

stored

in

the

memory

location

corresponding

to

that

point

THE

PSEUDO-DIMENSION

OF

VORONOI

ENCODERS

We

call

the

encoders

of

Voronoi

systems

the

Voronoi

encoders

In

the

following

we

introduce

the

notion

of

quantization

number

which

characterizes

the

optimal

size

of

Voronoi

encoders

for

a

given

error

bound

The

quantization

number

can

be

substantially

smaller

than

the

covering

number

Definition

Let

PX

be

a

probability

measure

over

X

and

let

x

be

the

random

vector

corresponding

to

PX

For

any

ffl

the

quantization

number

QPX

-LRB-

X

ffl

dX

-RRB-

of

PX

is

defined

as

the

smallest

integer

s

such

that

there

exists

a

Voronoi

encoder

of

size

s

that

satisfies

E

h

dX

-LRB-

x

u

-LRB-

x

-RRB-

-RRB-

i

ffl

For

a

class

PX

of

probability

measures

over

X

we

define

QPX

-LRB-

X

ffl

dX

-RRB-

QPX

-LRB-

X

ffl

dX

-RRB-

THE

PSEUDO-DIMENSION

OF

VORONOI

EN

CODERS

Building

on

the

work

of

Vapnik

and

Chervonenkis

-LSB-

-RSB-

Pollard

-LSB-

-RSB-

Dudley

-LSB-

-RSB-

and

Devroye

-LSB-

-RSB-

Haussler

-LSB-

-RSB-

introduced

the

notion

of

pseudo-dimension

which

is

a

generalization

of

VC

dimension

He

first

defined

the

notion

of

fullness

of

sets

Definition

For

x

let

sign

-LRB-

x

-RRB-

if

x

else

sign

-LRB-

x

-RRB-

For

x

-LRB-

xk

-RRB-

m

let

sign

-LRB-

x

-RRB-

-LRB-

sign

-LRB-

-RRB-

sign

-LRB-

xm

-RRB-

-RRB-

and

for

A

ae

m

let

sign

-LRB-

A

-RRB-

fsign

-LRB-

y

-RRB-

j

y

Ag

For

any

A

ae

m

and

x

m

let

A

x

fy

x

j

y

Ag

that

is

the

translation

of

A

obtained

by

adding

the

vector

x

We

say

that

A

is

full

if

there

exists

x

m

such

that

sign

-LRB-

A

x

-RRB-

that

is

if

there

exists

some

translation

of

A

that

intersect

all

orthants

of

m.

For

example

hyperplanes

in

m

are

not

full

since

no

hyperplanes

in

m

can

intersect

all

orthants

of

m

The

pseudo-dimension

is

defined

as

follows

Definition

Let

F

be

a

class

of

functions

from

a

set

X

into

For

any

sequence

X

-LRB-

xm

-RRB-

of

points

in

X

let

F

-LRB-

X

-RRB-

f

-LRB-

f

-LRB-

-RRB-

f

-LRB-

xm

-RRB-

-RRB-

f

Fg

If

F

-LRB-

X

-RRB-

is

full

then

we

say

that

X

is

shattered

by

F

The

pseudo-dimension

of

F

denoted

by

dimP

-LRB-

F

-RRB-

is

the

largest

m

such

that

there

exists

a

sequence

of

m

points

in

X

that

is

shattered

by

F

If

arbitrarily

long

sequences

are

shattered

then

dimP

-LRB-

F

-RRB-

is

infinite

VORONOI

ENCODERS

AND

QUANTIZATION

NUMBERS

It

is

clear

when

F

is

a

class

of

functions

that

the

definition

of

the

pseudo-dimension

is

the

same

as

that

of

the

VC

dimension

Dudley

and

Haussler

have

shown

the

following

useful

property

of

pseudo-dimension

Theorem

-LRB-

Dudley

-LSB-

-RSB-

-RRB-

Let

F

be

a

k-dimensional

vector

space

of

functions

from

a

set

X

to

Then

dimP

-LRB-

F

-RRB-

k.

Theorem

-LRB-

Haussler

-LSB-

-RSB-

-RRB-

Let

F

be

a

class

of

function

from

a

set

X

into

Fix

any

nondecreasing

-LRB-

or

nonincreasing

-RRB-

function

h

and

let

H

fh

ffi

f

f

Fg

Then

we

have

dimP

-LRB-

H

-RRB-

dimP

-LRB-

F

-RRB-

To

derive

the

pseudo-dimension

of

Voronoi

encoders

we

use

the

following

lemma

attributed

to

Sauer

-LSB-

-RSB-

Lemma

-LSB-

Sauer

's

Lemma

-RSB-

Let

F

be

a

class

of

functions

from

S

mg

into

with

jF

j

and

let

d

be

the

length

of

the

longest

sequence

of

points

S

from

S

such

that

F

-LRB-

S

-RRB-

Then

we

have

jF

j

-LRB-

em

=d

-RRB-

d

where

e

is

the

base

of

the

natural

logarithm

We

now

are

ready

to

bound

the

pseudo-dimension

of

Voronoi

encoders

Lemma

Let

Gs

be

the

Voronoi

system

of

size

at

most

s

and

let

dX

be

the

Euclidean

metric

For

each

possible

encoder

of

Gs

we

define

f

-LRB-

x

-RRB-

dX

-LRB-

x

u

-LRB-

x

-RRB-

-RRB-

and

let

s

X

-LSB-

MX

-RSB-

be

the

class

of

all

such

functions

f

-LRB-

x

-RRB-

Then

we

have

dimP

-LRB-

s

-RRB-

-LRB-

k

-RRB-

s

log

-LRB-

-RRB-

O

-LRB-

ks

log

s

-RRB-

where

k

is

the

dimension

of

the

input

space

Proof

By

the

definition

of

the

Euclidean

metric

we

can

write

-LRB-

f

-LRB-

x

-RRB-

-RRB-

as

a

polynomial

in

k

variables

with

coefficients

where

k

is

the

dimension

of

the

input

space

By

Theorems

and

we

have

dimP

-LRB-

-RRB-

Let

X

be

a

sequence

of

m

points

in

X

and

let

r

be

an

arbitrary

m-vector

Since

f

-LRB-

x

-RRB-

dX

-LRB-

x

u

-RRB-

we

have

jsign

-LRB-

s

-LRB-

X

-RRB-

r

-RRB-

j

jsign

-LRB-

-LRB-

X

-RRB-

r

-RRB-

js

em

-LRB-

-RRB-

s

SAMPLE

SIZE

FORTHE

UNIFORMCONVERGENCEOF

The

last

inequality

follows

from

Sauer

's

Lemma

If

m

-LRB-

k

-RRB-

s

log

-LRB-

-RRB-

then

-LRB-

em

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

s

Therefore

we

have

dimP

-LRB-

s

-RRB-

-LRB-

k

-RRB-

s

log

-LRB-

-RRB-

O

-LRB-

ks

log

s

-RRB-

SAMPLE

SIZE

FOR

THE

UNIFORM

CONVER

GENCE

OF

VORONOI

ENCODERS

In

this

section

we

bound

the

sample

size

for

estimating

the

error

of

Voronoi

encoders

In

the

following

let

E?X

-LRB-

f

-RRB-

Pmi

f

-LRB-

xi

-RRB-

be

the

empirical

mean

of

the

function

f

and

let

d

-LRB-

r

t

-RRB-

jr

tj

=-LRB-

r

t

-RRB-

We

need

the

following

corollary

from

Haussler

and

Long

-LSB-

-RSB-

Corollary

Let

F

be

a

family

of

functions

from

a

set

X

into

-LSB-

MX

-RSB-

where

dimP

-LRB-

F

-RRB-

d

for

some

d

Let

PX

be

a

probability

measure

on

X.

Assume

and

ff

Let

X

be

generated

by

m

independent

draws

from

X

according

to

PX

If

the

sample

size

is

m

ln

-LRB-

ffpff

-RRB-

ln

ffi

then

we

have

F

j

d

-LRB-

E?X

-LRB-

f

-RRB-

E

-LRB-

f

-RRB-

-RRB-

ffg

ffi

Lemma

and

Corollary

imply

the

following

theorem

Theorem

Let

s

be

defined

as

in

Lemma

Assume

and

ff

Let

PX

be

a

probability

measure

on

X

and

X

be

generated

by

m

independent

draws

from

X

according

to

PX

If

the

sample

size

is

m

-LRB-

-RRB-

s

log

-LRB-

-RRB-

ln

-LRB-

ffpff

-RRB-

ln

ffi

then

we

have

s

j

d

-LRB-

E?X

-LRB-

f

-RRB-

E

-LRB-

f

-RRB-

-RRB-

ffg

ffi

MEMORY-EFFICIENT

LEARNING

OF

SMOOTH

FUNCTIONS

In

this

section

we

investigate

in

detail

three

methods

of

learning

smooth

functions

by

Voronoi

systems

ffl-covering

hashing

and

clustering

Our

results

are

summarized

MEMORY-EFFICIENT

LEARNING

OF

SMOOTH

FUNCTIONS

in

Table

First

we

introduce

some

notation

Let

h

-LRB-

-RRB-

-LRB-

xm

ym

-RRB-

i

be

a

random

sample

sequence

of

length

m

We

denote

the

sequence

xmi

by

X

We

denote

the

random

vector

corresponding

to

a

probability

measure

P

P

by

-LRB-

x

y

-RRB-

We

denote

the

average

empirical

distance

from

the

x-components

of

the

examples

to

U

by

d?X

-LRB-

U

-RRB-

m

m

X

i

dX

-LRB-

xi

U

-RRB-

The

discrete

version

of

the

above

problem

is

to

restrict

U

to

be

a

subset

of

X

The

learning

problem

is

specified

as

follows

We

are

given

a

class

G

of

Voronoi

systems

and

a

class

P

of

probability

measures

generated

by

a

class

PX

of

probability

measures

over

X

and

a

class

F

of

smooth

functions

from

X

to

Y

with

kFkPX

L

K.

Each

sample

point

is

of

the

form

-LRB-

x

f

-LRB-

x

-RRB-

-RRB-

for

some

f

F

Give

ffi

ffl

and

sample

sequence

h

-LRB-

-RRB-

-LRB-

xm

ym

-RRB-

i

the

goal

of

learning

is

to

construct

a

Voronoi

system

g

G

such

that

the

size

of

g

is

as

small

as

possible

and

the

expected

error

rate

satisfies

erP

-LRB-

g

-RRB-

fflMY

with

probability

at

least

ffi

LEARNING

BY

ffl-COVERING

The

main

idea

of

ffl-covering

is

to

cover

the

input

space

with

small

cells

of

radius

ffl

and

assign

each

cell

a

constant

value

The

smoothness

condition

assures

a

small

expected

error

for

the

resulting

system

The

algorithm

essentially

learns

by

brute

force

Algorithm

LE

-LRB-

learning

by

ffl-covering

-RRB-

Let

U

be

an

fflMY

cover

of

size

N

where

N

N

-LRB-

X

fflMY

dX

-RRB-

Let

m

ln

Nffi

be

the

sample

size

For

each

ui

U

if

Vor

-LRB-

ui

-RRB-

X

then

we

choose

an

arbitrary

yj

such

that

xj

Vor

-LRB-

ui

-RRB-

X

and

set

Z

-LRB-

i

-RRB-

yj

otherwise

we

set

Z

-LRB-

i

-RRB-

arbitrarily

Theorem

With

probability

at

least

ffi

the

expected

error

for

Algorithm

LE

satisfies

erP

-LRB-

LE

-LRB-

-RRB-

-RRB-

fflMY

LEARNING

BY

ffl-COVERING

Algorithm

System

size

Sample

size

ffl-covering

-LRB-

LE

-RRB-

N

O

N

ffl

log

N

ffi

perfect

hashing

-LRB-

-RRB-

O

ffi

-LRB-

pN

-RRB-

O

pN

ffl

log

pN

ffi

universal

hashing

-LRB-

-RRB-

O

ffl

pN

O

pN

ffl

log

pN

ffi

coalesced

hashing

O

-LRB-

pN

-RRB-

O

pN

ffl

log

pN

ffi

optimal

clustering

-LRB-

-RRB-

s

O

ks

ffl

log

s

log

ffl

ffl

log

ffi

approx

clustering

-LRB-

-RRB-

O

s

log

ks

ffl

log

log

ffi

O

@ks

ffl

log

s

log

ks

ffl

ffl

log

ffi

A

Table

Upper

bounds

on

system

size

and

sample

size

for

six

algorithms

for

learning

smooth

functions

by

Voronoi

systems

The

goal

of

learning

for

each

learning

algorithm

L

is

to

achieve

with

probability

at

least

ffi

an

error

bound

of

erP

-LRB-

L

-LRB-

-RRB-

-RRB-

fflMY

In

the

table

k

is

the

dimension

of

the

input

space

N

is

the

covering

number

N

-LRB-

X

fflMY

dX

-RRB-

p

o

is

the

fraction

of

nonempty

Voronoi

cells

and

s

is

the

quantization

number

QPX

-LRB-

X

fflMY

dX

-RRB-

Proof

For

each

Voronoi

cell

Vor

-LRB-

ui

-RRB-

satisfying

PX

-LRB-

Vor

-LRB-

ui

-RRB-

-RRB-

we

have

Pr

-LRB-

Vor

-LRB-

ui

-RRB-

X

;-RRB-

ffl

ln

Nffi

ffi

N

Therefore

with

probability

at

least

ffi

all

Voronoi

cells

with

probability

over

will

be

hit

by

some

sample

point

By

the

Lipschitz

bound

for

each

Voronoi

cell

hit

we

have

E

h

dY

-LRB-

z

-LRB-

x

-RRB-

y

-RRB-

j

Vor

-LRB-

ui

-RRB-

i

fflMY

-LRB-

Vor

-LRB-

ui

-RRB-

-RRB-

MEMORY-EFFICIENT

LEARNING

OF

SMOOTH

FUNCTIONS

Furthermore

the

total

probability

measure

of

Voronoi

cells

with

less

than

probability

is

at

most

ffl

Therefore

we

have

erP

-LRB-

LE

-LRB-

-RRB-

-RRB-

ffl

fflMY

fflMY

fflMY

LEARNING

BY

HASHING

Algorithm

LE

in

the

previous

section

covers

the

whole

input

space

X

with

points

However

most

of

the

Voronoi

cells

formed

by

points

in

the

ffl-cover

U

are

likely

to

be

empty

In

this

section

we

use

hashing

techniques

to

take

advantage

of

this

property

Below

we

outline

three

hashing-based

algorithms

perfect

hashing

universal

hashing

and

hashing

with

collision-resolution

Let

h

be

a

hash

function

from

NN

to

NN

where

N

jU

j

and

N

is

a

positive

integer

For

each

address

i

N

we

define

h

-LRB-

i

-RRB-

to

be

the

subset

of

points

in

X

that

hash

to

memory

location

i

namely

fxj

j

h

-LRB-

-LRB-

xj

-RRB-

-RRB-

i

and

xj

Xg

We

let

HN

N

be

a

class

of

universal

hash

functions

-LRB-

Carter

and

Wegman

-LSB-

-RSB-

-RRB-

from

NN

to

NN

For

the

ease

of

exposition

we

assume

in

the

following

that

the

portion

p

of

nonempty

Voronoi

cells

is

known

This

assumption

can

be

removed

using

the

techniques

of

Haussler

Kearns

Littlestone

and

Warmuth

-LSB-

-RSB-

Perfect

Hashing

The

first

algorithm

uses

uniform

hash

functions

and

resorts

to

large

physical

memory

to

assure

perfect

hashing

with

high

probability

Algorithm

-LRB-

learning

by

perfect

hashing

-RRB-

Let

U

be

an

fflMY

cover

of

size

N

where

N

N

-LRB-

X

fflMY

dX

-RRB-

and

let

p

o

be

the

fraction

of

non-empty

Voronoi

cells

Let

m

ln

be

the

sample

perfect

hashing

techniques

as

surveyed

by

Ramakrishna

and

Awasthi

-LSB-

-RSB-

assume

a

static

set

of

keys

so

we

are

not

able

to

use

these

techniques

for

learning

which

is

dynamic

in

nature

However

when

the

learning

is

complete

-LRB-

the

set

of

keys

-LRB-

addresses

-RRB-

is

fixed

-RRB-

we

can

use

perfect

hashing

techniques

to

reduce

the

size

of

physical

memory

LEARNING

BY

HASHING

size

Let

N

-LRB-

pN

-RRB-

be

the

size

of

physical

memory

Z

and

choose

a

uniform

hash

function

h.

For

each

address

i

if

h

-LRB-

i

-RRB-

is

not

empty

then

we

choose

an

arbitrary

j

m

such

that

xj

h

-LRB-

i

-RRB-

and

set

Z

-LRB-

i

-RRB-

yj

otherwise

we

set

Z

-LRB-

i

-RRB-

arbitrarily

Theorem

With

probability

at

least

ffi

the

expected

error

for

Algorithm

satisfies

erP

-LRB-

-LRB-

-RRB-

-RRB-

fflMY

Proof

Without

any

collision

by

similar

analysis

as

in

the

proof

of

Theorem

with

probability

at

least

ffi

we

have

erP

-LRB-

-LRB-

-RRB-

-RRB-

fflMY

By

choosing

physical

memory

size

as

N

-LRB-

pN

-RRB-

we

bound

the

probability

that

at

least

one

hashing

collision

occurs

by

-LRB-

pN

-RRB-

-LRB-

pN

-RRB-

ffi

Therefore

with

probability

at

least

ffi

we

have

no

collisions

and

erP

-LRB-

-LRB-

-RRB-

-RRB-

fflMY

Universal

Hashing

It

is

not

necessary

to

avoid

collisions

completely

What

we

really

need

is

a

good

hash

function

that

incurs

not

too

many

collisions

The

following

algorithm

uses

universal

hashing

for

finding

a

good

hash

function

with

high

probability

We

denote

the

Lipschitz

bound

by

K.

Algorithm

-LRB-

learning

by

universal

hashing

-RRB-

Let

U

be

an

fflMY

cover

of

size

N

where

N

N

-LRB-

X

fflMY

dX

-RRB-

and

let

p

o

be

the

fraction

of

non-empty

cells

Let

m

ln

be

the

sample

size

and

let

N

pN

be

the

size

of

physical

memory

Z.

Repeat

the

following

procedure

-LRB-

ffi

-RRB-

times

and

choose

the

system

with

minimum

empirical

error

We

choose

a

hash

function

h

randomly

from

the

class

HN

N

of

universal

hash

functions

and

then

call

the

subroutine

H

-LRB-

h

-RRB-

which

is

given

immediately

below

MEMORY-EFFICIENT

LEARNING

OF

SMOOTH

FUNCTIONS

Subroutine

H

Given

a

sample

sequence

and

a

hash

function

h

for

each

address

i

if

h

-LRB-

i

-RRB-

is

not

empty

then

we

choose

an

arbitrary

j

m

such

that

xj

h

-LRB-

i

-RRB-

and

set

Z

-LRB-

i

-RRB-

yj

otherwise

we

set

Z

-LRB-

i

-RRB-

arbitrarily

Theorem

With

probability

at

least

ffi

the

expected

error

for

Algorithm

satisfies

erP

-LRB-

-LRB-

-RRB-

-RRB-

fflMY

Proof

For

each

Voronoi

cell

Vor

-LRB-

ui

-RRB-

with

PX

-LRB-

Vor

-LRB-

ui

-RRB-

-RRB-

pN

we

have

Pr

-LRB-

Vor

-LRB-

ui

-RRB-

X

;-RRB-

ffl

ln

ffi

Therefore

using

sample

size

m

ln

with

probability

at

least

ffi

all

Voronoi

cells

with

probability

over

ffl

-LRB-

-RRB-

will

be

hit

by

some

sample

point

By

the

property

of

universal

hashing

-LRB-

Carter

and

Wegman

-LSB-

-RSB-

-RRB-

for

each

Voronoi

cell

hit

the

probability

that

the

cell

is

involved

in

some

hash

collision

is

at

most

pN

N

ffl

Therefore

by

the

Lipschitz

bound

we

have

E

h

dY

-LRB-

zh

-LRB-

-LRB-

x

-RRB-

-RRB-

y

-RRB-

j

Vor

-LRB-

ui

-RRB-

i

PX

-LRB-

Vor

-LRB-

ui

-RRB-

-RRB-

ffl

fflMY

ffl

-LRB-

Vor

-LRB-

ui

-RRB-

-RRB-

where

h

is

the

random

universal

hash

function

Thus

we

have

E

-LSB-

erP

-LRB-

A

-LRB-

h

-RRB-

-RRB-

-RSB-

ffl

fflMY

where

the

expectation

is

taken

over

HN

N

We

say

that

a

hash

function

h

is

good

if

the

following

inequality

holds

erP

-LRB-

H

-LRB-

h

-RRB-

-RRB-

fflMY

By

Markov

's

inequality

at

least

one

fourth

of

hash

functions

in

HN

N

are

good

Therefore

by

calling

subroutine

H

at

least

-LRB-

ffi

-RRB-

times

the

probability

that

we

do

not

get

a

good

hash

function

is

at

most

ffi

Thus

with

probability

at

least

ffi

we

have

erP

-LRB-

-LRB-

-RRB-

-RRB-

fflMY

LEARNING

BY

CLUSTERING

Hashing

with

Collision-Resolution

The

physical

memory

size

can

be

reduced

to

O

-LRB-

pN

-RRB-

while

maintaining

an

O

-LRB-

-RRB-

worstcase

access

time

by

using

collision-resolution

techniques

This

can

be

achieved

for

example

by

using

coalesced

hashing

which

was

analyzed

in

detail

by

Vitter

and

Chen

-LSB-

-RSB-

and

Siegel

-LSB-

-RSB-

LEARNING

BY

CLUSTERING

Although

hashing

techniques

take

advantage

of

the

sparseness

of

distributions

they

do

not

take

advantage

of

the

skewness

of

distributions

We

can

exploit

the

skewness

of

distributions

by

using

clustering

-LRB-

or

median

-RRB-

algorithms

Given

a

positive

integer

s

m

the

-LRB-

continuous

-RRB-

s-median

-LRB-

or

clustering

-RRB-

problem

is

to

find

a

median

set

U

X

such

that

jU

j

s

and

the

average

empirical

distortion

d?X

-LRB-

U

-RRB-

is

minimized

The

discrete

s-median

problem

is

to

restrict

U

to

be

a

subset

of

X.

For

simplicity

we

assume

in

the

following

that

the

quantization

number

s

QPX

-LRB-

X

fflMY

dX

-RRB-

is

known

This

assumption

can

be

removed

using

the

techniques

Haussler

Kearns

Littlestone

and

Warmuth

-LSB-

-RSB-

In

the

following

we

also

assume

that

the

Lipschitz

bound

holds

with

probability

one

Optimal

Clustering

Ideally

we

would

like

to

use

an

algorithm

for

finding

optimal

clustering

for

learning

MEMORY-EFFICIENT

LEARNING

OF

SMOOTH

FUNCTIONS

Algorithm

-LRB-

learning

by

optimal

clustering

-RRB-

Let

m

-LRB-

ksffl

log

s

log

log

-RRB-

be

the

sample

size

where

s

is

the

quantization

number

QPX

-LRB-

X

fflMY

dX

-RRB-

Find

the

optimal

U

such

that

d?X

-LRB-

U

-RRB-

is

minimized

For

each

ui

U

we

choose

j

m

such

that

xj

is

the

closest

point

to

ui

and

set

Z

-LRB-

i

-RRB-

yj

Theorem

With

probability

at

least

ffi

the

expected

error

for

Algorithm

satisfies

erP

-LRB-

-LRB-

-RRB-

-RRB-

fflMY

Proof

In

Theorem

we

choose

ff

and

let

fflMY

-LRB-

-RRB-

Thus

by

choosing

sample

size

as

-LRB-

ksffl

log

s

log

log

-RRB-

with

probability

at

least

ffi

for

all

V

ae

X

of

size

s

we

have

d?X

-LRB-

V

-RRB-

-LSB-

dX

-LRB-

x

V

-RRB-

-RSB-

fflMY

and

E

-LSB-

dX

-LRB-

x

V

-RRB-

-RSB-

d?X

-LRB-

V

-RRB-

fflMY

Let

U

be

the

optimal

median

set

of

size

s

with

respect

to

PX

then

we

have

E

-LSB-

dX

-LRB-

x

U

-RRB-

-RSB-

d?X

-LRB-

U

-RRB-

fflMY

d?X

-LRB-

U

-RRB-

fflMY

-LSB-

dX

-LRB-

x

U

-RRB-

-RSB-

fflMY

fflMY

Since

U

is

optimal

we

have

E

-LSB-

dX

-LRB-

x

U

-RRB-

-RSB-

fflMY

Therefore

E

-LSB-

dX

-LRB-

x

U

-RRB-

-RSB-

fflMY

fflMY

fflMY

fflMY

We

lose

another

factor

of

by

the

way

we

set

Z

-LRB-

i

-RRB-

The

rest

of

the

proof

follows

from

the

Lipschitz

bound

LEARNING

BY

CLUSTERING

Approximate

Clustering

Unfortunately

finding

optimal

clusters

is

NP-hard

even

in

Euclidean

space

-LRB-

Kariv

and

Hakimi

-LSB-

-RSB-

Garey

and

Johnson

-LSB-

-RSB-

Papadimitriou

-LSB-

-RSB-

Megiddo

-LSB-

-RSB-

-RRB-

However

as

shown

by

Lin

and

Vitter

-LSB-

-RSB-

we

have

approximate

clustering

algorithms

with

provably

good

performance

guarantees

We

may

use

these

approximate

clustering

algorithms

for

learning

Algorithm

-LRB-

learning

by

approximate

clustering

-RRB-

Let

m

-LRB-

ksffl

log

s

-LRB-

log

ksffl

-RRB-

log

-RRB-

be

the

sample

size

where

s

is

the

quantization

number

QPX

-LRB-

X

fflMY

dX

-RRB-

Apply

the

greedy

-LRB-

discrete

-RRB-

s-median

algorithm

of

Lin

and

Vitter

-LSB-

-RSB-

with

relative

error

bound

on

distortion

as

-LRB-

For

convenience

the

greedy

s-median

algorithm

is

given

in

the

appendix

-RRB-

Let

U

be

the

median

set

returned

by

the

greedy

s-median

algorithm

For

each

xj

ui

U

we

set

Z

-LRB-

i

-RRB-

yj

By

Corollary

in

the

Appendix

and

the

triangle

inequality

we

have

the

following

corollary

Corollary

Let

U

be

the

median

set

returned

by

the

greedy

s-median

algorithm

and

let

U

be

the

set

of

optimal

s-medians

Then

we

have

d?X

-LRB-

U

-RRB-

-LRB-

-RRB-

d?X

-LRB-

U

-RRB-

d?X

-LRB-

U

-RRB-

and

jU

j

O

-LRB-

s

log

m

-RRB-

Proof

Let

U

usg

be

the

set

of

optimal

s-medians

Then

by

replacing

each

point

ui

U

by

its

nearest

neighbor

vi

in

X

we

obtain

a

median

set

V

X

such

that

d?X

-LRB-

V

-RRB-

m

m

X

i

dX

-LRB-

xi

V

-RRB-

m

sX

i

X

-LRB-

ui

-RRB-

X

dX

-LRB-

x

V

-RRB-

m

sX

i

X

-LRB-

ui

-RRB-

X

dX

-LRB-

x

vi

-RRB-

MEMORY-EFFICIENT

LEARNING

OF

SMOOTH

FUNCTIONS

By

the

triangle

inequality

we

have

d?X

-LRB-

V

-RRB-

m

sX

i

X

-LRB-

ui

-RRB-

X

-LRB-

dX

-LRB-

x

ui

-RRB-

dX

-LRB-

ui

vi

-RRB-

-RRB-

m

sX

i

X

-LRB-

ui

-RRB-

X

-LRB-

x

ui

-RRB-

d?X

-LRB-

U

-RRB-

The

rest

of

the

proof

follows

from

Corollary

in

the

Appendix

Theorem

With

probability

at

least

ffi

the

expected

error

for

Algorithm

satisfies

erP

-LRB-

-LRB-

-RRB-

-RRB-

fflMY

Proof

We

apply

Theorem

with

ff

and

fflMY

-LRB-

-RRB-

By

using

m

-LRB-

ksffl

log

s

-LRB-

log

ksffl

-RRB-

log

-RRB-

sample

points

with

probability

at

least

ffi

for

all

V

ae

X

of

size

at

most

jU

j

we

have

d?X

-LRB-

V

-RRB-

-LSB-

dX

-LRB-

x

V

-RRB-

-RSB-

fflMY

and

E

-LSB-

dX

-LRB-

x

V

-RRB-

-RSB-

d?X

-LRB-

V

-RRB-

fflMY

Let

U

be

the

set

of

optimal

s-medians

By

Corollary

and

by

algebraic

manipulations

similar

to

the

proof

of

Theorem

we

have

E

-LSB-

dX

-LRB-

x

U

-RRB-

-RSB-

fflMY

K

The

rest

of

the

proof

follows

from

the

Lipschitz

bound

SUMMARY

We

summarize

the

results

of

this

section

in

Table

We

remark

that

in

k

the

covering

number

is

exponential

in

the

dimensionality

of

the

input

space

That

is

we

have

N

N

-LRB-

X

fflMY

dX

-RRB-

-LRB-

-LRB-

-RRB-

k

-RRB-

On

the

other

hand

in

practice

it

is

reasonable

to

assume

that

the

quantization

number

s

QPX

-LRB-

X

fflMY

dX

-RRB-

is

a

low-degree

polynomial

in

as

explained

in

Section

In

such

typical

cases

clustering

algorithms

reduce

the

dependency

of

memory

size

on

dimensionality

by

an

exponential

factor

TREE-STRUCTURED

SYSTEMS

In

a

tree-structured

system

the

encoder

partitions

the

input

space

into

a

hierarchy

of

regions

An

input

is

mapped

to

the

memory

location

corresponding

to

the

region

represented

by

a

leaf

The

full

definition

of

tree-structured

systems

is

given

in

Section

We

call

the

encoders

of

tree-structured

systems

the

tree-structured

encoders

Lemma

Let

Gs

be

the

tree-structured

systems

of

size

s

and

let

dX

be

the

Euclidean

metric

For

each

possible

encoder

of

Gs

we

define

f

-LRB-

x

-RRB-

dX

-LRB-

x

u

-LRB-

x

-RRB-

-RRB-

and

let

s

X

-LSB-

MX

-RSB-

be

the

class

of

all

such

functions

Then

we

have

dimP

-LRB-

s

-RRB-

-LRB-

k

-RRB-

-LRB-

s

-RRB-

log

-LRB-

-LRB-

s

-RRB-

-RRB-

O

-LRB-

ks

log

s

-RRB-

Proof

There

are

s

branches

in

a

tree

of

size

s

in

which

each

branch

corresponds

to

a

comparison

By

derivation

similar

to

the

proof

of

Lemma

we

have

dimP

-LRB-

s

-RRB-

-LRB-

k

-RRB-

-LRB-

s

-RRB-

log

-LRB-

-LRB-

s

-RRB-

-RRB-

O

-LRB-

ks

log

s

-RRB-

Lemma

and

Corollary

imply

the

following

result

Theorem

Let

s

be

defined

as

in

Lemma

Assume

and

ff

Let

PX

be

a

probability

measure

on

X

and

X

be

generated

by

m

independent

draws

from

X

according

to

PX

If

the

sample

size

is

m

MX

ks

log

s

log

MX

-LRB-

ffpff

-RRB-

log

ffi

then

we

have

s

j

d

-LRB-

E?X

-LRB-

f

-RRB-

E

-LRB-

f

-RRB-

-RRB-

ffg

ffi

In

the

following

we

outline

an

algorithm

for

building

tree-structured

systems

Construct

a

tree-structured

encoder

for

the

input

space

from

the

x-components

of

the

sample

Estimate

a

functional

value

for

each

node

of

the

tree

by

averaging

the

ycomponents

of

examples

covered

by

the

region

represented

by

that

node

The

smoothness

of

the

function

to

be

learned

assures

that

the

resulting

system

has

small

expected

error

The

algorithm

for

building

a

tree-structured

encoder

is

given

HIGHER-ORDER

SYSTEMS

by

Lin

and

Vitter

-LSB-

-RSB-

In

addition

to

memory-based

learning

the

algorithm

also

has

applications

to

regression

computer

graphics

and

lossy

image

compression

-LRB-

Lin

and

Vitter

-LSB-

-RSB-

-RRB-

HIGHER-ORDER

SYSTEMS

In

a

higher-order

memory-based

learning

system

an

input

can

activate

more

than

one

memory

location

Higher-order

learning

systems

have

the

advantages

of

fault

tolerance

and

possibly

better

generalization

ability

given

a

limited

number

of

examples

By

fault

tolerance

we

mean

the

capability

to

deal

with

memory

failures

or

misclassification

of

sample

points

In

this

section

we

look

at

the

r-nearest-neighbor

systems

and

receptive-field

systems

based

upon

the

combinations

of

first-order

systems

VORONOI

SYSTEMS

OF

HIGHER

ORDER

The

definition

for

the

Voronoi

systems

of

order

r

-LRB-

r-nearest-neighbor

systems

-RRB-

is

given

in

Section

In

this

section

we

extend

our

analysis

in

Section

to

the

rth-order

Voronoi

Systems

We

call

the

encoders

of

Voronoi

systems

of

order

r

the

Voronoi

encoders

of

order

r.

Lemma

Let

Grs

be

the

Voronoi

systems

of

order

r

and

size

s

and

let

dX

be

the

Euclidean

distance

For

each

possible

encoder

of

Grs

we

define

f

-LRB-

x

-RRB-

r

rX

i

dX

-LRB-

x

u

-LRB-

i

-RRB-

-LRB-

x

-RRB-

-RRB-

where

-LRB-

i

-RRB-

-LRB-

x

-RRB-

maps

an

input

x

to

its

ith

nearest

neighbor

in

U

and

let

rs

X

-LSB-

MX

-RSB-

be

the

class

of

all

such

functions

Then

we

have

dimP

-LRB-

rs

-RRB-

O

-LRB-

krs

log

r

log

s

-RRB-

Proof

By

the

definition

of

f

-LRB-

x

-RRB-

it

is

clear

that

the

pseudo-dimension

of

rs

is

bounded

by

the

pseudo-dimension

of

sums

of

r

functions

from

s

which

is

defined

as

in

Lemma

By

derivation

similar

to

the

proof

of

Lemma

we

have

dimP

-LRB-

rs

-RRB-

O

-LRB-

krs

log

r

log

s

-RRB-

Lemma

and

Corollary

imply

the

following

RECEPTIVE-FIELD

SYSTEMS

Theorem

Let

rs

be

defined

as

in

Lemma

Assume

and

ff

Let

PX

be

a

probability

measure

on

X

and

X

be

generated

by

m

independent

draws

from

X

according

to

PX

If

the

sample

size

is

m

MX

krs

log

r

log

s

log

MX

-LRB-

ffpff

-RRB-

log

ffi

then

we

have

rs

j

d

-LRB-

E?X

-LRB-

f

-RRB-

E

-LRB-

f

-RRB-

-RRB-

ffg

ffi

RECEPTIVE-FIELD

SYSTEMS

In

a

receptive-field

system

the

regions

may

overlap

In

this

section

we

propose

to

model

the

receptive-field

systems

as

weighted

sums

of

first-order

Voronoi

systems

Definition

Let

Gs

be

the

class

of

-LRB-

first-order

-RRB-

Voronoi

systems

as

defined

in

Section

The

r-combinations

Grs

of

Voronoi

systems

are

defined

as

the

weighted

sums

of

r

Voronoi

systems

That

is

Grs

fPri

wigi

j

gi

Gs

and

wi

MY

g

A

receptive-field

system

as

defined

above

can

be

arranged

in

a

multi-resolution

manner

-LRB-

Moody

-LSB-

-RSB-

-RRB-

that

is

as

a

sum

of

r

Voronoi

systems

of

different

sizes

The

learning

algorithm

for

such

systems

can

start

by

approximating

the

function

to

be

learned

by

the

smallest

-LRB-

lowest-resolution

-RRB-

component

system

and

then

approximating

the

errors

by

the

second

smallest

component

system

and

so

forth

until

the

largest

-LRB-

highest-resolution

-RRB-

component

system

is

trained

CONCLUSIONS

In

this

paper

we

propose

a

model

for

memory-based

learning

and

use

it

to

analyze

several

methods

for

learning

smooth

functions

by

memory-based

learning

systems

Our

model

is

closely

related

to

the

generalized

PAC

learning

model

of

Haussler

-LSB-

-RSB-

and

the

methods

of

vector

quantization

in

data

compression

Our

main

result

is

that

we

can

build

memory-based

learning

systems

using

new

clustering

algorithms

-LRB-

Lin

and

Vitter

-LSB-

-RSB-

-RRB-

to

PAC-learn

in

polynomial

time

using

only

polynomial

storage

in

typical

situations

We

also

extend

our

analysis

to

tree-structured

and

higher-order

memory-based

learning

systems

CONCLUSIONS

The

memory-based

learning

systems

that

we

have

examined

in

this

paper

approximate

the

functional

value

in

each

region

by

a

constant

In

practice

we

might

get

better

approximations

by

using

more

complicated

basis

functions

However

this

usually

makes

the

training

problem

harder

most

work

along

this

line

has

been

mostly

experimental

in

terms

of

computational

complexity

Interested

readers

are

referred

to

the

work

of

Friedman

-LSB-

-RSB-

Moody

and

Darken

-LSB-

-RSB-

and

Poggio

and

Girosi

-LSB-

-RSB-

Our

memory-based

learning

algorithms

mainly

take

advantage

of

the

skewness

of

distributions

over

the

input

space

and

assume

the

smoothness

of

functions

over

the

input

space

However

the

degree

of

smoothness

may

vary

widely

from

one

region

to

the

other

-LRB-

Dean

-LSB-

-RSB-

-RRB-

In

practice

after

the

initial

clustering

we

may

estimate

the

degree

of

smoothness

of

each

region

and

then

merge

or

split

regions

according

to

their

degrees

of

smoothness

Alternatively

we

could

incorporate

either

a

priori

or

observed

local

estimates

of

smoothness

directly

into

the

distance

matrix

in

such

a

way

so

that

minimizing

the

distance

corresponds

to

minimizing

the

error

for

example

we

could

define

dX

-LRB-

x

-RRB-

to

be

dY

-LRB-

f

-LRB-

x

-RRB-

f

-LRB-

-RRB-

-RRB-

From

a

theoretical

viewpoint

we

must

develop

models

that

adequately

capture

this

property

and

are

computationally

tractable

Appendix

Approximate

Clustering

In

this

appendix

we

adapt

the

greedy

-LRB-

discrete

-RRB-

s-median

algorithm

of

Lin

and

Vitter

-LSB-

-RSB-

to

do

the

clustering

needed

for

Algorithm

in

Section

The

discrete

s-median

problem

is

defined

as

follows

Let

X

xmi

be

a

sequence

of

points

in

X

and

let

s

be

a

positive

integer

The

goal

is

to

select

a

subset

U

X

of

s

points

such

that

the

average

distance

-LRB-

distortion

-RRB-

d?X

-LRB-

U

-RRB-

m

m

X

i

dX

-LRB-

xi

U

-RRB-

is

minimized

The

discrete

s-median

problem

can

be

formulated

as

a

integer

program

of

minimizing

m

m

X

i

m

X

j

dX

-LRB-

xi

xj

-RRB-

pij

-LRB-

-RRB-

subject

to

m

X

j

pij

i

m

-LRB-

-RRB-

m

X

j

qj

s

-LRB-

-RRB-

pij

qj

i

j

m

-LRB-

-RRB-

pij

qj

i

j

m

-LRB-

-RRB-

where

qj

if

and

only

if

xj

is

chosen

as

a

cluster

center

and

pij

if

and

only

if

qj

and

xi

is

assigned

to

xj

The

linear

program

relaxation

of

the

above

program

is

to

allow

qj

and

pij

to

take

rational

values

between

and

Clearly

the

optimal

fractional

solution

-LRB-

linear

program

solution

-RRB-

is

a

lower

bound

on

the

solutions

of

the

discrete

s-median

problem

Our

greedy

algorithm

for

the

s-median

problem

works

as

follows

Solve

the

linear

program

relaxation

of

the

discrete

s-median

problem

by

linear

programming

techniques

denote

the

fractional

solution

by

bq

bp

For

each

i

compute

c

Di

Pmj

dX

-LRB-

xi

xj

-RRB-

bpij

Given

a

relative

error

bound

c

for

each

j

such

that

bqj

construct

a

set

Sj

A

point

xi

is

in

Sj

if

and

only

if

dX

-LRB-

xi

xj

-RRB-

-LRB-

c

-RRB-

c

Di

Apply

the

greedy

set

cover

algorithm

-LRB-

Johnson

-LSB-

-RSB-

Lov?asz

-LSB-

-RSB-

Chv?atal

-LSB-

-RSB-

-RRB-

Choose

the

set

which

covers

the

most

uncovered

points

Repeat

this

process

until

all

points

are

covered

Let

U

be

the

set

of

indices

of

sets

chosen

by

the

greedy

heuristic

Output

U

as

the

median

set

The

linear

programming

problem

can

be

solved

in

provably

polynomial

time

by

the

ellipsoid

algorithm

-LRB-

Khachian

-LSB-

-RSB-

-RRB-

or

by

the

interior

point

method

-LRB-

Karmarkar

-LSB-

-RSB-

-RRB-

The

simplex

method

-LRB-

Dantzig

-LSB-

-RSB-

-RRB-

works

very

efficiently

in

practice

although

in

the

worst

case

its

performance

is

not

polynomial-time

The

results

of

Lin

and

Vitter

-LSB-

-RSB-

yield

the

following

application

Corollary

Given

any

c

the

greedy

algorithm

outputs

a

median

set

U

of

size

less

than

-LRB-

c

-RRB-

s

-LRB-

ln

n

-RRB-

CONCLUSIONS

such

that

d?X

-LRB-

U

-RRB-

-LRB-

c

-RRB-

c

D

where

c

D

is

the

average

distance

of

the

optimal

fractional

solution

for

the

discrete

s-median

problem

Acknowledgements

Support

was

provided

in

part

by

a

National

Science

Foundation

Presidential

Young

Investigator

Award

CCR

-LCB-

with

matching

funds

from

IBM

by

NSF

research

grant

CCR

-LCB-

and

by

Army

Research

Office

grant

-LCB-

-LCB-

G

-LCB-

References

-LSB-

-RSB-

J.

S.

Albus

Brains

Behaviour

and

Robotics

Byte

Books

Peterborough

NH

-LSB-

-RSB-

J.

S.

Albus

Data

Storage

in

the

Cerebellar

Model

Articulation

Controller

-LRB-

CMAC

-RRB-

Journal

of

Dynamic

Systems

Measurement

and

Control

-LRB-

September

-RRB-

-LCB-

-LSB-

-RSB-

J.

S.

Albus

A

New

Approach

to

Manipulator

Control

The

Cerebellar

Model

Articulation

Controller

-LRB-

CMAC

-RRB-

Journal

of

Dynamic

Systems

Measurement

and

Control

-LRB-

September

-RRB-

-LCB-

-LSB-

-RSB-

J.

L.

Carter

and

M.

N.

Wegman

Universal

Classes

of

Hash

Functions

Journal

of

Computer

System

and

Science

-LRB-

April

-RRB-

-LCB-

-LSB-

-RSB-

V.

Chv?atal

A

Greedy

Heuristic

for

the

Set-Covering

Problem

Mathematics

of

Operations

Research

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

G.

Dantzig

Programming

of

Interdependent

Activities

II

Mathematical

Models

in

Activity

Analysis

of

Production

and

Allocation

John

Wiley

Sons

Inc

New

York

-LCB-

-LSB-

-RSB-

T.

L.

Dean

and

M.

P.

Wellman

Planning

and

Control

Morgan

Kaufmann

Publishers

-LSB-

-RSB-

L.

Devroye

Automatic

Pattern

Recognition

A

Study

of

the

Probability

of

Error

IEEE

Transactions

on

Pattern

Recognition

and

Machine

Intelligence

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

R.

M.

Dudley

Central

Limit

Theorems

for

Empirical

Measures

Annals

of

Probability

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

R.

M.

Dudley

A

Course

on

Empirical

Processes

in

Lecture

Note

in

Mathematics

-LSB-

-RSB-

J.

H.

Friedman

Multivariate

Adaptive

Regression

Splines

Technical

Report

Standford

University

Lab

for

Computational

Statistics

-LSB-

-RSB-

M.

R.

Garey

and

D.

S.

Johnson

Computers

and

intractability

A

Guide

to

the

Theory

of

NP-completeness

W.

H.

Freeman

and

Co.

San

Francisco

CA

-LSB-

-RSB-

A.

Gersho

On

the

Structure

of

Vector

Quantizers

IEEE

Transactions

on

Information

Theory

-LRB-

March

-RRB-

-LCB-

-LSB-

-RSB-

A.

Gersho

and

R.

M.

Gray

Vector

Quantization

and

Signal

Compression

Kluwer

Academic

Press

Massachusetts

-LSB-

-RSB-

R.

M.

Gray

Vector

Quantization

IEEE

ASSP

Magazine

-LRB-

April

-RRB-

-LCB-

-LSB-

-RSB-

D.

Haussler

Generalizing

the

PAC

Model

Sample

Size

Bounds

from

Metric

Dimension-Based

Uniform

Convergence

Results

in

Proceedings

of

the

Annual

IEEE

Symposium

on

Foundations

of

Computer

Science

-LCB-

-LSB-

-RSB-

D.

Haussler

M.

Kearns

N.

Littlestone

and

M.

K.

Warmuth

Equivalence

of

Models

for

Polynomial

Learnability

Information

and

Computation

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

D.

Haussler

and

P.

Long

A

Generalization

of

Sauer

's

Lemma

April

-LSB-

-RSB-

D.

S.

Johnson

Approximation

Algorithms

for

Combinatorial

Problems

Journal

of

Computer

and

System

Sciences

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

O.

Kariv

and

S.

L.

Hakimi

An

Algorithmic

Approach

to

Network

Location

Problems

II

The

p-Medians

SIAM

Journal

on

Applied

Mathematics

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

N.

Karmarkar

A

New

Polynomial-Time

Algorithm

for

Linear

Programming

Combinatorica

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

L.

G.

Khachiyan

A

Polynomial

Algorithm

in

Linear

Programming

Soviet

Math

Doklady

-LRB-

-RRB-

-LCB-

CONCLUSIONS

-LSB-

-RSB-

J.-H

Lin

and

J.

S.

Vitter

Nearly

Optimal

Vector

Quantization

via

Linear

Programming

in

Proceedings

of

the

IEEE

Data

Compression

Conference

Snowbird

Utah

March

-LCB-

-LSB-

-RSB-

J.-H

Lin

and

J.

S.

Vitter

ffl-Approximations

with

MinimumPacking

Constraint

Violation

in

Proceedings

of

the

Annual

ACM

Symposium

on

Theory

of

Computing

Victoria

BC

Canada

May

-LCB-

-LSB-

-RSB-

L.

Lov?asz

On

the

Ratio

of

Optimal

Integral

and

Fractional

Covers

Discrete

Mathematics

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

N.

Megiddo

and

K.

J.

Supowit

On

the

Complexity

of

Some

Common

Geometric

Location

Problems

SIAM

Journal

on

Computing

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

W.

T.

Miller

Sensor-Based

Control

of

Robotic

Manipulators

Using

a

General

Learning

Algorithms

IEEE

Journal

of

Robotics

and

Automation

-LRB-

April

-RRB-

-LCB-

-LSB-

-RSB-

W.

T.

Miller

F.

H.

Glanz

and

L.

G.

Kraft

Application

of

a

General

Learning

Algorithm

to

the

Control

of

Robotic

Manipulators

International

Journal

of

Robotics

Research

-LRB-

Summer

-RRB-

-LCB-

-LSB-

-RSB-

J.

Moody

Fast

Learning

in

Multi-Resolution

Hierarchies

in

Advances

in

Neural

Information

Processing

Systems

I

Morgan

Kaufmann

Publisher

-LCB-

-LSB-

-RSB-

J.

Moody

and

C.

Darken

Learning

with

Localized

Receptive

Fields

in

Proceedings

of

the

Connectionist

Models

Summer

School

Morgan

Kaufmann

Publisher

-LCB-

-LSB-

-RSB-

A.

W.

Moore

Acquisition

of

Dynamic

Control

Knowledge

for

Robotic

Manipulator

Manuscript

-LSB-

-RSB-

C.

H.

Papadimitriou

Worst-case

and

Probabilistic

Analysis

of

a

Geometric

Location

Problem

SIAM

Journal

on

Computing

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

T.

Poggio

and

F.

Girosi

A

Theory

of

Networks

for

Approximation

and

Learning

MIT

Artificial

Intelligence

Laboratory

A.

I.

Memo

No

Boston

MA

-LSB-

-RSB-

T.

Poggio

and

F.

Girosi

Extensions

of

a

Theory

of

Networks

for

Approximation

and

Learning

Dimensionality

Reduction

and

Clustering

MIT

Artificial

Intelligence

Laboratory

A.

I.

Memo

No.

Boston

MA

-LSB-

-RSB-

D.

Pollard

Convergence

of

Stochastic

Processes

Springer-Verlag

New

York

Inc

-LSB-

-RSB-

D.

Pollard

Empirical

Processes

Theory

and

Applications

NSF-CBMS

Regional

Conference

Series

in

Probability

and

Statistics

Volume

-LSB-

-RSB-

M.

V.

Ramakrishna

and

V.

Awasthi

Manuscript

December

-LSB-

-RSB-

E.

A.

Riskin

Variable

Rate

Vector

Quantization

of

Images

Ph.

D.

Dissertation

Stanford

University

-LSB-

-RSB-

N.

Sauer

On

the

Density

of

Families

of

Sets

Journal

of

Combinatorial

Theory

-LRB-

A

-RRB-

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

A.

Siegel

Coalesced

Hashing

is

Computably

Good

Manuscript

-LSB-

-RSB-

V.

N.

Vapnik

Estimation

of

Dependences

Based

on

Empirical

Data

Springer

Verlag

New

York

-LSB-

-RSB-

V.

N.

Vapnik

and

A.

Ya

Chervonenkis

On

the

Uniform

Convergence

of

Relative

Frequencies

of

Events

to

their

Probabilities

Theory

of

Probability

and

its

Applications

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

J.

S.

Vitter

and

W.-C

Chen

Design

and

Analysis

of

Coalesced

Hashing

Oxford

University

Press

approximation
clustering
covering
hashing
linear programming
pac learning
relaxation