Locally

Weighted

Learning

Christopher

G.

Atkeson

Andrew

W.

Moorey

and

Stefan

Schaalz

College

of

Computing

Georgia

Institute

of

Technology?z

Atlantic

Drive

Atlanta

GA

cga@cc.gatech.edu,

sschaal@cc.gatech.edu

fax

Carnegie

Mellon

Universityy

Forbes

Ave

Pittsburgh

PA

awm@cs.cmu.edu

ATR

Human

Information

Processing

Research

Laboratoriesz

Hikaridai

Seika-cho

Soraku-gun

Kyoto

Japan

May

at

Abstract

This

paper

surveys

locally

weighted

learning

a

form

of

lazy

learning

and

memorybased

learning

and

focuses

on

locally

weighted

linear

regression

The

survey

discusses

distance

functions

smoothing

parameters

weighting

functions

local

model

structures

regularization

of

the

estimates

and

bias

assessing

predictions

handling

noisy

data

and

outliers

improving

the

quality

of

predictions

by

tuning

fit

parameters

interference

between

old

and

new

data

implementing

locally

weighted

learning

efficiently

and

applications

of

locally

weighted

learning

A

companion

paper

surveys

how

locally

weighted

learning

can

be

used

in

robot

learning

and

control

Introduction

Lazy

learning

methods

defer

processing

of

training

data

until

a

query

needs

to

be

answered

This

usually

involves

storing

the

training

data

in

memory

and

finding

relevant

data

in

the

database

to

answer

a

particular

query

This

type

of

learning

is

also

referred

to

as

memory-based

learning

Relevance

is

often

measured

using

a

distance

function

with

nearby

points

having

high

relevance

One

form

of

lazy

learning

finds

a

set

of

nearest

neighbors

and

selects

or

votes

on

the

predictions

made

by

each

of

the

stored

points

This

paper

surveys

another

form

of

lazy

learning

locally

weighted

learning

that

uses

locally

weighted

training

to

average

interpolate

between

extrapolate

from

or

otherwise

combine

training

data

-LRB-

Vapnik

Bottou

and

Vapnik

Vapnik

and

Bottou

-RRB-

In

most

learning

methods

a

single

global

model

is

used

to

fit

all

of

the

training

data

Since

the

query

to

be

answered

is

known

during

processing

of

training

data

training

queryspecific

local

models

is

possible

in

lazy

learning

Local

models

attempt

to

fit

the

training

data

only

in

a

region

around

the

location

of

the

query

-LRB-

the

query

point

-RRB-

Examples

of

types

of

local

models

include

nearest

neighbor

weighted

average

and

locally

weighted

regression

-LRB-

Figure

-RRB-

Each

of

these

local

models

combine

points

near

a

query

point

to

estimate

the

appropriate

output

Nearest

neighbor

local

models

simply

choose

the

closest

point

and

use

its

output

value

Weighted

average

local

models

average

the

outputs

of

nearby

points

inversely

weighted

by

their

distance

to

the

query

point

Locally

weighted

regression

fits

a

surface

to

nearby

points

using

a

distance

weighted

regression

Weighted

averages

and

locally

weighted

regression

will

be

discussed

in

the

following

sections

and

our

survey

focuses

on

locally

weighted

linear

regression

The

core

of

the

survey

discusses

distance

functions

smoothing

parameters

weighting

functions

and

local

model

structures

Among

the

lessons

learned

from

research

on

locally

weighted

learning

are

that

practical

implementations

require

dealing

with

locally

inadequate

amounts

of

training

data

regularization

of

the

estimates

by

deliberate

introduction

of

bias

methods

for

predicting

prediction

quality

filtering

of

noise

and

identifying

outliers

automatic

tuning

of

the

learning

algorithm

's

parameters

to

specific

tasks

or

data

sets

and

efficient

implementation

techniques

Our

motivation

for

exploring

locally

weighted

learning

techniques

came

from

their

suitability

for

real

time

online

robot

learning

because

of

their

fast

incremental

learning

and

their

avoidance

of

negative

interference

between

old

and

new

training

data

We

provide

an

example

of

interference

to

clarify

this

point

We

briefly

survey

published

applications

of

locally

weighted

learning

A

companion

paper

-LRB-

Atkeson

Nearest

neighbor

Weighted

average

Locally

weighted

regression

Figure

Fits

using

different

types

of

local

models

for

three

and

five

data

points

et

al.

-RRB-

surveys

how

locally

weighted

learning

can

be

used

in

robot

learning

and

control

This

review

is

augmented

by

a

Web

page

-LRB-

Atkeson

-RRB-

This

review

emphasizes

a

statistical

view

of

learning

in

which

function

approximation

plays

the

central

role

In

order

to

be

concrete

the

review

focuses

on

a

narrow

problem

formulation

in

which

training

data

consists

of

input

vectors

of

specific

attribute

values

and

the

corresponding

output

values

Both

the

input

and

output

values

are

assumed

to

be

continuous

Alternative

approaches

for

this

problem

formulation

include

other

statistical

nonparametric

regression

techniques

multi-layer

sigmoidal

neural

networks

radial

basis

functions

regression

trees

projection

pursuit

regression

and

global

regression

techniques

The

discussion

section

-LRB-

Section

-RRB-

argues

that

locally

weighted

learning

can

be

applied

in

a

much

broader

context

Global

learning

methods

can

often

be

improved

by

localizing

them

using

locally

weighted

training

criteria

-LRB-

Vapnik

Bottou

and

Vapnik

Vapnik

and

Bottou

-RRB-

Although

this

survey

emphasizes

regression

applications

-LRB-

real

valued

outputs

-RRB-

the

discussion

section

outlines

how

these

techniques

have

been

applied

in

classification

-LRB-

discrete

outputs

-RRB-

We

conclude

with

a

short

discussion

of

future

research

directions

Notation

In

this

paper

scalars

are

represented

by

italic

lower

case

letters

-LRB-

y

-RRB-

Column

vectors

are

represented

as

boldface

lower

case

letters

-LRB-

x

-RRB-

and

row

vectors

are

represented

as

the

column

vectors

transposed

-LRB-

xT

-RRB-

Matrices

are

represented

by

bold

face

upper

case

letters

-LRB-

X

-RRB-

Distance

Weighted

Averaging

To

illustrate

how

locally

weighted

learning

using

a

distance

function

is

applied

we

will

first

consider

a

simple

example

distance

weighted

averaging

This

will

turn

out

to

be

a

form

of

locally

weighted

regression

in

which

the

local

model

is

a

constant

A

prediction

y

can

be

based

on

an

average

of

n

training

values

yng

y

P

yi

n

-LRB-

-RRB-

This

estimate

minimizes

a

criterion

C

X

i

-LRB-

y

yi

-RRB-

-LRB-

-RRB-

In

the

case

where

the

training

values

yng

are

taken

under

different

conditions

xng

it

makes

sense

to

emphasize

data

that

is

similar

to

the

query

q

and

deemphasize

dissimilar

data

rather

than

treat

all

the

training

data

equally

We

can

do

this

in

two

equivalent

ways

weighting

the

data

directly

or

weighting

the

error

criterion

used

to

choose

y.

Weighting

the

Data

Directly

Weighting

the

data

can

be

viewed

as

replicating

relevant

instances

and

discarding

irrelevant

instances

In

our

case

an

instance

is

represented

as

a

data

point

-LRB-

x

y

-RRB-

Relevance

is

measured

by

calculating

a

distance

d

-LRB-

xi

q

-RRB-

between

the

query

point

q

and

each

data

point

input

vector

xi

A

typical

distance

function

is

the

Euclidean

distance

-LRB-

xi

is

the

ith

input

vector

while

xj

is

the

jth

component

of

the

vector

x

-RRB-

dE

-LRB-

x

q

-RRB-

sX

j

-LRB-

xj

qj

-RRB-

q

-LRB-

x

q

-RRB-

T

-LRB-

x

q

-RRB-

-LRB-

-RRB-

A

weighting

function

or

kernel

function

K

-LRB-

-RRB-

is

used

to

calculate

a

weight

for

that

data

point

from

the

distance

A

typical

weighting

function

is

a

Gaussian

-LRB-

Figure

-RRB-

K

-LRB-

d

-RRB-

-LRB-

-RRB-

The

weight

is

then

used

in

a

weighted

average

y

-LRB-

q

-RRB-

P

yiK

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

PK

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

-LRB-

-RRB-

Note

that

the

estimate

y

depends

on

the

location

of

the

query

point

q.

Weighting

the

Error

Criterion

We

are

trying

to

find

the

best

estimate

for

the

outputs

yi

using

a

local

model

that

is

a

constant

y.

Distance

weighting

the

error

criterion

corresponds

to

requiring

the

local

Query

Point

Figure

Unweighted

averaging

using

springs

Query

Point

Figure

Locally

weighted

averaging

using

springs

model

to

fit

nearby

points

well

with

less

concern

for

distant

points

C

-LRB-

q

-RRB-

nX

i

h

-LRB-

y

yi

-RRB-

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

i

-LRB-

-RRB-

The

best

estimate

y

-LRB-

q

-RRB-

will

minimize

the

cost

C

-LRB-

q

-RRB-

For

that

value

of

y

@C

y

This

is

achieved

by

the

y

given

in

Equation

and

so

in

this

case

weighting

the

error

criterion

and

weighting

the

data

are

equivalent

Note

that

both

the

criterion

C

-LRB-

q

-RRB-

and

the

estimate

y

-LRB-

q

-RRB-

depend

on

the

location

of

the

query

point

q

This

process

has

a

physical

interpretation

Figures

and

show

the

data

points

-LRB-

black

dots

-RRB-

which

are

fixed

in

space

pulling

on

a

horizontal

line

-LRB-

the

constant

model

-RRB-

with

springs

The

strength

of

the

springs

are

equal

in

the

unweighted

case

and

the

position

of

the

horizontal

line

minimizes

the

sum

of

the

stored

energy

in

the

springs

-LRB-

Equation

-RRB-

We

will

ignore

a

factor

of

in

all

our

energy

calculations

to

simplify

notation

In

the

weighted

case

the

springs

are

not

equal

and

the

spring

constant

of

each

spring

is

given

by

K

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

The

stored

energy

in

the

springs

in

this

case

is

C

of

Equation

which

is

minimized

by

the

physical

process

Note

that

the

locally

weighted

average

emphasizes

points

close

to

the

query

point

and

produces

an

answer

-LRB-

the

height

of

the

horizontal

line

-RRB-

that

is

closer

to

the

height

of

points

near

the

query

point

than

the

unweighted

case

The

Distance

Weighted

Averaging

Literature

In

statistics

the

approach

of

fitting

constants

using

a

locally

weighted

training

criterion

is

known

as

kernel

regression

and

has

a

vast

literature

-LRB-

Hardle

Wand

and

Jones

-RRB-

Nadaraya

-LRB-

-RRB-

and

Watson

-LRB-

-RRB-

proposed

using

a

weighted

average

of

a

set

of

nearest

neighbors

for

regression

The

approach

was

also

independently

reinvented

in

computer

graphics

-LRB-

Shepard

-RRB-

Specht

-LRB-

-RRB-

describes

a

memory-based

neural

network

approach

based

on

a

probabilistic

model

that

motivates

using

weighted

averaging

as

the

local

model

for

regression

Connell

and

Utgoff

-LRB-

-RRB-

Kibler

et

al.

-LRB-

-RRB-

and

Aha

-LRB-

-RRB-

have

applied

weighted

averaging

to

artificial

intelligence

problems

Locally

Weighted

Regression

In

locally

weighted

regression

-LRB-

LWR

-RRB-

local

models

are

fit

to

nearby

data

As

described

later

in

this

section

this

can

be

derived

by

either

weighting

the

training

criterion

for

the

local

model

-LRB-

in

the

general

case

-RRB-

or

by

directly

weighting

the

data

-LRB-

in

the

case

that

the

local

model

is

linear

in

the

unknown

parameters

-RRB-

LWR

is

derived

from

standard

regression

procedures

for

global

models

We

will

start

our

exploration

of

LWR

by

reviewing

regression

procedures

for

global

models

Nonlinear

Local

Models

Nonlinear

Global

Models

A

general

global

model

can

be

trained

to

minimize

the

following

unweighted

training

criterion

C

X

i

L

-LRB-

f

-LRB-

xi

-RRB-

yi

-RRB-

-LRB-

-RRB-

where

the

yi

are

the

output

values

corresponding

to

the

input

vectors

xi

is

the

parameter

vector

for

the

nonlinear

model

yi

f

-LRB-

xi

-RRB-

and

L

-LRB-

yi

yi

-RRB-

is

a

general

loss

function

for

predicting

yi

when

the

training

data

is

yi

For

example

if

the

model

were

a

neural

net

then

would

be

a

vector

of

the

synaptic

weights

Often

the

least

squares

criterion

is

used

for

the

loss

function

-LRB-

L

-LRB-

yi

yi

-RRB-

-LRB-

yi

yi

-RRB-

-RRB-

leading

to

the

training

criterion

C

X

i

-LRB-

f

-LRB-

xi

-RRB-

yi

-RRB-

-LRB-

-RRB-

Sometimes

no

values

of

the

parameters

of

a

global

model

can

provide

a

good

approximation

of

the

true

function

There

are

two

approaches

to

this

problem

First

we

could

use

a

larger

more

complex

global

model

and

hope

that

it

can

approximate

the

data

sufficiently

The

second

approach

which

we

discuss

here

is

to

fit

the

simple

model

to

local

patches

instead

of

the

whole

region

of

interest

A

Training

Criterion

For

Nonlinear

Local

Models

The

data

set

can

be

tailored

to

the

query

point

by

emphasizing

nearby

points

in

the

regression

We

can

do

this

by

weighting

the

training

criterion

C

-LRB-

q

-RRB-

X

i

-LSB-

L

-LRB-

f

-LRB-

xi

-RRB-

yi

-RRB-

K

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

-RSB-

-LRB-

-RRB-

where

K

-LRB-

-RRB-

is

the

weighting

or

kernel

function

and

d

-LRB-

xi

q

-RRB-

is

the

distance

between

the

data

point

xi

and

the

query

q.

Using

this

training

criterion

f

-LRB-

x

-LRB-

q

-RRB-

-RRB-

now

becomes

a

local

model

and

can

have

a

different

set

of

parameters

-LRB-

q

-RRB-

for

each

query

point

q.

Linear

Local

Models

Given

that

we

are

using

local

models

it

seems

advantageous

to

keep

them

simple

and

to

keep

the

training

criterion

simple

as

well

This

leads

us

to

explore

local

models

that

are

linear

in

the

unknown

parameters

and

to

use

the

least

squares

training

criterion

We

derive

least

squares

training

algorithms

for

linear

local

models

from

regression

procedures

for

linear

global

models

Linear

Global

Models

A

global

model

that

is

linear

in

the

parameters

can

be

expressed

as

-LRB-

Myers

-RRB-

xTi

yi

-LRB-

-RRB-

In

what

follows

we

will

assume

that

the

constant

has

been

appended

to

all

the

input

vectors

xi

to

include

a

constant

term

in

the

regression

The

training

examples

can

be

collected

in

a

matrix

equation

X

y

-LRB-

-RRB-

where

X

is

a

matrix

whose

ith

row

is

xTi

and

y

is

a

vector

whose

ith

element

is

yi

Thus

the

dimensionality

of

X

is

n

d

where

n

is

the

number

of

data

points

and

d

is

the

dimensionality

of

x.

Estimating

the

parameters

using

an

unweighted

regression

minimizes

the

criterion

C

X

i

-LRB-

xTi

yi

-RRB-

-LRB-

-RRB-

by

solving

the

normal

equations

XTX

XTy

-LRB-

-RRB-

for

XTX

XTy

-LRB-

-RRB-

Inverting

the

matrix

XTX

is

not

the

numerically

best

way

to

solve

the

normal

equations

from

the

point

of

view

of

efficiency

or

accuracy

and

usually

other

matrix

techniques

are

used

to

solve

Equation

-LRB-

Press

et

al.

-RRB-

Weighting

the

Criterion

A

Physical

Interpretation

In

fitting

a

line

or

plane

to

a

set

of

points

unweighted

regression

gives

distant

points

equal

influence

with

nearby

points

on

the

ultimate

answer

to

the

query

for

equally

spaced

data

The

linear

local

model

can

be

specialized

to

the

query

by

emphasizing

nearby

points

As

with

the

distance

weighted

average

example

we

can

either

weight

the

error

criterion

that

is

minimized

or

weight

the

data

directly

The

two

approaches

are

equivalent

for

planar

local

models

Weighting

the

criterion

is

done

in

the

following

way

C

-LRB-

q

-RRB-

X

i

h

-LRB-

xTi

yi

-RRB-

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

i

-LRB-

-RRB-

We

again

have

a

physical

interpretation

for

C

-LRB-

q

-RRB-

of

Equation

Much

as

thin

plate

splines

minimize

a

bending

energy

of

a

plate

and

the

energy

of

the

constraints

pulling

on

the

plate

locally

weighted

regression

can

also

be

interpreted

as

a

physical

process

In

LWR

with

a

planar

local

model

the

line

in

Figures

and

can

now

rotate

as

well

as

translate

The

springs

are

forced

to

remain

oriented

vertically

rather

than

move

to

the

smallest

distance

between

the

data

points

and

the

line

Figure

shows

the

fit

-LRB-

the

line

-RRB-

produced

by

equally

strong

springs

to

a

set

of

data

points

-LRB-

the

black

dots

-RRB-

minimizing

the

criterion

of

Equation

Figure

shows

what

happens

to

the

fit

as

the

springs

nearer

to

the

query

point

are

strengthened

and

the

springs

further

away

are

weakened

The

Query

Point

Figure

Unweighted

springs

Query

Point

Figure

Weighted

springs

strengths

of

the

springs

are

given

by

K

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

and

the

fit

minimizes

the

criterion

of

Equation

Direct

Data

Weighting

Our

version

of

directly

weighting

the

data

involves

the

following

steps

For

computational

and

analytical

simplicity

the

origin

of

the

input

data

is

first

shifted

by

subtracting

the

query

point

from

each

data

point

-LRB-

making

the

query

point

q

-LRB-

-RRB-

T

where

the

is

appended

for

the

constant

term

in

the

regression

-RRB-

A

distance

is

calculated

from

each

of

the

stored

data

points

to

the

query

point

q

The

weight

for

each

stored

data

point

is

the

square

root

of

the

kernel

function

used

in

Equation

to

simplify

notation

later

wi

q

K

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

-LRB-

-RRB-

Each

row

i

of

X

and

y

is

multiplied

by

the

corresponding

weight

wi

creating

new

variables

Z

and

v

This

can

be

done

using

matrix

notation

by

creating

a

diagonal

matrix

W

with

diagonal

elements

Wii

wi

and

zeros

elsewhere

and

multiplying

W

times

the

original

variables

zi

wixi

-LRB-

-RRB-

Z

WX

-LRB-

-RRB-

and

vi

wiyi

-LRB-

-RRB-

v

Wy

-LRB-

-RRB-

Equation

is

solved

for

using

the

new

variables

ZTZ

ZTv

-LRB-

-RRB-

Formally

this

gives

us

an

estimator

of

the

form

y

-LRB-

q

-RRB-

qT

-LRB-

ZTZ

-RRB-

-LRB-

-RRB-

The

Relationship

of

Kernel

Regression

and

Locally

Weighted

Regression

For

data

distributed

on

a

regular

grid

away

from

any

boundary

locally

weighted

regression

and

kernel

regression

are

equivalent

-LRB-

Lejeune

Muller

-RRB-

However

for

irregular

data

distributions

there

is

a

significant

difference

and

LWR

has

many

advantages

over

kernel

regression

-LRB-

Hastie

and

Loader

Jones

et

al.

-RRB-

LWR

with

a

planar

local

model

is

often

preferred

over

kernel

smoothing

because

it

exactly

reproduces

a

line

-LRB-

with

any

data

distribution

-RRB-

The

failure

to

reproduce

a

function

indicates

the

bias

of

a

function

approximation

method

LWR

methods

with

a

planar

local

model

will

fail

to

reproduce

a

quadratic

function

reflecting

the

bias

due

to

the

planar

local

model

LWR

methods

with

a

quadratic

local

model

will

fail

to

reproduce

a

cubic

function

and

so

on

The

Locally

Weighted

Regression

Literature

Cleveland

and

Loader

-LRB-

c

-RRB-

Fan

-LRB-

-RRB-

and

Fan

and

Gijbels

-LRB-

-RRB-

review

the

history

of

locally

weighted

regression

and

discuss

current

research

trends

Barnhill

-LRB-

-RRB-

and

Sabin

-LRB-

-RRB-

survey

the

use

of

distance

weighted

nearest

neighbor

interpolators

to

fit

surfaces

to

arbitrarily

spaced

points

and

Eubank

-LRB-

-RRB-

surveys

their

use

in

nonparametric

regression

Lancaster

and

Salkauskas

-LRB-

-RRB-

refer

to

nearest

neighbor

approaches

as

moving

least

squares

and

survey

their

use

in

fitting

surfaces

to

data

Hardle

-LRB-

-RRB-

surveys

kernel

and

LWR

approaches

to

nonparametric

regression

Farmer

and

Sidorowich

-LRB-

b

-RRB-

survey

the

use

of

nearest

neighbor

and

local

model

approaches

in

modeling

chaotic

dynamic

systems

Local

models

-LRB-

often

polynomials

-RRB-

have

been

used

for

over

a

century

to

smooth

regularly

sampled

time

series

and

interpolate

and

extrapolate

from

data

arranged

on

rectangular

grids

Crain

and

Bhattacharyya

-LRB-

-RRB-

Falconer

-LRB-

-RRB-

and

McLain

-LRB-

-RRB-

suggested

using

a

weighted

regression

on

irregularly

spaced

data

to

fit

a

local

polynomial

model

at

each

point

a

function

evaluation

was

desired

All

of

the

available

data

points

were

used

Each

data

point

was

weighted

by

a

function

of

its

distance

to

the

desired

point

in

the

regression

Many

authors

have

suggested

fitting

a

polynomial

surface

only

to

nearby

points

also

using

distance

weighted

regression

-LRB-

McIntyre

et

al.

Pelto

et

al.

Legg

and

Brent

Palmer

Walters

Lodwick

and

Whittle

Stone

Benedetti

Tukey

Franke

and

Nielson

Friedman

-RRB-

Cleveland

-LRB-

-RRB-

proposed

using

robust

regression

procedures

to

eliminate

outlying

or

erroneous

points

in

the

regression

process

Programs

implementing

a

refined

version

of

this

approach

-LRB-

LOCFIT

and

LOESS

-RRB-

are

available

directly

and

also

as

part

of

the

S

package

-LRB-

Cleveland

et

al.

Cleveland

and

Loader

b

c

-RRB-

Katkovnik

-LRB-

-RRB-

also

developed

a

robust

locally

weighted

smoothing

procedure

Cleveland

et

al.

-LRB-

-RRB-

analyzes

the

statistical

properties

of

the

LOESS

algorithm

and

Cleveland

and

Devlin

-LRB-

-RRB-

and

Cleveland

et

al.

-LRB-

-RRB-

show

examples

of

its

use

Stone

-LRB-

-RRB-

Devroye

-LRB-

-RRB-

Lancaster

-LRB-

-RRB-

Lancaster

and

Salkauskas

-LRB-

-RRB-

Cheng

-LRB-

-RRB-

Li

-LRB-

-RRB-

Tsybakov

-LRB-

-RRB-

and

Farwig

-LRB-

-RRB-

provide

analyses

of

LWR

approaches

Stone

-LRB-

-RRB-

shows

that

LWR

has

an

optimal

rate

of

convergence

in

a

minimax

sense

Fan

-LRB-

-RRB-

shows

that

local

linear

regression

smoothers

are

the

best

smoothers

in

that

they

are

the

asymptotic

minimax

linear

smoother

and

have

a

high

asymptotic

efficiency

-LRB-

which

can

be

with

a

suitable

choice

of

kernel

and

bandwidth

-RRB-

among

all

possible

linear

smoothers

including

those

produced

by

kernel

orthogonal

series

and

spline

methods

when

the

unknown

regression

function

is

in

the

class

of

functions

having

bounded

second

derivatives

Fan

-LRB-

-RRB-

extends

this

result

to

show

that

LWR

has

a

high

minimax

efficiency

among

all

possible

estimators

including

nonlinear

smoothers

such

as

median

regression

Fan

-LRB-

-RRB-

Fan

and

Gijbels

-LRB-

-RRB-

Hastie

and

Loader

-LRB-

-RRB-

and

Jones

et

al.

-LRB-

-RRB-

show

that

LWR

handles

a

wide

range

of

data

distributions

and

avoids

boundary

and

cluster

effects

Ruppert

and

Wand

-LRB-

-RRB-

derive

asymptotic

bias

and

variance

formulas

for

multivariate

LWR

while

Cleveland

and

Loader

-LRB-

-RRB-

argue

that

asymptotic

results

have

limited

practical

relevance

Fan

and

Gijbels

-LRB-

-RRB-

explore

the

use

of

a

variable

bandwidth

locally

weighted

regression

Vapnik

and

Bottou

-LRB-

-RRB-

give

error

bounds

for

local

learning

algorithms

Locally

weighted

regression

was

introduced

into

the

domain

of

machine

learning

and

robot

learning

by

Atkeson

-LRB-

Atkeson

and

Reinkensmeyer

Atkeson

-RRB-

who

also

explored

techniques

for

detecting

irrelevant

features

and

Zografski

-LRB-

Zografski

Zografski

and

Durrani

-RRB-

Atkeson

and

Schaal

-LRB-

-RRB-

explore

locally

weighted

learning

from

the

point

of

view

of

neural

networks

Dietterich

et

al.

-LRB-

-RRB-

report

on

a

recent

workshop

on

memory-based

learning

including

locally

weighted

learning

Distance

Functions

Locally

weighted

learning

is

critically

dependent

on

the

distance

function

There

are

many

different

approaches

to

defining

a

distance

function

and

this

section

briefly

surveys

them

Distance

functions

in

locally

weighted

learning

do

not

need

to

satisfy

the

formal

mathematical

requirements

for

a

distance

metric

The

relative

importance

of

the

input

dimensions

in

generating

the

distance

measurement

depends

on

how

the

inputs

are

scaled

-LRB-

i.e.

how

much

they

are

stretched

or

squashed

-RRB-

We

use

the

term

scaling

for

this

purpose

having

reserved

the

term

weight

for

the

contribution

of

individual

points

-LRB-

not

dimensions

-RRB-

in

a

regression

We

refer

to

the

scaling

factors

as

mj

in

this

paper

There

are

many

ways

to

define

and

use

distance

functions

-LRB-

Scott

-RRB-

ffl

Global

distance

functions

The

same

distance

function

is

used

at

all

parts

of

the

input

space

ffl

Query-based

local

distance

functions

The

form

of

d

-LRB-

-RRB-

or

the

distance

function

parameters

are

set

on

each

query

by

an

optimization

process

that

typically

minimizes

cross

validation

error

or

a

related

criterion

This

approach

is

referred

to

as

a

uniform

metric

by

Stanfill

-LRB-

-RRB-

and

is

discussed

in

Stanfill

and

Waltz

-LRB-

-RRB-

Hastie

and

Tibshirani

-LRB-

-RRB-

and

Friedman

-LRB-

-RRB-

ffl

Point-based

local

distance

functions

Each

stored

data

point

has

associated

with

it

a

distance

function

and

the

values

of

corresponding

parameters

The

training

criterion

uses

a

different

di

-LRB-

-RRB-

for

each

point

xi

C

-LRB-

q

-RRB-

X

i

h

-LRB-

f

-LRB-

xi

-RRB-

yi

-RRB-

-LRB-

di

-LRB-

xi

q

-RRB-

-RRB-

i

-LRB-

-RRB-

The

di

-LRB-

-RRB-

can

be

selected

either

by

a

direct

computation

or

by

minimizing

cross

validation

error

Frequently

the

di

-LRB-

-RRB-

are

chosen

in

advance

of

the

queries

and

are

stored

with

the

data

points

This

approach

is

referred

to

as

a

variable

metric

by

Stanfill

-LRB-

-RRB-

For

classifiers

one

version

of

a

point-based

local

distance

function

is

to

have

a

different

distance

function

for

each

class

-LRB-

Waltz

Aha

and

McNulty

Aha

-RRB-

Aha

and

Goldstone

-LRB-

-RRB-

explore

the

use

of

pointbased

distance

functions

by

human

subjects

Distance

functions

can

be

asymmetric

and

nonlinear

so

that

a

distance

along

a

particular

dimension

can

depend

on

whether

the

query

point

's

value

for

the

dimension

is

larger

or

smaller

than

the

stored

point

's

value

for

that

dimension

-LRB-

Medin

and

Shoben

-RRB-

The

distance

along

a

dimension

can

also

depend

on

the

values

being

compared

-LRB-

Nosofsky

et

al.

-RRB-

Feature

Scaling

Altering

the

distance

function

can

serve

two

purposes

If

the

feature

scaling

factors

mj

are

all

nonzero

the

input

space

is

warped

or

distorted

which

might

lead

to

more

accurate

predictions

If

some

of

the

scaling

factors

are

set

to

zero

those

dimensions

are

ignored

by

the

distance

function

and

the

local

model

becomes

global

in

those

directions

Zeroing

feature

scaling

factors

can

be

used

as

a

tool

to

combat

the

curse

of

dimensionality

by

reducing

the

locality

of

the

function

approximation

process

in

this

way

Note

that

a

feature

scaling

factor

of

zero

does

not

mean

the

local

model

ignores

that

feature

in

locally

weighted

learning

Instead

all

points

aligned

along

that

direction

get

the

same

weight

and

the

feature

can

affect

the

output

of

the

local

model

For

example

fitting

a

global

model

using

all

features

is

equivalent

to

setting

all

feature

scaling

factors

to

zero

and

fitting

the

same

model

as

a

local

model

Local

model

feature

selection

is

a

separate

process

from

distance

function

feature

scaling

Ignoring

features

using

ridge

regression

dimensionality

reduction

of

the

entire

modeling

process

and

algorithms

for

feature

scaling

and

selection

are

discussed

in

later

sections

Stanfill

and

Waltz

-LRB-

-RRB-

describe

a

variant

of

feature

selection

-LRB-

predictor

restriction

-RRB-

in

which

the

scaling

factor

for

a

feature

becomes

so

large

that

any

difference

from

the

query

in

that

dimension

causes

the

training

point

to

be

ignored

They

also

describe

using

an

initial

prediction

of

the

output

in

an

augmented

distance

function

to

select

training

data

with

similar

or

equal

outputs

-LRB-

goal

restriction

-RRB-

-LRB-

Jabbour

et

al.

-RRB-

Distance

Functions

For

Continuous

Inputs

The

functions

discussed

in

this

section

are

especially

appropriate

for

ordered

-LRB-

vs.

categorical

symbolic

or

nominal

-RRB-

input

values

which

are

either

continuous

or

an

ordered

set

of

discrete

values

ffl

Unweighted

Euclidean

distance

dE

-LRB-

x

q

-RRB-

sX

j

-LRB-

xj

qj

-RRB-

q

-LRB-

x

q

-RRB-

T

-LRB-

x

q

-RRB-

-LRB-

-RRB-

ffl

Diagonally

weighted

Euclidean

distance

dm

-LRB-

x

q

-RRB-

sX

j

-LRB-

mj

-LRB-

xj

qj

-RRB-

-RRB-

q

-LRB-

x

q

-RRB-

TMTM

-LRB-

x

q

-RRB-

dE

-LRB-

Mx

Mq

-RRB-

-LRB-

-RRB-

where

M

is

diagonal

and

Mjj

mj

ffl

Fully

weighted

Euclidean

distance

dM

-LRB-

x

q

-RRB-

q

-LRB-

x

q

-RRB-

TMTM

-LRB-

x

q

-RRB-

dE

-LRB-

Mx

Mq

-RRB-

-LRB-

-RRB-

where

M

is

no

longer

diagonal

but

can

be

arbitrary

This

is

also

known

as

the

Mahalanobis

distance

-LRB-

Tou

and

Gonzalez

Weisberg

-RRB-

Weighting

function

-LRB-

gaussian

-RRB-

Figure

Contours

of

constant

distance

from

the

center

with

a

diagonal

M

matrix

Weighting

function

-LRB-

gaussian

-RRB-

Figure

Contours

of

a

constant

distance

from

the

center

in

which

the

M

matrix

has

off-diagonal

elements

ffl

Unweighted

Lp

norm

-LRB-

Minkowski

metric

-RRB-

dp

-LRB-

x

q

-RRB-

X

i

jxi

qijp

-LRB-

-RRB-

ffl

Diagonally

weighted

and

fully

weighted

Lp

norm

The

weighted

Lp

norm

is

dp

-LRB-

Mx

Mq

-RRB-

A

diagonal

distance

function

matrix

M

-LRB-

coefficient

for

each

dimension

-RRB-

can

make

a

radially

symmetric

scaling

function

into

an

axis

parallel

ellipse

-LRB-

Figure

shows

ellipses

with

the

axes

of

symmetry

aligned

with

the

coordinate

axes

-RRB-

Figure

shows

an

example

of

how

a

full

distance

function

matrix

M

with

cross

terms

can

arbitrarily

orient

the

ellipse

-LRB-

Ruppert

and

Wand

Wand

and

Jones

-RRB-

Cleveland

and

Grosse

-LRB-

-RRB-

Cleveland

et

al.

-LRB-

-RRB-

and

Cleveland

-LRB-

-RRB-

point

out

that

for

distance

functions

with

zero

coefficients

-LRB-

mi

an

entire

column

of

M

is

zero

or

M

is

singular

-RRB-

the

model

is

global

in

the

corresponding

directions

They

refer

to

this

as

a

conditionally

parametric

model

Fukunaga

-LRB-

-RRB-

James

-LRB-

-RRB-

and

Tou

and

Gonzalez

-LRB-

-RRB-

describe

how

to

choose

a

distance

function

matrix

to

maximize

the

ratio

of

the

variance

between

classes

to

the

variance

of

all

the

cases

in

classification

Mohri

and

Tanaka

-LRB-

-RRB-

extend

this

approach

to

symbolic

input

values

This

approach

uses

an

eigenvalue/eigenvector

decomposition

and

can

help

distinguish

relevant

attributes

from

irrelevant

attributes

and

filter

out

noisy

data

This

approach

is

localized

by

Hastie

and

Tibshirani

-LRB-

-RRB-

Distance

functions

for

symbolic

inputs

have

been

developed

and

are

discussed

in

Atkeson

-LRB-

-RRB-

Smoothing

Parameters

A

smoothing

or

bandwidth

parameter

h

defines

the

scale

or

range

over

which

generalization

is

performed

There

are

several

ways

to

use

this

parameter

-LRB-

Scott

Cleveland

and

Loader

-RRB-

ffl

Fixed

bandwidth

selection

h

is

a

constant

value

-LRB-

Fan

and

Marron

-RRB-

and

therefore

constant

size

and

shape

volumes

of

data

are

used

In

this

case

h

can

appear

implicitly

in

the

distance

function

as

the

determinant

of

M

for

fully

weighted

distance

functions

-LRB-

h

jMj

-RRB-

or

the

magnitude

of

the

vector

m

in

diagonally

weighted

distance

functions

-LRB-

h

jmj

-RRB-

and/or

explicitly

in

the

weighting

function

K

d

-LRB-

xi

q

-RRB-

h

-LRB-

-RRB-

These

parameters

although

redundant

in

the

explicit

case

provide

a

convenient

way

to

adjust

the

radius

of

the

weighting

function

The

redundancy

can

be

eliminated

by

requiring

the

determinant

of

the

scaling

factor

matrix

to

be

one

-LRB-

jMj

-RRB-

or

fixing

some

element

of

M.

ffl

Nearest

neighbor

bandwidth

selection

h

is

set

to

be

the

distance

to

the

kth

nearest

data

point

-LRB-

Stone

Cleveland

Farmer

and

Sidorowich

b

Townshend

Hastie

and

Loader

Fan

and

Gijbels

Ge

et

al.

Naes

et

al.

Naes

and

Isaksson

Wang

et

al.

Cleveland

and

Loader

-RRB-

The

data

volume

increases

and

decreases

in

size

according

to

the

density

of

nearby

data

In

this

case

changes

in

scale

of

the

distance

function

are

canceled

by

corresponding

changes

in

h

giving

a

scale

invariant

weighting

pattern

to

the

data

However

h

will

not

cancel

changes

in

distance

function

coefficients

that

alter

the

shape

of

the

weighting

function

and

the

identity

of

the

kth

neighbor

can

change

with

distance

function

shape

changes

ffl

Global

bandwidth

selection

h

is

set

globally

by

an

optimization

process

that

typically

minimizes

cross

validation

error

over

all

the

data

ffl

Query-based

local

bandwidth

selection

h

is

set

on

each

query

by

an

optimization

process

that

typically

minimizes

cross

validation

error

or

a

related

criterion

-LRB-

Vapnik

-RRB-

ffl

Point-based

local

bandwidth

selection

Each

stored

data

point

has

associated

with

it

a

bandwidth

h

The

weighted

criterion

uses

a

different

hi

for

each

point

xi

C

-LRB-

q

-RRB-

X

i

-LRB-

f

-LRB-

xi

-RRB-

yi

-RRB-

d

-LRB-

xi

q

-RRB-

hi

-LRB-

-RRB-

The

hi

can

be

set

either

by

a

direct

computation

or

by

an

optimization

process

that

typically

minimizes

cross

validation

error

or

a

related

criterion

Typically

the

hi

are

computed

in

advance

of

the

queries

and

are

stored

with

the

data

points

Fan

and

Marron

-LRB-

-RRB-

argue

that

a

fixed

bandwidth

is

easy

to

interpret

but

of

limited

use

Cleveland

and

Loader

-LRB-

-RRB-

argue

in

favor

of

nearest

neighbor

smoothing

over

fixed

bandwidth

smoothing

A

fixed

bandwidth

and

a

weighting

function

that

goes

to

zero

at

a

finite

distance

can

have

large

variance

in

regions

of

low

data

density

This

problem

is

present

at

edges

or

between

data

clusters

and

gets

worse

in

higher

dimensions

In

general

fixed

bandwidth

selection

has

much

larger

changes

in

variance

than

nearest

neighbor

bandwidth

selection

A

fixed

bandwidth

smoother

can

also

not

have

any

data

within

its

span

leading

to

undefined

estimates

-LRB-

Cleveland

and

Loader

-RRB-

Fan

and

Marron

-LRB-

-RRB-

describe

three

reasons

to

use

variable

local

bandwidths

to

adapt

to

the

data

distribution

to

adapt

for

different

levels

of

noise

-LRB-

heteroscedasticity

-RRB-

and

to

adapt

to

changes

in

the

smoothness

or

curvature

of

the

function

Fan

and

Gijbels

-LRB-

-RRB-

argue

for

point-based

in

favor

of

query-based

local

bandwidth

selection

explaining

that

having

a

bandwidth

associated

with

each

data

point

will

allow

rapid

or

asymmetric

changes

in

the

behavior

of

the

data

to

be

accommodated

Section

discusses

global

and

local

tuning

of

bandwidths

Weighting

Functions

The

requirements

on

a

weighting

function

-LRB-

also

known

as

a

kernel

function

-RRB-

are

straightforward

-LRB-

Gasser

and

Muller

Cleveland

and

Loader

Fedorov

et

al.

-RRB-

The

maximum

value

of

the

weighting

function

should

be

at

zero

distance

and

the

function

should

decay

smoothly

as

the

distance

increases

Discontinuities

in

weighting

functions

lead

to

discontinuities

in

the

predictions

since

training

points

cross

the

discontinuity

as

the

query

changes

In

general

the

smoother

the

weight

function

the

smoother

the

estimated

function

Weights

that

go

to

infinity

when

a

query

equals

a

stored

data

point

allow

exact

interpolation

of

the

stored

data

Finite

weights

lead

to

smoothing

of

the

data

Weight

functions

that

go

to

zero

at

a

finite

distance

allow

faster

implementations

since

points

further

from

the

query

than

that

distance

can

be

ignored

with

no

error

As

mentioned

previously

kernels

with

a

fixed

finite

radius

raise

the

possibility

of

not

having

enough

or

any

points

within

the

non-zero

area

a

possibility

that

must

be

handled

by

the

locally

weighted

learning

system

It

is

not

necessary

to

normalize

the

kernel

function

and

the

kernel

function

does

not

need

to

be

unimodal

The

kernel

function

should

always

be

non-negative

since

a

negative

value

would

lead

to

the

training

process

increasing

training

error

in

order

to

decrease

the

training

criterion

The

weights

-LRB-

i.e.

the

square

root

of

the

kernel

function

-RRB-

can

be

positive

or

negative

We

have

only

used

non-negative

weights

Some

of

the

kernel

functions

discussed

in

this

section

are

shown

in

Figure

A

simple

weighting

function

just

raises

the

distance

to

a

negative

power

-LRB-

Shepard

Atkeson

Ruprecht

et

al.

Ruprecht

and

Muller

-RRB-

The

magnitude

of

the

power

determines

how

local

the

regression

will

be

-LRB-

i.e.

the

rate

of

dropoff

of

the

weights

with

distance

-RRB-

K

-LRB-

d

-RRB-

dp

-LRB-

-RRB-

This

type

of

weighting

function

goes

to

infinity

as

the

query

point

approaches

a

stored

data

point

and

forces

the

locally

weighted

regression

to

exactly

match

that

stored

point

If

the

data

is

noisy

exact

interpolation

is

not

desirable

and

a

weighting

scheme

with

limited

magnitude

is

desired

The

inverse

distance

-LRB-

Wolberg

-RRB-

K

-LRB-

d

-RRB-

dp

-LRB-

-RRB-

d

d

exp

-LRB-

d

d

-RRB-

d

d

d

d

exp

-LRB-

d

-RRB-

d

-LRB-

d

-RRB-

d

-LRB-

d

-RRB-

d

Uniform

d

-LRB-

d

-RRB-

Figure

Some

of

the

kernel

shapes

described

in

the

text

can

be

used

to

approximate

functions

like

Equation

and

the

quadratic

hyperbola

kernel

-LRB-

-RRB-

with

a

well

defined

value

at

d

Another

smoothing

weight

function

is

a

Gaussian

kernel

-LRB-

Deheuvels

Wand

and

Schucany

Schaal

and

Atkeson

-RRB-

K

-LRB-

d

-RRB-

exp

-LRB-

-RRB-

This

kernel

also

has

infinite

extent

A

related

kernel

is

the

exponential

kernel

which

has

been

used

in

psychological

models

-LRB-

Aha

and

Goldstone

-RRB-

K

-LRB-

d

-RRB-

exp

-LSB-

jdj

-RSB-

-LRB-

-RRB-

These

kernels

have

infinite

extent

and

can

be

truncated

when

they

become

smaller

than

a

threshold

value

to

ignore

data

further

from

a

particular

radius

from

the

query

The

quadratic

kernel

also

known

as

the

Epanechnikov

kernel

and

the

Bartlett-Priestley

kernel

is

-LRB-

Epanechnikov

Lejeune

Altman

Hastie

and

Loader

Fan

and

Gijbels

a

Fan

and

Hall

-RRB-

K

-LRB-

d

-RRB-

-LRB-

-RRB-

if

jdj

otherwise

-LRB-

-RRB-

This

kernel

has

finite

extent

and

ignores

data

further

than

a

radius

of

from

the

query

when

building

the

local

model

Fan

and

Marron

-LRB-

-RRB-

and

Muller

-LRB-

-RRB-

argue

that

this

kernel

function

is

optimal

in

a

mean

squared

error

sense

However

there

is

a

discontinuity

in

the

derivative

at

d

which

makes

this

kernel

less

attractive

in

real

applications

and

analytical

treatments

The

tricube

kernel

is

used

by

Cleveland

-LRB-

-RRB-

Cleveland

and

Devlin

-LRB-

-RRB-

Diebold

and

Nason

-LRB-

-RRB-

LeBaron

-LRB-

-RRB-

Naes

et

al.

-LRB-

-RRB-

Naes

and

Isaksson

-LRB-

-RRB-

Wang

et

al.

-LRB-

-RRB-

and

Ge

et

al.

-LRB-

-RRB-

K

-LRB-

d

-RRB-

if

jdj

otherwise

-LRB-

-RRB-

This

kernel

also

has

finite

extent

and

a

continuous

first

and

second

derivative

which

means

the

first

and

second

derivative

of

the

prediction

will

also

be

continuous

For

comparison

the

uniform

weighting

kernel

-LRB-

or

boxcar

weighting

kernel

-RRB-

is

used

by

Stone

-LRB-

-RRB-

Friedman

-LRB-

-RRB-

Tsybakov

-LRB-

-RRB-

and

Muller

-LRB-

-RRB-

K

-LRB-

d

-RRB-

if

jdj

otherwise

-LRB-

-RRB-

and

the

triangular

kernel

-LRB-

used

in

locally

weighted

median

regression

-RRB-

is

K

-LRB-

d

-RRB-

jdj

if

jdj

otherwise

-LRB-

-RRB-

A

variant

of

the

triangular

kernel

is

the

following

-LRB-

Franke

and

Nielson

Ruprecht

and

Muller

Ruprecht

et

al.

-RRB-

K

-LRB-

d

-RRB-

jdj

jdj

if

jdj

otherwise

-LRB-

-RRB-

In

general

new

kernel

functions

can

be

created

by

raising

these

kernel

functions

to

a

power

For

example

the

biquadratic

kernel

is

the

square

of

the

quadratic

kernel

The

power

can

be

non-integral

and

also

less

than

one

The

triangular

biquadratic

and

tricube

kernels

form

a

family

Ruprecht

and

Muller

-LRB-

-RRB-

generalize

the

distance

function

to

a

point-set

metric

In

our

view

there

is

no

clear

evidence

that

the

choice

of

weighting

function

is

critical

-LRB-

Scott

Cleveland

and

Loader

c

-RRB-

however

there

are

examples

where

one

can

show

differences

-LRB-

Fedorov

et

al.

-RRB-

Cleveland

and

Loader

-LRB-

-RRB-

criticize

the

uniform

kernel

for

similar

reasons

as

are

used

in

signal

processing

and

spectrum

estimation

Optimal

kernels

are

discussed

by

Gasser

and

Muller

-LRB-

-RRB-

Gasser

et

al.

-LRB-

-RRB-

Scott

-LRB-

-RRB-

Blyth

-LRB-

-RRB-

and

Fedorov

et

al.

-LRB-

-RRB-

Finite

extent

of

the

kernel

function

is

useful

but

other

than

that

the

literature

and

our

own

work

have

not

noted

any

substantial

empirical

difference

in

most

cases

Local

Model

Structures

So

far

we

have

discussed

only

a

few

kinds

of

local

models

constant

and

linear

local

models

There

are

no

limits

on

what

model

structure

can

be

used

as

a

local

model

Models

that

are

linear

in

the

unknown

parameters

such

as

local

polynomials

train

faster

than

more

general

models

Since

a

major

component

of

the

lookup

cost

is

the

training

cost

this

is

an

important

benefit

Cleveland

and

Devlin

-LRB-

-RRB-

Atkeson

-LRB-

-RRB-

Farmer

and

Sidorowich

-LRB-

b

-RRB-

Cleveland

and

Loader

-LRB-

-RRB-

Naes

et

al.

-LRB-

-RRB-

Naes

and

Isaksson

-LRB-

-RRB-

and

Wang

et

al.

-LRB-

-RRB-

have

applied

local

quadratic

and

cubic

models

which

are

analyzed

by

Ruppert

and

Wand

-LRB-

-RRB-

Higher

order

polynomials

reduce

the

bias

but

increase

the

variance

of

the

estimates

Locally

constant

models

handle

flat

regions

well

while

quadratics

and

cubics

handle

areas

of

high

curvature

such

as

peaks

and

valleys

well

Cleveland

and

Loader

-LRB-

c

-RRB-

present

an

approach

to

blending

polynomial

models

where

a

non-integral

model

order

indicates

a

weighted

blend

between

two

integral

model

orders

They

use

cross

validation

to

optimize

the

local

model

order

on

each

query

Regularization

Insufficient

Data

and

Prediction

Bias

To

uniquely

interpolate

between

and

extrapolate

from

the

training

data

we

must

express

a

preference

or

learning

bias

In

function

approximation

that

preference

is

typically

expressed

as

a

smoothness

criterion

to

optimize

In

the

case

of

locally

weighted

learning

the

smoothness

constraint

is

not

explicit

However

there

are

several

fit

parameters

that

affect

the

smoothness

of

the

predicted

outputs

The

smoothing

bandwidth

is

an

important

control

knob

as

is

a

ridge

regression

parameter

to

be

described

in

the

next

section

The

order

of

the

local

model

also

can

serve

as

a

smoothing

parameter

The

shape

of

the

distance

and

weighting

functions

play

a

secondary

role

in

smoothing

the

estimates

although

in

general

the

number

of

derivatives

with

respect

to

x

of

K

-LRB-

d

-LRB-

x

q

-RRB-

-RRB-

that

exist

determine

the

order

of

smoothness

of

the

predicted

outputs

There

is

an

important

link

between

smoothness

control

and

overfitting

Seifert

and

Gasser

-LRB-

-RRB-

explore

a

variety

of

approaches

to

handling

insufficient

data

in

local

regression

Ridge

Regression

A

potential

problem

is

that

the

data

points

can

be

distributed

in

such

a

way

as

to

make

the

regression

matrix

ZTZ

in

Equation

nearly

singular

If

there

are

not

enough

nearby

points

with

non-zero

weights

in

all

directions

there

are

not

enough

different

equations

to

solve

for

the

unknown

parameters

Ridge

regression

-LRB-

Draper

and

Smith

-RRB-

is

used

to

prevent

problems

due

to

a

singular

data

matrix

The

following

equation

instead

of

Equation

is

solved

for

ZTZ

ZTv

-LRB-

-RRB-

where

is

a

diagonal

matrix

with

small

positive

diagonal

elements

BBBBBBBB

CCCCCCCCA

-LRB-

-RRB-

and

is

an

a

priori

estimate

or

expectation

of

what

the

local

model

parameters

will

be

-LRB-

often

is

taken

to

be

a

vector

of

all

zeros

-RRB-

This

is

equivalent

to

adding

n

extra

rows

to

Z

each

having

a

single

non-zero

element

i

in

the

ith

column

The

equation

Z

v

becomes

Z

n

v

n

n

-LRB-

-RRB-

Adding

additional

rows

can

be

viewed

as

adding

fake

data

which

in

the

absence

of

sufficient

real

data

biases

the

parameter

estimates

to

-LRB-

Draper

and

Smith

-RRB-

Another

view

of

ridge

regression

parameters

is

that

they

are

the

Bayesian

assumptions

about

the

apriori

distributions

of

the

estimated

parameters

-LRB-

Seber

-RRB-

As

described

in

Section

on

tuning

optimizing

the

ridge

regression

parameters

using

cross

validation

can

identify

irrelevant

dimensions

These

techniques

also

help

combat

overfitting

Dimensionality

Reduction

Principal

components

analysis

-LRB-

PCA

-RRB-

can

also

be

used

globally

to

eliminate

directions

in

which

there

is

no

data

-LRB-

Wettschereck

-RRB-

However

it

is

rarely

the

case

that

there

is

absolutely

no

data

in

a

particular

direction

A

closely

related

technique

the

singular

value

decomposition

-LRB-

SVD

-RRB-

is

typically

used

in

locally

weighted

regression

to

perform

dimensionality

reduction

Cleveland

and

Grosse

-LRB-

-RRB-

compute

the

inverse

of

ZTZ

using

the

singular

value

decomposition

and

then

set

small

singular

values

to

zero

in

the

calculated

inverse

This

corresponds

to

eliminating

those

directions

from

the

local

model

Principal

components

analysis

can

also

be

done

locally

on

the

weighted

data

either

around

each

stored

data

point

or

in

response

to

a

query

Directions

can

be

eliminated

in

either

a

hard

Figure

input

points

scattered

on

a

nonlinear

manifold

fashion

explicitly

setting

the

corresponding

parameters

to

zero

or

in

a

soft

fashion

-LRB-

such

as

performing

ridge

regression

in

the

coordinate

system

defined

by

the

PCA

or

SVD

-RRB-

In

Bregler

and

Omohundro

-LRB-

-RRB-

an

interesting

locally

weighted

learning

approach

is

presented

for

identifying

low-dimensional

submanifolds

on

which

data

is

lying

In

Figure

the

space

has

two

dimensions

and

yet

each

dot

is

locally

embedded

on

a

one

dimensional

curve

Bregler

and

Omohundro

's

method

uses

locally

weighted

principal

component

analysis

-LRB-

which

performs

a

singular

value

decomposition

of

the

Z

matrix

from

Equation

-RRB-

to

identify

local

manifolds

This

is

a

useful

analysis

tool

for

identifying

local

dependencies

between

variables

in

a

dataset

But

it

also

has

important

consequences

for

developing

a

local

distance

function

the

principal

component

matrix

reveals

the

directions

in

input

space

for

which

there

is

no

data

support

These

approaches

only

consider

the

input

space

-LRB-

the

space

spanned

by

xi

-RRB-

It

is

often

important

to

also

consider

the

outputs

-LRB-

the

yi

-RRB-

when

performing

distance

function

or

smoothing

parameter

optimization

The

outputs

can

provide

more

opportunities

for

dimensionality

reduction

if

they

are

flat

in

some

direction

or

can

be

predicted

by

a

local

model

An

alternative

perspective

is

to

consider

the

conditional

probability

p

-LRB-

yjx

-RRB-

Perhaps

one

could

do

local

principal

components

analysis

in

the

joint

density

space

p

-LRB-

x

y

-RRB-

and

eliminate

the

input

directions

that

contribute

least

to

predicting

the

outputs

A

potential

problem

with

dimensionality

reduction

in

general

is

that

the

new

dimensions

if

not

aligned

with

the

previous

dimensions

are

not

necessarily

meaningful

Our

focus

is

on

reducing

prediction

error

ignoring

comprehensibility

of

the

local

models

Assessing

the

Predictions

An

important

aspect

of

locally

weighted

learning

is

that

it

is

possible

to

estimate

the

prediction

error

and

derive

confidence

bounds

on

the

predictions

Bottou

and

Vapnik

-LRB-

Vapnik

and

Bottou

-RRB-

analyze

confidence

intervals

for

locally

weighted

classifiers

We

start

our

analysis

of

locally

weighted

regression

by

pointing

out

that

LWR

is

an

estimator

that

is

linear

in

the

output

data

y

-LRB-

using

Equations

and

-RRB-

y

-LRB-

q

-RRB-

qT

-LRB-

ZTZ

-RRB-

sTq

y

nX

i

si

-LRB-

q

-RRB-

yi

-LRB-

-RRB-

The

vector

sq

also

written

as

s

-LRB-

q

-RRB-

will

be

useful

for

calculating

the

bias

and

variance

of

locally

weighted

learning

Estimating

the

Variance

To

calculate

the

variance

of

a

prediction

we

assume

the

training

data

came

from

a

sampling

process

that

measures

output

values

with

additive

random

noise

yi

f

-LRB-

xi

-RRB-

ffli

-LRB-

-RRB-

where

the

ffli

are

independent

have

zero

mean

and

have

variance

-LRB-

xi

-RRB-

Under

the

assumption

that

-LRB-

xi

-RRB-

-LRB-

oe

is

a

constant

-RRB-

and

that

the

linear

model

correctly

models

the

structure

of

the

data

linear

regression

generates

an

unbiased

estimate

of

the

regression

parameters

Additionally

if

the

error

is

normally

distributed

ffli

N

-LRB-

-RRB-

the

regression

estimate

becomes

the

best

linear

unbiased

estimate

in

the

maximum

likelihood

sense

However

unless

stated

explicitly

in

this

paper

we

will

avoid

any

distributional

assumption

on

the

form

of

the

noise

Given

this

model

of

the

additive

noise

-LRB-

and

dropping

the

assumption

that

a

linear

model

correctly

models

the

structure

of

the

data

-RRB-

the

expectation

and

variance

of

the

estimate

y

is

-LRB-

s

is

from

Equation

-RRB-

E

-LRB-

y

-LRB-

q

-RRB-

-RRB-

E

-LRB-

sTq

y

-RRB-

sTq

E

-LRB-

y

-RRB-

X

i

si

-LRB-

q

-RRB-

f

-LRB-

xi

-RRB-

-LRB-

-RRB-

Var

-LRB-

y

-LRB-

q

-RRB-

-RRB-

E

-LSB-

y

-LRB-

q

-RRB-

E

-LRB-

y

-LRB-

q

-RRB-

-RRB-

-RSB-

X

i

-LRB-

q

-RRB-

-LRB-

xi

-RRB-

-LRB-

-RRB-

One

way

to

derive

confidence

intervals

for

the

predictions

from

locally

weighted

learning

is

to

assume

a

locally

constant

variance

-LRB-

q

-RRB-

at

the

prediction

point

q

and

to

use

Equation

This

equation

has

to

be

modified

to

reflect

both

the

additive

noise

in

sampling

at

the

new

point

-LRB-

-LRB-

q

-RRB-

-RRB-

and

the

prediction

error

of

the

estimator

-LRB-

-LRB-

q

-RRB-

sTq

sq

-RRB-

Var

-LRB-

ynew

-LRB-

q

-RRB-

-RRB-

-LRB-

q

-RRB-

-LRB-

q

-RRB-

sTq

sq

-LRB-

-RRB-

This

expression

of

the

prediction

intervals

is

independent

of

the

output

values

of

the

training

data

yi

and

reflects

how

well

the

data

is

distributed

in

the

input

space

However

the

variance

only

reflects

the

difference

between

the

prediction

and

the

mean

prediction

and

not

the

difference

between

the

prediction

and

the

true

value

which

requires

knowledge

of

the

predictor

's

bias

Only

when

the

local

model

structure

is

correct

will

the

bias

be

zero

To

conveniently

derive

an

estimate

of

-LRB-

x

-RRB-

we

will

define

some

additional

quantities

in

terms

of

the

weighted

variables

A

locally

weighted

linear

regression

centered

at

a

point

q

produces

local

model

parameters

-LRB-

q

-RRB-

It

also

produces

errors

-LRB-

residuals

-RRB-

at

all

training

points

The

weighted

residual

ri

-LRB-

q

-RRB-

is

given

by

-LRB-

vi

is

defined

in

Equation

-RRB-

ri

-LRB-

q

-RRB-

zTi

-LRB-

q

-RRB-

-LRB-

q

-RRB-

vi

-LRB-

q

-RRB-

-LRB-

-RRB-

The

training

criteria

which

is

the

weighted

sum

of

the

squared

errors

is

given

by

C

-LRB-

q

-RRB-

X

i

-LRB-

q

-RRB-

-LRB-

-RRB-

A

reasonable

estimator

for

the

local

value

of

the

noise

variance

is

-LRB-

q

-RRB-

P

-LRB-

q

-RRB-

nLWR

-LRB-

q

-RRB-

C

-LRB-

q

-RRB-

nLWR

-LRB-

q

-RRB-

-LRB-

-RRB-

where

nLWR

is

a

modified

measure

of

how

many

data

points

there

are

nLWR

-LRB-

q

-RRB-

nX

i

nX

i

K

d

-LRB-

xi

q

-RRB-

h

-LRB-

-RRB-

In

analogy

to

unweighted

regression

-LRB-

Myers

-RRB-

we

can

reduce

the

bias

of

the

estimate

-LRB-

q

-RRB-

by

taking

into

account

the

number

of

parameters

in

the

locally

weighted

regression

-LRB-

q

-RRB-

P

-LRB-

q

-RRB-

nLWR

-LRB-

q

-RRB-

pLWR

-LRB-

q

-RRB-

-LRB-

-RRB-

where

pLWR

is

a

measure

of

the

local

number

of

free

parameters

in

the

local

model

pLWR

-LRB-

q

-RRB-

X

i

zTi

-LRB-

ZTZ

-RRB-

-LRB-

-RRB-

We

have

described

a

variance

estimator

that

uses

only

local

information

An

alternative

way

to

obtain

a

variance

estimate

uses

global

information

i.e.

information

from

more

than

one

LWR

fit

and

assumes

a

single

global

value

for

the

additive

noise

-LRB-

Cleveland

et

al.

Cleveland

and

Grosse

Cleveland

et

al.

-RRB-

Estimating

the

Bias

Assessing

the

bias

requires

making

assumptions

about

the

underlying

form

of

the

true

function

and

the

data

distribution

In

the

case

of

locally

weighted

learning

this

is

a

weak

assumption

since

we

need

to

know

only

the

local

behavior

of

the

function

and

the

local

distribution

of

the

data

Let

us

assume

that

the

real

function

f

is

described

locally

by

a

quadratic

model

f

-LRB-

x

-RRB-

f

-LRB-

q

-RRB-

gT

-LRB-

x

q

-RRB-

-LRB-

x

q

-RRB-

TH

-LRB-

x

q

-RRB-

-LRB-

-RRB-

where

q

is

the

query

point

g

is

the

true

gradient

at

the

query

point

and

H

is

the

true

Hessian

-LRB-

matrix

of

second

derivatives

-RRB-

at

the

query

point

The

expected

value

of

the

estimate

is

given

by

Equation

which

can

be

used

to

find

the

bias

bias

E

-LRB-

y

-LRB-

q

-RRB-

-RRB-

ytrue

-LRB-

q

-RRB-

X

-LSB-

si

-LRB-

q

-RRB-

f

-LRB-

xi

-RRB-

-RSB-

f

-LRB-

q

-RRB-

-LRB-

-RRB-

This

equation

can

be

solved

if

we

know

the

true

function

For

example

for

the

locally

quadratic

function

we

can

plug

the

quadratic

function

for

f

-LRB-

x

-RRB-

in

Equation

into

Equation

to

get

bias

f

-LRB-

q

-RRB-

X

-LSB-

si

-LRB-

q

-RRB-

-RSB-

f

-LRB-

q

-RRB-

gT

X

-LSB-

si

-LRB-

q

-RRB-

-LRB-

x

q

-RRB-

-RSB-

X

h

si

-LRB-

q

-RRB-

-LRB-

x

q

-RRB-

TH

-LRB-

x

q

-RRB-

i

-LRB-

-RRB-

The

locally

weighted

regression

process

that

generates

sq

guarantees

that

P

si

-LRB-

q

-RRB-

and

since

the

linear

local

model

exactly

matches

any

linear

trend

in

the

data

P

si

-LRB-

q

-RRB-

-LRB-

x?q

-RRB-

Therefore

the

bias

depends

only

on

the

quadratic

term

-LRB-

Katkovnik

Cleveland

and

Loader

-RRB-

bias

X

i

si

-LRB-

q

-RRB-

-LRB-

xi

q

-RRB-

TH

-LRB-

xi

q

-RRB-

-LRB-

-RRB-

assuming

the

ridge

regression

parameters

i

have

been

set

to

zero

This

formula

raises

the

temptation

to

estimate

and

cancel

the

bias

by

estimating

the

second

derivative

matrix

H

It

is

not

clear

that

this

is

better

than

simply

using

a

quadratic

local

model

instead

of

a

linear

local

model

The

quadratic

local

model

would

eliminate

the

local

bias

due

to

the

quadratic

term

-LRB-

and

also

remove

the

need

for

the

distance

metric

to

compensate

for

different

curvature

in

different

directions

-RRB-

Of

course

if

a

quadratic

local

model

is

used

the

bias

will

then

be

due

to

cubic

terms

in

the

Taylor

series

for

the

true

function

whose

elimination

would

require

estimation

of

the

cubic

terms

with

a

cubic

local

model

and

so

on

We

have

not

yet

found

a

principled

termination

of

this

cycle

Assessment

Using

Cross

Validation

We

can

assess

how

well

locally

weighted

learning

is

doing

by

testing

how

well

each

experience

-LRB-

xi

yi

-RRB-

in

the

memory

is

predicted

by

the

rest

of

the

experiences

A

simple

measure

of

the

ith

prediction

error

is

the

difference

between

the

predicted

output

of

the

input

xi

and

the

observed

value

yi

However

for

non-parametric

learners

that

are

overfitting

the

data

this

measure

may

be

deceptively

small

For

example

a

nearest

neighbor

learner

would

always

have

an

error

measure

of

zero

because

-LRB-

xi

yi

-RRB-

will

be

the

closest

neighbor

to

itself

A

more

sophisticated

measure

of

the

ith

prediction

error

is

the

leave-one-out

cross

validation

error

in

which

the

experience

is

first

removed

from

the

memory

before

prediction

Let

ycvi

be

the

output

predicted

for

input

xi

using

the

memory

with

the

ith

point

removed

f

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

xi

yi

-RRB-

-LRB-

xi

yi

-RRB-

-LRB-

xn

yn

-RRB-

g

-LRB-

-RRB-

The

ith

leave-one-out

cross

validation

error

is

ecvi

-LRB-

ycvi

yi

-RRB-

With

the

lazy

learning

formalism

in

which

most

work

takes

place

at

prediction

time

it

is

no

more

expensive

to

predict

a

value

with

one

data

point

removed

than

with

it

included

This

contrasts

with

the

majority

of

learning

methods

that

have

an

explicit

training

stage

in

these

cases

it

is

not

easy

to

pick

an

earlier

experience

and

temporarily

pretend

we

did

not

see

it

Ignoring

a

training

data

point

typically

requires

retraining

from

scratch

with

a

modified

training

set

which

can

be

fairly

expensive

with

a

nonlinear

parametric

model

such

as

a

neural

network

In

addition

the

dependence

of

nonlinear

parametric

training

on

initial

parameter

values

further

complicates

the

analysis

To

handle

this

effect

correctly

many

training

runs

with

different

starting

values

must

be

undertaken

for

each

different

data

set

All

of

this

is

avoided

with

locally

weighted

learning

with

local

models

that

are

linear

in

the

unknown

parameters

although

tuning

of

fit

parameters

does

reintroduce

the

problem

However

tuning

of

fit

parameters

can

be

a

background

process

that

operates

on

a

slower

time

scale

than

adding

new

data

and

answering

queries

Cross

validation

can

also

be

performed

locally

i.e

from

just

fitting

a

locally

linear

model

at

one

query

point

q

-LRB-

Cleveland

and

Loader

-RRB-

We

first

consider

the

locally

weighted

average

of

the

squared

cross

validation

error

MSEcv

at

each

training

point

-LRB-

Myers

-RRB-

MSEcv

-LRB-

q

-RRB-

P

-LRB-

ecvi

xi

-RRB-

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

PK

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

-LRB-

-RRB-

This

estimate

requires

a

locally

weighted

regression

to

be

performed

at

each

training

point

with

non-zero

weight

K

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

One

could

imagine

storing

ecvi

xi

with

each

training

point

but

this

value

would

have

to

be

updated

as

new

data

was

learned

We

approximate

ecvi

xi

ss

ecvi

q.

to

generate

the

following

MSEcv

-LRB-

q

-RRB-

P

-LRB-

ecvi

q

-RRB-

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

PK

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

P

-LRB-

rcvi

q

-RRB-

nLWR

-LRB-

-RRB-

where

rcvi

q

is

the

weighted

cross

validation

error

with

point

i

removed

from

a

locally

weighted

regression

centered

at

q

The

weighted

cross

validation

residual

rcvi

q

is

related

to

the

weighted

residual

-LRB-

ri

wiei

-RRB-

by

-LRB-

Myers

-RRB-

rcvi

ri

zTi

-LRB-

ZTZ

-RRB-

-LRB-

-RRB-

Thus

we

obtain

the

final

equation

for

MSEcv

as

MSEcv

-LRB-

q

-RRB-

nLWR

X

i

ri

zTi

-LRB-

ZTZ

-RRB-

-LRB-

-RRB-

This

equation

is

a

local

version

of

the

PRESS

statistic

-LRB-

Myers

-RRB-

It

allows

us

to

perform

leave-one-out

cross

validation

without

recalculating

the

regression

parameters

for

every

excluded

point

Often

this

is

computationally

very

efficient

Optimal

Fit

Parameters

An

Example

In

this

section

we

will

try

to

find

optimal

fit

parameters

-LRB-

distance

metric

d

-LRB-

-RRB-

weighting

function

K

-LRB-

-RRB-

and

smoothing

bandwidth

h

-RRB-

for

a

simple

example

We

will

make

the

restrictive

assumption

that

the

data

is

uniformly

spaced

on

a

rectangular

grid

We

first

approach

this

question

by

exploring

kernel

shapes

in

one

dimension

We

allow

the

weights

wi

to

be

unknown

and

numerically

optimize

them

to

minimize

the

mean

squared

error

We

assume

the

underlying

function

is

quadratic

with

second

derivative

H

-LRB-

Equation

-RRB-

and

that

there

is

additive

independent

identically

distributed

zero

mean

noise

-LRB-

Equation

-RRB-

with

constance

variance

The

sampled

data

is

regularly

spaced

with

a

distance

of

between

each

data

point

-LRB-

in

Figure

-RRB-

Equation

is

solved

for

s

with

the

query

at

x

The

mean

squared

error

is

the

sum

of

the

bias

-LRB-

Equation

-RRB-

squared

and

the

variance

-LRB-

Equation

-RRB-

This

quantity

is

minimized

by

adjusting

the

weights

wi

The

resulting

kernel

shape

K

-LRB-

d

-RRB-

is

shown

in

Figure

This

kernel

shape

matches

the

quadratic

kernel

K

-LRB-

x

-RRB-

-LRB-

-RRB-

if

jxj

otherwise

-LRB-

-RRB-

which

has

been

described

in

Section

Further

numerical

experimentation

in

one

dimension

revealed

that

the

optimal

scaling

factor

m

for

the

one

dimensional

distance

function

is

approximately

ss

cH

-LRB-

-RRB-

where

c

is

a

constant

that

takes

into

account

issues

such

as

data

spacing

and

the

standard

deviation

of

the

additive

noise

c

oe

-LRB-

-RRB-

The

width

of

the

resulting

kernel

is

directly

related

to

the

optimal

smoothing

bandwidth

In

two

dimensions

we

can

explore

optimization

of

the

distance

metric

Optimizing

the

values

of

the

kernel

at

each

of

the

data

points

is

beyond

our

current

computational

resources

so

we

will

assume

the

form

of

the

kernel

function

is

the

quadratic

kernel

We

will

Figure

The

kernel

shape

that

minimizes

mean

squared

error

in

one

dimension

The

large

single

dot

is

the

predicted

value

whose

deviation

from

zero

the

correct

value

reveals

the

bias

The

vertical

bars

show

the

standard

deviation

of

the

prediction

-LRB-

i.e.

the

square

root

of

the

variance

-RRB-

which

is

greatly

reduced

from

the

standard

deviation

of

of

the

original

data

The

set

of

large

dots

have

been

optimized

to

minimize

the

mean

squared

error

of

the

prediction

and

reveal

the

optimal

kernel

shape

for

this

criterion

The

line

through

these

points

is

a

quadratic

kernel

with

the

appropriate

bandwidth

to

match

the

optimized

kernel

values

The

small

dots

are

the

value

of

the

quadratic

portion

of

the

underlying

function

for

comparison

Figure

Contour

plot

of

f

-LRB-

x

-RRB-

Figure

Contour

plot

of

optimal

kernel

choose

a

particular

value

for

the

Hessian

H

in

Equation

and

then

optimize

the

scaling

matrix

M

for

the

multidimensional

distance

function

to

minimize

the

mean

squared

error

We

found

that

the

optimal

M

approximately

satisfies

the

following

equation

MTM

ss

cH

-LRB-

-RRB-

where

c

is

the

same

as

the

one

dimensional

case

Figure

shows

how

the

Hessian

matrix

H

can

orient

the

quadratic

component

in

an

arbitrary

orientation

The

distance

function

matrix

MTM

needs

to

be

a

full

matrix

in

order

to

allow

the

optimal

kernel

-LRB-

Figure

-RRB-

to

match

the

orientation

of

the

quadratic

component

of

f

-LRB-

x

-RRB-

-LRB-

Figure

-RRB-

For

this

numerical

experiment

H

was

chosen

to

be

H

B

CA

-LRB-

-RRB-

The

optimal

scaling

matrix

M

was

found

by

numerical

search

to

be

M

B

CA

-LRB-

-RRB-

and

Equation

is

approximately

satisfied

as

-LRB-

MTM

-RRB-

H

is

almost

a

multiple

of

the

identity

matrix

for

c

-LRB-

MTM

-RRB-

H

B

CA

-LRB-

-RRB-

Input

Output

Figure

Locally

weighted

regression

approximating

a

dataset

shown

by

the

black

dots

There

is

an

outlier

at

x

ss

Input

Output

Figure

Locally

weighted

regression

supplemented

with

outlier

removal

Noisy

Training

Data

and

Outliers

The

averaging

performed

by

the

locally

weighted

regression

process

naturally

filters

out

noise

if

the

weighting

function

is

not

infinite

at

zero

distance

The

tuning

process

can

optimize

the

noise

filtering

by

adjusting

fit

parameters

such

as

smoothing

parameters

weighting

function

parameters

ridge

regression

parameters

and

choice

of

local

model

structure

However

it

is

often

useful

to

explicitly

identify

outliers

training

points

that

are

erroneous

or

whose

noise

is

much

larger

than

that

of

neighboring

points

An

example

of

the

effect

of

an

outlier

is

given

in

Figure

and

the

effect

of

outlier

rejection

is

shown

in

Figure

Robust

regression

-LRB-

see

for

example

Hampel

et

al.

-RRB-

and

cross

validation

allow

extensions

to

locally

weighted

learners

in

which

we

can

identify

or

reduce

the

effects

of

outliers

Outliers

can

be

identified

and

removed

globally

or

they

can

be

identified

and

ignored

on

a

query

by

query

basis

Query-based

outlier

detection

allows

training

points

to

be

ignored

for

some

queries

and

used

for

other

queries

Other

areas

that

have

been

explored

are

detecting

discontinuities

and

nonstationarity

in

the

training

data

Global

Weighting

of

Stored

Points

and

Finding

Outliers

It

is

possible

to

attach

weights

to

stored

points

during

the

training

process

and

during

lookup

that

downweight

points

that

are

suspected

of

being

unreliable

-LRB-

Aha

and

Kibler

Cost

and

Salzberg

-RRB-

These

weights

can

multiply

the

weight

based

on

the

weighting

function

Totally

unreliable

points

can

be

assigned

a

weight

of

zero

leading

them

to

be

ignored

The

reliability

weights

can

be

based

on

cross

validation

whether

a

stored

point

correctly

predicts

or

classifies

its

neighbors

Another

approach

is

to

only

utilize

stored

points

that

have

shown

that

they

can

reduce

the

cross

validation

error

-LRB-

Aha

-RRB-

Important

issues

are

when

the

weighting

decision

is

made

and

how

often

the

decision

is

reevaluated

Global

methods

typically

assign

a

weight

to

a

point

during

training

in

which

case

the

decision

is

usually

never

reevaluated

or

during

an

asynchronous

database

maintenance

process

in

which

decisions

are

reevaluated

each

time

the

process

cycles

through

the

entire

database

Local

Weighting

of

Stored

Points

and

Finding

Outliers

Local

outlier

detection

methods

do

not

label

points

as

outliers

for

all

queries

as

do

global

methods

Points

can

be

outliers

for

some

queries

and

not

outliers

for

others

We

can

generate

weights

for

training

data

at

query

time

based

on

cross

validation

using

nearby

points

The

PRESS

statistic

-LRB-

Myers

-RRB-

can

be

modified

to

serve

as

a

local

outlier

detector

in

locally

weighted

regression

For

this

we

need

the

standardized

individual

PRESS

residual

-LRB-

also

called

the

Studentized

residual

-RRB-

ePRESS

ri

oe

q

zTi

-LRB-

ZTZ

-RRB-

-LRB-

-RRB-

This

measure

has

zero

mean

and

unit

variance

and

assumes

a

locally

normal

distribution

of

the

error

If

for

a

given

data

point

it

deviates

from

zero

more

than

a

certain

threshold

the

point

can

be

called

an

outlier

A

conservative

threshold

would

be

discarding

all

points

lying

outside

the

area

of

the

normal

distribution

In

our

applications

we

used

cutting

off

all

data

outside

the

area

of

the

normal

distribution

Robust

Regression

Approaches

Data

with

outliers

can

be

viewed

as

having

additive

noise

with

long-tailed

symmetric

distributions

Robust

regression

is

useful

for

both

global

and

local

detection

of

outliers

-LRB-

Cleveland

-RRB-

A

bisquare

weighting

function

is

used

to

additionally

downweight

points

based

on

their

residuals

ui

ei

if

jeij

otherwise

-LRB-

-RRB-

where

eMED

is

the

median

of

the

absolute

value

of

the

residuals

ei

The

weights

now

become

wi

uiK

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

This

process

is

repeated

about

times

to

refine

the

estimates

of

ui

Tuning

Like

most

learning

algorithms

locally

weighted

learning

usually

needs

to

be

tuned

to

work

well

for

a

particular

problem

Tuning

means

adjusting

the

parameters

of

the

learning

algorithm

itself

The

locally

weighted

fit

criteria

is

C

-LRB-

q

-RRB-

X

i

-LRB-

f

-LRB-

xi

-RRB-

yi

-RRB-

d

-LRB-

xi

q

-RRB-

h

-LRB-

-RRB-

It

includes

several

fit

parameters

the

bandwidth

or

smoothing

parameter

h

the

distance

metric

d

-LRB-

-RRB-

and

the

weighting

or

kernel

function

K

-LRB-

-RRB-

There

are

additional

fit

parameters

such

as

ridge

regression

parameters

and

outlier

thresholds

There

are

several

ways

to

tune

these

fit

parameters

ffl

Global

tuning

The

fit

parameters

are

set

globally

by

an

optimization

process

that

typically

minimizes

cross

validation

error

over

all

the

data

and

therefore

constant

size

and

shape

volumes

of

data

are

used

to

answer

queries

ffl

Query-based

local

tuning

The

fit

parameters

are

set

on

each

query

based

on

local

information

ffl

Point-based

local

tuning

The

weighted

training

criteria

uses

different

fit

parameters

for

each

point

xi

a

bandwidth

hi

a

distance

metric

di

-LRB-

-RRB-

a

weighting

function

Ki

-LRB-

-RRB-

and

possibly

a

weight

wxi

C

-LRB-

q

-RRB-

X

i

-LRB-

f

-LRB-

xi

-RRB-

yi

-RRB-

di

-LRB-

xi

q

-RRB-

hi

-LRB-

-RRB-

In

typical

implementations

of

this

approach

the

fit

parameters

are

computed

in

advance

of

the

queries

and

are

stored

with

the

data

points

There

are

several

approaches

to

computing

the

fit

parameter

values

ffl

Plug-in

approach

The

fit

parameters

can

be

set

by

a

direct

computation

ffl

Optimization

approaches

The

fit

parameters

can

be

set

by

an

optimization

process

that

either

-LRB-

Marron

-RRB-

-LCB-

minimizes

the

training

set

error

-LCB-

minimizes

the

test

or

validation

set

error

-LCB-

minimizes

the

cross

validation

error

-LRB-

CV

-RRB-

-LCB-

minimizes

the

generalized

cross

validation

error

-LRB-

GCV

-RRB-

-LRB-

Myers

-RRB-

-LCB-

maximizes

Akaike

's

information

criterion

-LRB-

AIC

-RRB-

-LCB-

or

adjusts

Mallow

's

Cp

Fit

parameters

can

not

be

optimized

in

isolation

The

combination

of

all

fit

parameters

generates

a

particular

fit

quality

If

one

fit

parameter

is

changed

typically

the

optimal

values

of

other

parameters

change

in

response

If

a

locally

constant

model

is

used

then

the

smoothing

parameter

and

distance

function

must

reflect

the

flatness

of

the

neighborhood

in

different

directions

If

the

local

model

is

a

hyperplane

the

smoothing

parameter

and

distance

function

must

reflect

the

second

derivative

of

the

neighborhood

If

the

local

model

is

quadratic

it

is

the

third

spatial

derivative

of

the

data

that

must

be

dealt

with

For

practical

purposes

it

would

be

useful

to

have

a

clear

understanding

of

how

accurate

the

non-linear

fit

parameters

should

be

for

a

good

fit

Our

intuition

is

that

approximate

values

usually

result

in

barely

distinguishable

performance

from

optimal

parameters

in

practical

use

although

-LRB-

Brockmann

et

al.

-RRB-

states

that

this

is

not

true

for

h

in

kernel

regression

The

next

section

considers

optimizing

a

single

set

of

parameters

for

all

possible

future

queries

-LRB-

global

tuning

-RRB-

Section

considers

optimizing

multiple

sets

of

parameters

for

specific

queries

-LRB-

local

tuning

-RRB-

Global

Tuning

Global

cross-validation

can

be

a

particularly

robust

method

for

tuning

parameters

because

it

does

not

make

any

special

assumptions

Independent

of

the

noise

distribution

data

distribution

and

underlying

function

the

cross-validation

value

is

an

unbiased

estimate

of

how

well

a

given

set

of

parameters

will

perform

on

new

data

drawn

from

the

same

distribution

as

the

old

data

This

robustness

has

lead

to

the

use

of

global

cross-validation

in

applications

that

attempt

to

achieve

high

autonomy

by

making

few

assumptions

such

as

the

General

Memory

Based

Learning

-LRB-

GMBL

-RRB-

system

described

in

-LRB-

Moore

et

al.

-RRB-

GMBL

performs

large

amounts

of

cross

validation

search

to

optimize

feature

subsets

the

diagonal

elements

of

the

distance

metric

the

smoothing

parameter

and

the

order

of

the

regression

Continuous

Search

Continuous

fit

parameters

make

continuous

search

possible

Inevitably

this

is

local

hill

climbing

with

a

large

risk

of

getting

stuck

in

local

optima

The

sum

of

the

squared

cross

validation

errors

is

minimized

using

a

nonlinear

parameter

estimation

procedure

-LRB-

e.g.

MINPACK

-LRB-

More

et

al.

-RRB-

or

-LRB-

Dennis

et

al.

-RRB-

-RRB-

As

discussed

in

Section

in

this

locally

weighted

learning

approach

computing

the

cross

validation

error

for

a

single

point

is

no

more

computationally

expensive

than

answering

a

query

This

is

quite

different

from

parametric

approaches

such

as

a

neural

network

where

a

new

model

-LRB-

network

-RRB-

must

be

trained

for

each

cross

validation

training

set

with

a

particular

point

removed

In

addition

if

the

local

model

is

linear

in

the

unknown

parameters

we

can

analytically

take

the

derivative

of

the

cross

validation

error

with

respect

to

the

parameters

to

be

estimated

which

greatly

speeds

up

the

search

process

We

can

use

the

optimized

distance

metric

to

find

which

input

variables

are

more

or

less

important

to

the

function

being

represented

Distance

scaling

factors

that

go

to

zero

indicate

directions

that

are

irrelevant

or

are

consistent

with

the

local

model

structure

and

that

a

global

model

will

suffice

for

those

directions

We

can

also

interpret

the

ridge

regression

parameters

The

ridge

regression

parameter

for

irrelevant

terms

in

the

local

model

become

very

large

in

the

fit

parameter

optimization

process

The

effect

of

this

is

to

force

the

corresponding

estimated

parameter

i

to

the

apriori

value

i

which

corresponds

to

a

dimensionality

reduction

A

relatively

unexplored

area

is

stochastic

gradient

descent

approaches

to

optimizing

fit

parameters

Rather

than

use

all

the

cross

validation

errors

and

their

associated

contributions

to

the

derivative

why

not

use

only

a

small

random

sample

of

the

cross

validation

errors

and

their

associated

derivative

contributions

Racine

-LRB-

-RRB-

describes

an

approach

to

optimizing

fit

parameters

based

on

partitioning

the

training

data

into

subsets

calculating

cross

validation

errors

for

each

subset

based

only

on

data

in

the

subset

and

then

averaging

the

results

Discrete

Search

Discrete

search

algorithms

for

good

fit

parameters

is

an

active

area

of

research

Maron

and

Moore

-LRB-

-RRB-

describe

racing

techniques

to

find

good

fit

parameter

values

These

techniques

compare

a

wide

range

of

different

types

of

models

simultaneously

and

handle

models

with

discrete

parameters

Bad

models

are

quickly

dropped

from

the

race

which

focuses

computational

effort

on

distinguishing

between

the

good

models

Typically

any

continuous

fit

parameters

are

discretized

-LRB-

Maron

and

Moore

-RRB-

Techniques

for

selecting

features

in

the

distance

metric

and

local

model

have

been

developed

in

statistics

-LRB-

Draper

and

Smith

Miller

-RRB-

including

all

subsets

forward

regression

backwards

regression

and

stepwise

regression

We

have

explored

stepwise

regression

procedures

to

determine

which

terms

of

the

local

model

are

useful

with

similar

results

to

the

gradient

based

search

described

above

Feature

selection

is

a

hard

problem

because

the

features

can

not

be

examined

independently

The

value

of

a

feature

depends

on

which

other

features

are

also

selected

Thus

the

goal

is

to

find

a

set

of

feature

weights

not

individual

feature

weights

for

each

feature

In

Maron

and

Moore

-LRB-

-RRB-

a

number

of

algorithms

for

doing

this

are

described

and

compared

including

methods

based

on

Monte-Carlo

sampling

Aha

-LRB-

-RRB-

gives

an

algorithm

that

constructs

new

features

in

addition

to

selecting

features

Friedman

-LRB-

-RRB-

gives

techniques

for

query

dependent

feature

construction

Continuous

vs.

Discrete

Search

Discrete

search

can

explore

settings

for

discrete

fit

parameters

and

even

select

training

algorithm

features

or

function

approximation

methods

-LRB-

e.g.

locally

weighted

regression

neural

networks

rule-based

systems

-RRB-

It

would

seem

that

continuous

fit

parameter

optimization

can

not

make

these

choices

However

this

is

not

the

case

By

blending

the

output

of

different

approaches

with

a

blending

parameter

continuous

search

can

choose

model

order

algorithm

features

and

approximation

method

For

example

could

be

optimized

to

blend

two

methods

-LRB-

-RRB-

and

-LRB-

-RRB-

in

the

following

equation

f

-LRB-

q

-RRB-

-LRB-

q

-RRB-

-LRB-

-RRB-

-LRB-

q

-RRB-

-LRB-

-RRB-

Cleveland

and

Loader

-LRB-

c

-RRB-

present

an

approach

to

automatically

choose

the

local

model

structure

-LRB-

i.e.

order

of

the

polynomial

model

-RRB-

by

blending

polynomial

models

where

a

non-integral

model

order

indicates

a

weighted

blend

between

two

integral

model

orders

They

use

cross

validation

to

optimize

the

local

model

order

on

each

query

Local

Tuning

Local

fit

parameter

optimization

is

referred

to

as

adaptive

or

variable

in

the

statistics

literature

as

in

adaptive

bandwidth

or

variable

bandwidth

smoothers

There

are

several

reasons

to

consider

local

tuning

although

it

dramatically

increases

the

number

of

degrees

of

freedom

in

the

training

process

leading

to

increased

variance

of

the

predictions

and

an

increased

risk

of

overfitting

the

data

-LRB-

Cleveland

and

Loader

-RRB-

ffl

Adaptation

to

the

data

density

and

distribution

This

adaptation

is

in

addition

to

the

adaptation

provided

by

the

locally

weighted

regression

procedure

itself

-LRB-

Bottou

and

Vapnik

-RRB-

ffl

Adaptation

to

variations

in

the

noise

level

in

the

training

data

These

variations

are

known

as

heteroscedasticity

ffl

Adaptation

to

variations

in

the

behavior

of

the

underlying

function

The

function

may

be

locally

planer

in

some

regions

and

have

high

curvature

in

others

Normalized

RMS

Torque

TrainingAfter

Figure

Performance

of

various

methods

on

two

joint

arm

dynamics

Plug-in

estimators

have

been

derived

and

local

-LRB-

locally

weighted

-RRB-

training

set

error

cross

validation

or

validation

-LRB-

test

-RRB-

set

error

can

drive

an

optimization

of

the

local

model

Interference

Negative

interference

between

old

and

new

training

data

is

one

of

the

most

important

motivations

for

exploring

locally

weighted

learning

To

illustrate

the

differences

between

global

parametric

representations

and

a

locally-weighted

learning

approach

a

sigmoidal

feedforward

neural

network

approach

was

compared

to

a

locally

weighted

learning

approach

on

the

same

problem

The

architecture

for

the

sigmoidal

feedforward

neural

network

was

taken

from

-LRB-

Goldberg

and

Pearlmutter

section

-RRB-

who

modeled

arm

inverse

dynamics

The

ability

of

each

of

these

methods

to

predict

the

torques

of

the

simulated

two

joint

arm

at

random

points

was

compared

-LRB-

Atkeson

-RRB-

Figure

plots

the

normalized

RMS

prediction

error

The

points

were

sampled

uniformly

using

ranges

comparable

to

those

used

in

Miller

et

al.

-LRB-

-RRB-

which

also

looked

at

two

joint

arm

inverse

dynamics

modeling

Initially

each

method

was

trained

on

a

training

set

of

random

samples

and

then

the

predictions

of

the

torques

on

a

separate

test

set

of

random

samples

of

the

two

joint

arm

dynamics

function

were

assessed

The

solid

bar

marked

LWR

at

location

shows

the

test

set

error

of

a

locally

weighted

regression

with

a

quadratic

local

model

The

light

bar

marked

NN

at

location

shows

the

best

test

set

error

of

the

neural

network

Both

methods

generalize

well

on

this

problem

-LRB-

bars

and

have

low

error

-RRB-

Each

method

was

then

trained

on

ten

attempts

to

make

a

particular

desired

movement

Each

method

successfully

learned

the

desired

movement

After

this

second

round

of

training

performance

on

the

random

test

set

was

again

measured

-LRB-

bars

at

locations

and

-RRB-

The

sigmoidal

feedforward

neural

network

lost

its

memory

of

the

full

dynamics

-LRB-

the

light

bar

at

location

has

a

large

error

-RRB-

and

represented

only

the

dynamics

of

the

particular

movements

being

learned

in

the

second

training

set

This

interference

between

new

and

previously

learned

data

was

not

prevented

by

increasing

the

number

of

hidden

units

in

the

single

layer

network

from

up

to

The

locally

weighted

learning

method

did

not

show

this

interference

effect

-LRB-

solid

bar

at

location

-RRB-

The

interference

is

caused

by

the

failure

of

the

neural

network

model

structure

to

match

the

arm

inverse

dynamics

structure

perfectly

There

is

no

noise

in

the

data

and

no

concept

drift

so

these

causes

are

eliminated

as

possible

sources

of

the

interference

It

can

be

argued

that

the

sigmoidal

neural

network

forgot

the

original

training

data

because

we

did

not

include

that

data

in

the

second

training

data

set

-LRB-

learning

a

specific

movement

-RRB-

That

is

exactly

our

point

if

all

past

data

is

retained

to

combat

interference

then

the

method

becomes

a

lazy

learning

method

In

that

case

we

argue

that

one

should

take

advantage

of

the

opportunity

to

locally

weight

the

training

procedure

and

get

better

performance

-LRB-

Vapnik

Bottou

and

Vapnik

Vapnik

and

Bottou

-RRB-

Implementing

Locally

Weighted

Learning

There

are

several

concerns

about

locally

weighted

learning

systems

including

whether

locally

weighted

learning

systems

can

answer

queries

fast

enough

and

whether

their

speed

will

unacceptably

degrade

as

the

size

of

the

database

grows

This

section

explores

these

concerns

We

discuss

fast

ways

to

find

relevant

data

using

either

k-d

trees

in

software

special

purpose

hardware

or

massively

parallel

computers

and

the

current

performance

of

our

LWR

implementation

Our

goal

is

to

minimize

the

need

for

compromises

such

as

forgetting

-LRB-

discarding

data

-RRB-

to

keep

the

database

size

under

a

limit

instance

averaging

which

averages

similar

data

or

maintaining

an

elaborate

data

structure

of

intermediate

results

to

accelerate

query

processing

We

will

not

discuss

LWR

acceleration

approaches

that

are

limited

to

low

dimensional

problems

such

as

binning

-LRB-

Fan

and

Marron

Turlach

and

Wand

-RRB-

Other

discussions

of

fast

implementations

include

Seifert

et

al.

-LRB-

-RRB-

and

Seifert

and

Gasser

-LRB-

-RRB-

Retrieving

Relevant

Data

The

choice

of

method

for

storing

experiences

depends

on

what

fraction

of

the

experiences

are

used

in

each

locally

weighted

regression

and

what

computational

technology

is

available

If

all

of

the

experiences

are

used

in

each

locally

weighted

regression

then

simply

maintaining

a

list

or

array

of

experiences

is

sufficient

If

only

nearby

experiences

are

included

in

the

locally

weighted

regression

then

an

efficient

method

of

finding

nearest

neighbors

is

required

Nearest

neighbor

lookup

can

be

accelerated

on

a

serial

processor

using

the

k-d

tree

data

structure

Parallel

processors

and

special

purpose

processors

typically

use

parallel

exhaustive

search

K-d

Trees

Naively

implemented

search

for

a

d

dimensional

nearest

neighbor

in

a

database

of

size

n

requires

n

distance

computations

Nearest

neighbor

search

can

be

implemented

efficiently

by

means

of

a

k-d

tree

-LRB-

Bentley

Friedman

et

al.

Bentley

and

Friedman

Bentley

et

al.

Murphy

and

Selkow

Ramasubramanian

and

Paliwal

Broder

Samet

Sproull

-RRB-

A

k-d

tree

is

a

binary

data

structure

that

recursively

splits

a

d-dimensional

space

into

smaller

subregions

The

search

for

a

nearest

neighbor

proceeds

by

initially

searching

the

k-d

tree

in

the

branches

nearest

the

query

point

Frequently

distance

constraints

mean

there

is

no

need

to

explore

further

branches

Figure

shows

a

k-d

tree

segmenting

a

two

dimensional

space

The

shaded

regions

correspond

to

areas

of

the

k-d

tree

that

were

not

searched

The

access

time

is

asymptotically

logarithmic

in

n

the

size

of

the

memory

although

Figure

Generally

during

a

nearest

neighbor

search

only

a

few

leaf

nodes

need

to

be

inspected

The

query

point

is

marked

by

an

x

and

the

distance

to

the

nearest

neighbor

is

indicated

by

a

circle

Black

nodes

are

those

inspected

on

the

path

to

the

leaf

node

often

overhead

costs

mean

that

nearly

all

the

data

points

will

be

accessed

in

a

supposed

logarithmic

search

for

example

with

eight

dimensions

or

more

and

fewer

than

approximately

uniformly

distributed

data

points

In

fact

given

uniformly

distributed

data

points

the

tree

size

for

which

logarithmic

performance

is

noticeable

increases

exponentially

with

dimensionality

Two

things

can

alleviate

this

problem

First

the

data

points

are

unlikely

to

be

distributed

uniformly

In

fact

the

less

randomly

distributed

the

training

data

is

the

better

Second

there

are

approximate

algorithms

that

can

find

one

or

more

nearby

experiences

without

guaranteeing

they

are

the

nearest

that

do

operate

in

logarithmic

time

Empirically

these

approximations

do

not

greatly

reduce

prediction

accuracy

-LRB-

Omohundro

Moore

-RRB-

Bump

trees

-LRB-

Omohundro

-RRB-

are

another

promising

efficient

approximation

Cleveland

et

al.

-LRB-

-RRB-

Farmer

and

Sidorowich

-LRB-

b

-RRB-

Renka

-LRB-

-RRB-

Grosse

-LRB-

-RRB-

Moore

-LRB-

-RRB-

Cleveland

and

Grosse

-LRB-

-RRB-

Karali

c

-LRB-

-RRB-

Townshend

-LRB-

-RRB-

Loader

-LRB-

-RRB-

Wess

et

al.

-LRB-

-RRB-

Deng

and

Moore

-LRB-

-RRB-

Lowe

-LRB-

-RRB-

and

van

der

Smagt

et

al.

-LRB-

-RRB-

have

used

trees

in

memory-based

learning

and

locally

weighted

regression

Special

Purpose

Devices

Special

purpose

hardware

for

finding

nearest

neighbors

has

a

long

history

-LRB-

Taylor

Steinbuch

Steinbuch

and

Piske

Kazmierczak

and

Steinbuch

Batchelor

-RRB-

These

machines

calculated

either

a

Manhattan

or

Euclidean

distance

for

all

stored

points

and

then

did

comparisons

to

pick

the

winning

point

The

current

version

of

this

technology

is

the

wafer

scale

memory

-LCB-

based

reasoning

devices

proposed

by

Yasunaga

and

Kitano

-LRB-

-RRB-

The

devices

allocate

one

processor

per

data

point

and

can

handle

approximately

million

data

points

per

inch

wafer

The

designers

have

exploited

the

properties

of

memory-based

learning

in

two

ways

First

the

resolution

of

the

computed

distance

is

not

critical

so

analog

adders

and

multipliers

are

used

for

weighting

and

distance

calculations

instead

of

digital

circuits

saving

much

space

on

the

silicon

for

other

processors

Second

the

device

is

robust

to

faulty

processors

in

that

a

faulty

processor

only

causes

the

loss

of

a

single

data

point

The

authors

advocate

simply

ignoring

processor

failures

although

it

would

be

possible

to

map

the

faulty

processors

and

skip

them

when

loading

data

Massively

Parallel

Implementations

Many

nearest

neighbor

systems

have

been

implemented

on

massively

parallel

Connection

Machines

-LRB-

Waltz

-RRB-

On

a

massively

parallel

computer

such

as

the

and

-LRB-

Hillis

-RRB-

exhaustive

search

is

often

faster

than

using

k-d

trees

due

to

the

limited

number

of

experiences

allocated

to

each

processor

The

Connection

Machine

can

have

up

to

-LRB-

-RRB-

processors

and

can

simulate

a

parallel

computer

with

many

more

processors

Experiences

are

stored

in

the

local

memory

associated

with

each

processor

An

experience

can

be

compared

to

the

desired

experience

in

each

processor

with

the

processors

running

in

parallel

and

then

a

hardwired

global-OR

bus

can

be

used

to

find

the

closest

match

in

constant

time

independent

of

the

number

of

stored

experiences

The

search

time

depends

linearly

on

the

number

of

dimensions

in

the

distance

metric

and

the

distance

metric

can

be

changed

easily

or

made

to

depend

on

the

current

query

point

The

critical

feature

of

the

massively

parallel

computer

system

is

the

use

of

associative

memories

in

addition

to

multiple

processors

-LRB-

Higuchi

et

al.

-RRB-

There

are

processors

-LRB-

Transputers

-RRB-

in

the

but

each

processor

has

bits

of

associative

memory

which

increases

the

effective

number

of

processors

to

This

architecture

is

well

suited

for

memory-based

learning

where

the

distance

metric

involves

exact

matches

of

symbolic

fields

as

that

is

the

operation

the

associative

memory

chips

can

support

Future

associative

memories

might

implement

Euclidean

distance

as

a

basic

operation

There

have

been

implementations

of

memory-based

translation

and

parsing

on

the

-LRB-

Kitano

and

Higuchi

b

Sumita

et

al.

Kitano

b

-RRB-

The

current

generic

parallel

computer

seems

to

be

on

the

order

of

standard

microprocessors

tightly

connected

with

a

communication

network

Examples

of

this

design

are

the

and

the

SNAP

system

-LRB-

Kitano

et

al.

-RRB-

The

details

of

the

communication

network

are

not

critical

to

locally

weighted

learning

since

the

time

critical

processing

consists

of

broadcasting

the

query

to

the

processors

and

determining

which

answer

is

the

best

which

can

easily

be

done

with

a

prespecified

communication

pattern

This

form

of

communication

is

not

difficult

to

implement

This

machine

does

not

have

the

thousands

of

processors

that

make

exhaustive

search

the

obvious

nearest

neighbor

algorithm

The

processors

will

probably

maintain

some

sort

of

search

data

structure

such

as

a

k-d

tree

although

the

local

k-d

trees

may

be

too

small

for

efficient

search

performance

Kitano

et

al.

-LRB-

-RRB-

describe

an

implementation

of

memory-based

reasoning

on

the

SNAP

system

This

type

of

parallel

computer

is

excellent

for

locally

weighted

learning

where

the

regression

calculation

dominates

the

lookup

time

if

a

large

fraction

of

the

points

are

used

in

each

regression

Implementing

Locally

Weighted

Regression

Locally

weighted

learning

minimizes

the

computational

cost

of

training

new

data

points

are

simply

stored

in

the

memory

The

price

for

trivial

training

costs

is

a

more

expensive

lookup

procedure

Locally

weighted

regression

uses

a

relatively

complex

regression

procedure

to

form

the

local

model

and

is

thus

more

expensive

than

nearest

neighbor

and

weighted

average

memory-based

learning

procedures

For

each

query

a

new

local

model

is

formed

The

rate

at

which

local

models

can

be

formed

and

evaluated

limits

the

rate

at

which

queries

can

be

answered

We

have

implemented

the

locally

weighted

regression

procedure

on

a

Intel

microprocessor

The

peak

computation

rate

of

this

processor

is

MFlops

We

have

achieved

effective

computation

rates

of

MFlops

on

a

learning

problem

with

input

dimensions

and

output

dimensions

using

a

linear

local

model

This

leads

to

a

lookup

time

of

approximately

milliseconds

on

a

database

of

points

using

exhaustive

search

This

time

includes

distance

and

weight

calculation

for

all

the

stored

points

forming

the

regression

matrix

and

solving

the

normal

equations

Applications

of

Locally

Weighted

Learning

The

presence

of

the

LOWESS

and

LOESS

software

in

the

S

statistics

package

has

lead

to

the

use

of

locally

weighted

regression

as

a

standard

tool

in

many

areas

including

modeling

biological

motor

control

feeding

cycles

in

smokers

and

nonsmokers

lead-induced

anemia

categories

of

tonal

alignment

in

spoken

English

and

growth

and

sexual

maturation

during

disease

-LRB-

Cleveland

Cleveland

et

al.

-RRB-

Atkeson

et

al.

-LRB-

-RRB-

survey

our

own

work

in

applying

locally

weighted

learning

to

robot

control

Zografski

has

explored

the

use

of

locally

weighted

regression

in

robot

control

and

modeling

time

series

and

also

compared

LWR

to

neural

networks

and

other

methods

-LRB-

Zografski

Zografski

and

Durrani

-RRB-

Gorinevsky

and

Connolly

-LRB-

-RRB-

compared

several

different

approximation

schemes

-LRB-

neural

nets

Kohonen

maps

radial

basis

functions

and

local

polynomical

fits

-RRB-

on

simulated

robot

inverse

kinematics

with

added

noise

and

showed

that

local

polynomial

fits

were

more

accurate

than

all

other

methods

van

der

Smagt

et

al.

-LRB-

-RRB-

learned

robot

kinematics

using

local

linear

models

at

the

leaves

of

a

tree

data

structure

Tadepalli

and

Ok

-LRB-

-RRB-

apply

local

linear

regression

to

reinforcement

learning

Thrun

-LRB-

-RRB-

and

Thrun

and

O'Sullivan

-LRB-

-RRB-

apply

nearest

neighbor

techniques

and

weighted

averaging

to

robot

learning

Farmer

and

Sidorowich

-LRB-

b

-RRB-

apply

locally

weighted

regression

to

modeling

and

prediction

of

chaotic

dynamic

systems

Lawrence

et

al.

-LRB-

-RRB-

compare

neural

networks

and

local

regression

methods

on

several

benchmark

problems

Local

regression

outperformed

neural

networks

on

half

the

benchmarks

Factors

affecting

performance

included

whether

the

data

had

differing

density

over

the

input

space

noise

level

dimensionality

and

the

nature

of

the

function

underlying

the

data

Several

researchers

have

applied

locally

weighted

averaging

and

regression

to

free

form

and

deformation

morphing

and

image

interpolation

in

computer

graphics

-LRB-

Goshtasby

Wolberg

Ruprecht

and

Muller

Ruprecht

et

al.

-RRB-

Coughran

Jr.

and

Grosse

-LRB-

-RRB-

describe

using

locally

weighted

regression

for

scientific

visualization

and

auralization

of

data

Ge

et

al.

-LRB-

-RRB-

apply

locally

weighted

regression

to

predict

cell

density

in

a

fermentation

process

They

used

nearest

neighbor

weighting

and

a

tricube

weighting

function

They

also

used

principal

components

and

cross

validation

to

select

features

globally

Locally

weighted

regression

outperformed

other

methods

including

a

global

nonlinear

regression

Hammond

-LRB-

-RRB-

used

LWR

to

model

fermentation

as

well

Naes

et

al.

-LRB-

-RRB-

Naes

and

Isaksson

-LRB-

-RRB-

and

Wang

et

al.

-LRB-

-RRB-

apply

locally

weighted

regression

to

analytical

chemistry

They

use

global

principal

components

to

reduce

the

dimensionality

of

the

inputs

and

they

use

cross

validation

to

set

the

number

of

components

to

use

They

also

explore

several

weighted

Euclidean

distance

metrics

including

weighting

depending

on

the

range

of

the

data

in

principal

component

coordinates

weighting

depending

on

how

good

that

dimension

is

in

predicting

the

output

and

a

distance

metric

that

includes

the

output

value

They

use

a

quadratic

local

model

and

the

tricube

weighting

function

They

use

cross

validation

to

select

the

number

of

points

to

include

in

the

local

regression

They

make

the

important

point

that

optimal

experiment

design

is

quite

different

when

using

locally

weighted

regression

as

compared

to

global

linear

regression

Tamada

et

al.

-LRB-

-RRB-

apply

memory-based

learning

to

water

demand

forecasting

They

select

features

using

Akaike

's

Information

Criterion

-LRB-

AIC

-RRB-

and

use

locally

weighted

averaging

within

a

neighborhood

They

use

a

default

temporally

local

regression

scheme

if

no

points

are

found

in

the

neighborhood

They

use

error

rates

to

set

feature

weights

and

to

perform

outlier

removal

Townshend

-LRB-

-RRB-

applies

locally

weighted

regression

to

the

analysis

modeling

coding

and

prediction

of

speech

signals

He

uses

a

singular

value

decomposition

to

reduce

the

dimensionality

of

the

regression

to

a

fixed

value

D

determined

from

other

criteria

He

uses

the

k

closest

points

to

form

the

local

model

The

distance

to

the

nearest

point

is

used

as

an

estimate

of

the

confidence

in

the

prediction

A

clustering

process

on

the

inputs

and

the

outputs

-LRB-

xi

yi

-RRB-

is

used

to

handle

noise

and

one

to

many

mapping

problems

A

k-d

tree

is

used

to

speed

up

nearest

neighbor

search

This

process

lead

to

a

significant

improvement

over

a

linear

predictor

Wijnberg

and

Johnson

-LRB-

-RRB-

apply

locally

weighted

regression

to

interpolating

air

quality

measurements

They

used

cross

validation

to

optimize

the

smoothing

parameter

globally

but

did

not

find

a

well

defined

minimum

for

the

smoothing

parameter

Kozek

-LRB-

-RRB-

describe

using

LWR

to

model

automobile

emissions

Walden

and

Prescott

-LRB-

-RRB-

use

LWR

to

remove

trends

in

time

series

involving

climate

data

Solow

-LRB-

-RRB-

estimated

the

variance

or

noise

level

in

time

series

climate

data

after

having

removed

the

mean

using

LWR

Locally

weighted

regression

has

also

been

applied

in

economics

and

econometrics

-LRB-

Meese

and

Wallace

LeBaron

-RRB-

Meese

and

Rose

-LRB-

-RRB-

used

LWR

to

model

exchange

rates

and

conclude

that

no

significant

nonlinearity

exists

in

the

data

Diebold

and

Nason

-LRB-

-RRB-

also

used

LWR

to

predict

exchange

rates

without

any

more

success

than

other

nonparametric

regression

techniques

Turetsky

et

al.

-LRB-

-RRB-

and

Raz

et

al.

-LRB-

-RRB-

use

LWR

to

smooth

biological

evoked

potential

data

and

explore

approaches

to

choosing

the

smoothing

parameter

Bottou

and

Vapnik

-LRB-

-RRB-

apply

locally

weighted

classification

to

optical

character

recognition

-LRB-

OCR

-RRB-

Rust

and

Bornman

-LRB-

-RRB-

apply

LWR

to

marketing

data

There

have

been

a

range

of

applications

of

locally

weighted

techniques

in

statistics

-LRB-

Cleveland

Cleveland

and

Loader

-RRB-

The

idea

of

local

fitting

was

extended

to

likelihood-based

regression

models

by

Tibshirani

and

Hastie

-LRB-

-RRB-

and

Hastie

and

Tibshirani

-LRB-

-RRB-

applied

locally

weighted

techniques

to

many

distributional

settings

such

as

logistic

regression

and

developed

general

fitting

algorithms

Lejeune

and

Sarda

-LRB-

-RRB-

applied

locally

weighted

regression

to

estimation

of

distribution

and

density

functions

Cleveland

et

al.

-LRB-

-RRB-

applied

locally

weighted

regression

to

density

estimation

spectrum

estimation

and

predicting

binary

variables

Fan

and

Kreutzberger

-LRB-

-RRB-

applied

locally

weighted

regression

to

spectral

density

estimation

Discussion

What

Is

A

Local

Learning

Approach

To

explore

the

idea

of

local

learning

it

is

useful

to

first

consider

what

a

global

learner

is

A

global/distributed

representation

is

typically

characterized

by

Incrementally

learning

a

single

new

training

point

affects

many

parameters

A

prediction

or

answer

to

a

query

also

depends

on

many

parameters

and

are

characteristics

of

distributed

representations

An

additional

criterion

There

are

many

fewer

parameters

than

data

could

serve

as

a

definition

of

a

global

representation

or

model

and

is

a

good

predictor

that

and

will

be

true

for

a

particular

method

However

it

is

also

possible

to

have

local

methods

with

attribute

and

not

attributes

and

such

as

a

low

resolution

tabular

representation

with

non-overlapping

cells

A

part

of

the

design

space

that

has

not

been

explored

are

learning

algorithms

with

huge

numbers

of

parameters

that

use

distributed

representations

-LRB-

and

but

not

-RRB-

There

are

at

least

three

different

views

of

what

constitutes

local

learning

local

representations

local

selection

and

locally

weighted

learning

This

has

lead

to

some

confusion

and

convoluted

terminology

In

a

local

representation

each

new

data

point

affects

a

small

subset

of

the

parameters

and

answering

a

query

involves

a

small

subset

of

the

parameters

as

well

This

view

of

local

learning

stems

from

the

distinction

between

local

and

distributed

representations

in

neuroscience

-LRB-

Thorpe

-RRB-

Examples

of

local

representations

are

lookup

tables

and

exemplar/prototype

based

classifiers

It

is

not

necessarily

the

case

that

the

number

of

parameters

in

the

representation

be

on

the

order

of

the

number

of

data

points

-LRB-

i.e.

a

considerable

amount

of

local

averaging

can

occur

-RRB-

Local

selection

methods

store

all

-LRB-

or

most

-RRB-

of

the

training

data

and

use

a

distance

function

to

determine

which

stored

points

are

relevant

to

the

query

The

function

of

local

selection

is

to

select

a

single

output

using

nearest

neighbor

or

using

a

distance-based

voting

scheme

-LRB-

k-nearest

neighbor

-RRB-

Examples

of

these

types

of

approaches

are

common

and

include

Stanfill

and

Waltz

-LRB-

-RRB-

and

Aha

-LRB-

-RRB-

Locally

weighted

learning

stores

the

training

data

explicitly

-LRB-

as

do

local

selection

approaches

-RRB-

and

only

fits

parameters

to

the

training

data

when

a

query

is

known

The

critical

feature

of

locally

weighted

learning

is

that

a

criterion

locally

weighted

with

respect

to

the

query

location

is

used

to

fit

some

type

of

parametric

model

to

the

data

-LRB-

Vapnik

Bottou

and

Vapnik

Vapnik

and

Bottou

-RRB-

We

have

the

paradoxical

situation

that

seemingly

global

model

structures

-LRB-

e.g.

polynomials

multilayer

sigmoidal

neural

nets

-RRB-

are

being

called

local

models

because

of

the

locally

weighted

training

criterion

All

of

the

data

can

be

involved

in

training

the

local

model

as

long

as

distant

data

matters

less

than

nearby

data

This

paper

explores

locally

weighted

training

procedures

which

involves

deferring

processing

the

training

data

until

a

query

is

present

leading

to

the

use

of

the

terms

lazy

learning

and

least

commitment

learning

There

are

many

global

approaches

and

representations

such

as

rules

decision

trees

and

parametric

models

-LRB-

e.g.

polynomials

sigmoidal

neural

nets

radial

basis

functions

projection

pursuit

networks

-RRB-

All

of

the

above

approaches

can

be

transformed

into

locally

weighted

approaches

by

using

a

locally

weighted

training

criterion

-LRB-

Vapnik

Bottou

and

Vapnik

Vapnik

and

Bottou

Kozek

-RRB-

so

the

scope

of

locally

weighted

learning

is

quite

broad

We

will

discuss

locally

weighted

classification

as

an

example

Locally

Weighted

Classification

In

classification

there

are

several

ways

to

incorporate

distance

weighting

In

k-nearest

neighbor

approaches

the

number

of

occurrences

of

each

class

in

the

k

closest

points

to

the

query

are

counted

and

the

class

with

the

most

occurrences

-LRB-

or

votes

-RRB-

is

predicted

Distance

weighting

could

be

used

to

weight

the

votes

so

that

nearby

data

points

receive

more

votes

than

distant

points

A

second

way

to

incorporate

distance

weighting

in

classifier

training

is

to

incorporate

it

into

the

cost

criterion

that

is

being

minimized

by

training

-LRB-

Vapnik

Bottou

and

Vapnik

Vapnik

and

Bottou

-RRB-

C

-LRB-

q

-RRB-

X

i

-LSB-

L

-LRB-

ci

ctruei

-RRB-

K

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

-RSB-

-LRB-

-RRB-

C

is

the

cost

to

be

minimized

and

L

-LRB-

ci

ctruei

-RRB-

is

the

cost

of

predicting

class

ci

on

training

point

xi

when

the

true

class

is

ctruei

K

-LRB-

-RRB-

is

the

weighting

or

kernel

function

A

simple

version

of

this

approach

is

to

select

the

k

nearest

points

and

just

train

a

classifier

on

that

data

In

this

case

K

-LRB-

-RRB-

is

a

uniform

or

boxcar

kernel

The

form

of

the

classifier

is

not

constrained

in

any

way

Locally

weighted

learning

specifies

the

form

of

the

training

criterion

only

and

not

the

form

of

the

performance

algorithm

A

third

way

to

incorporate

distance

weighting

is

to

treat

classification

as

a

regression

problem

where

there

are

decision

functions

for

each

class

and

the

decision

function

with

the

largest

value

at

the

query

point

determines

the

class

of

the

query

Training

these

decision

functions

can

be

distance

weighted

as

well

C

-LRB-

q

-RRB-

X

i

@X

j

-LRB-

gj

-LRB-

xi

-RRB-

tij

-RRB-

AK

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

-LRB-

-RRB-

where

gj

-LRB-

-RRB-

is

the

decision

function

for

class

j

and

tij

is

the

target

for

decision

function

gj

-LRB-

-RRB-

on

training

point

i.

Hastie

and

Tibshirani

-LRB-

-RRB-

describe

an

approach

in

which

global

approaches

to

finding

discriminants

are

localized

by

locally

weighting

the

algorithm

directly

rather

than

the

criterion

In

this

paper

we

described

fitting

simple

linear

models

using

distance

weighted

fit

criterion

One

can

imagine

using

distance

weighted

criterion

to

train

linear

decision

functions

and

linear

discriminants

to

create

local

classifiers

It

is

also

possible

to

train

general

models

such

as

logistic

regression

to

perform

classification

in

a

locally

weighted

fashion

Requirements

for

Locally

Weighted

Learning

Locally

weighted

learning

has

several

requirements

ffl

Distance

function

Locally

weighted

learning

systems

require

a

measure

of

relevance

The

major

assumption

made

by

locally

weighted

learning

is

that

relevance

can

be

measured

using

a

measure

of

distance

Nearby

training

points

are

more

relevant

There

are

many

other

possible

measures

of

relevance

and

also

more

general

notions

of

similarity

The

distance

function

d

-LRB-

a

b

-RRB-

needs

to

input

two

objects

and

produce

a

number

The

distance

function

does

not

need

to

satisfy

the

formal

requirements

for

a

distance

metric

ffl

Separable

criterion

Locally

weighted

learning

systems

compute

a

weight

for

each

training

point

To

apply

this

weight

the

training

criterion

can

not

be

a

general

function

of

the

predictions

of

the

training

points

C

L

-LRB-

yn

yn

-RRB-

-LRB-

-RRB-

but

must

be

separable

in

some

way

We

use

additive

separability

C

X

i

-LSB-

L

-LRB-

yi

yi

-RRB-

K

-LRB-

d

-LRB-

xi

q

-RRB-

-RRB-

-RSB-

-LRB-

-RRB-

although

there

are

other

forms

of

separability

ffl

Enough

data

There

needs

to

be

enough

data

to

compute

statistics

which

is

also

true

of

other

statistical

learning

approaches

How

much

is

enough

We

have

run

robot

learning

experiments

where

performance

improvements

started

to

occur

with

on

the

order

of

ten

points

in

the

training

set

although

we

typically

collect

between

and

points

during

an

experiment

The

amount

of

training

data

needed

is

highly

problem

dependent

ffl

Labelled

data

Each

training

point

needs

to

have

an

associated

output

yi

For

classification

this

is

a

label

and

for

regression

-LRB-

function

approximation

-RRB-

it

is

a

number

ffl

Representations

Although

the

above

requirements

are

enough

for

a

system

using

nearest

neighbor

techniques

locally

weighted

regression

requires

that

each

object

produces

a

fixed

length

vector

of

the

values

-LRB-

symbolic

or

numeric

-RRB-

for

a

list

of

specified

features

x

BBBBBBBB

xn

CCCCCCCCA

-LRB-

-RRB-

However

more

general

representations

can

be

handled

by

locally

weighted

learning

approaches

For

example

a

more

general

training

criterion

is

C

X

i

fL

-LRB-

f

-LRB-

Xi

-RRB-

Yi

-RRB-

K

-LRB-

d

-LRB-

Xi

Q

-RRB-

-RRB-

g

-LRB-

-RRB-

The

inputs

Xi

outputs

Yi

and

query

Q

can

be

complex

objects

such

as

entire

semantic

networks

with

the

distance

functions

being

graph

matching

algorithms

or

graph

difference

measuring

algorithms

and

f

-LRB-

-RRB-

being

a

graph

transformation

with

as

adjustable

parameters

-LRB-

Elliot

and

Scott

-RRB-

Or

the

objects

can

be

text

computer

files

with

the

inputs

X

in

Japanese

and

the

outputs

Y

in

English

the

distance

functions

can

be

the

number

of

characters

in

the

output

of

a

file

difference

program

such

as

the

UNIX

diff

and

the

local

model

f

-LRB-

-RRB-

can

be

a

machine

translation

program

with

adjustable

parameters

Typical

parameters

for

an

expert

system

might

be

strengths

of

rules

so

changing

affects

which

rules

are

selected

for

application

The

input

space

distance

d

-LRB-

-RRB-

can

be

generalized

to

take

into

account

the

output

space

distance

between

the

output

values

of

the

training

data

and

a

predicted

output

C

X

i

L

-LRB-

f

-LRB-

Xi

-RRB-

Yi

-RRB-

K

B@d

B

B

Xi

Yi

CA

B

Q

Ypred

CA

CA

CA

-LRB-

-RRB-

This

is

useful

to

when

the

function

being

approximated

has

several

distinct

outputs

for

similar

inputs

Although

it

has

not

yet

been

extensively

explored

by

current

research

it

is

possible

for

locally

weighted

learning

systems

to

have

stored

objects

that

provide

separate

information

to

the

query

distance

function

-LRB-

Xi

-RRB-

and

to

the

local

model

-LRB-

Xi

-RRB-

-LRB-

Hammond

Callan

et

al.

Nguyen

et

al.

-RRB-

In

this

case

the

training

criterion

might

be

C

X

i

fL

-LRB-

f

-LRB-

Xi

-RRB-

Yi

-RRB-

K

-LRB-

d

-LRB-

Xi

Q

-RRB-

-RRB-

g

-LRB-

-RRB-

One

example

of

this

is

to

use

measures

of

volatility

of

the

stock

market

to

measure

distance

between

data

points

and

a

query

d

-LRB-

Xi

Q

-RRB-

but

use

price

histories

and

other

factors

to

form

local

-LRB-

with

respect

to

volatility

-RRB-

predictive

models

for

future

prices

f

-LRB-

Xi

-RRB-

-LRB-

LeBaron

-RRB-

Another

example

is

to

use

nationality

as

the

input

to

the

distance

function

-LRB-

requiring

a

distance

calculation

for

symbolic

values

-RRB-

and

to

use

numeric

features

such

as

age

height

weight

and

blood

pressure

to

build

a

locally

-LRB-

with

respect

to

the

nationality

distance

-RRB-

weighted

regression

to

predict

heart

attack

risk

Future

Research

Directions

Our

view

of

interesting

areas

of

future

research

include

ffl

Hybrid

Tuning

Algorithms

We

have

developed

independent

continuous

and

discrete

fit

parameter

optimization

techniques

It

is

clear

that

a

hybrid

approach

can

do

better

than

either

approach

alone

Parameters

could

initially

be

treated

as

discrete

and

then

more

and

more

continuous

optimization

could

be

performed

as

optimal

values

were

approached

for

example

Another

approach

is

for

the

racing

algorithms

to

allow

continuous

tuning

by

each

contestant

during

the

race

rather

than

racing

fixed

sets

of

parameters

ffl

New

forms

of

local

tuning

So

far

research

has

focused

on

locally

tuning

bandwidth

and

smoothing

parameters

More

work

needs

to

be

done

on

locally

tuning

distance

metrics

ridge

regression

parameters

outlier

thresholds

etc.

without

overfitting

ffl

Multiscale

local

tuning

One

dimensional

fit

parameters

such

as

bandwidth

and

model

order

can

be

locally

optimized

using

small

neighborhoods

Multidimensional

fit

parameters

such

as

the

distance

scale

parameters

in

a

distance

matrix

M

or

the

set

of

ridge

regression

parameters

need

much

larger

neighborhoods

and

different

kinds

of

regularization

to

be

tuned

locally

How

should

these

different

tuning

processes

interact

ffl

Stochastic

gradient

approaches

to

continuous

tuning

Continuous

optimization

based

on

estimates

of

the

gradient

using

small

numbers

of

random

queries

rather

than

exhaustive

query

sets

seems

a

promising

approach

to

efficient

tuning

algorithms

-LRB-

Moore

and

Schneider

-RRB-

ffl

Properties

of

massive

cross-validation

We

have

discussed

the

use

of

crossvalidation

and

why

locally

weighted

learning

is

particularly

well

suited

to

its

use

Better

understanding

of

how

much

cross

validation

can

take

place

before

it

is

in

danger

of

overfitting

-LRB-

which

must

be

guarded

by

an

extra

level

of

cross-validation

-RRB-

would

be

desirable

ffl

Probabilistic

approaches

We

would

like

to

explore

further

the

analogies

between

locally

weighted

learning

and

probabilistic

models

including

Bayesian

models

-LRB-

Rachlin

et

al.

Ting

and

Cameron-Jones

-RRB-

ffl

Forgetting

So

far

forgetting

has

not

played

an

important

role

in

our

implementations

of

robot

learning

as

we

have

not

run

out

of

memory

However

we

expect

forgetting

to

play

a

more

important

role

in

the

future

and

expect

it

to

be

necessary

to

implement

a

principled

approach

to

storage

control

ffl

Computational

Techniques

For

enormous

dataset

sizes

new

data

management

algorithms

may

be

needed

They

include

principled

ways

to

forget

or

coalesce

old

data

compactly

represent

high

dimensional

data

clouds

ways

of

using

samples

of

datasets

instead

of

entire

datasets

and

in

the

case

of

multi-gigabyte

datasets

hardware

and

software

techniques

for

managing

data

on

secondary

storage

ffl

Less

Lazy

Learning

This

review

has

focussed

on

a

pure

form

of

lazy

learning

in

which

only

the

data

is

stored

between

queries

This

purist

approach

will

be

too

extreme

in

some

circumstances

and

most

tuning

algorithms

for

fit

parameters

store

the

optimized

fit

parameters

in

between

queries

Substantial

amounts

of

data

compression

can

be

achieved

by

building

a

set

of

local

models

at

fixed

locations

using

the

techniques

described

in

this

paper

In

addition

to

computational

speedup

in

the

presence

of

large

datasets

there

may

be

statistical

advantages

to

compressing

data

instead

of

merely

storing

it

all

-LRB-

Fritzke

Schaal

and

Atkeson

-RRB-

Summary

This

paper

has

surveyed

locally

weighted

learning

Local

weighting

whether

by

weighting

the

data

or

the

error

criterion

can

turn

global

function

approximation

into

powerful

alternative

approaches

By

means

of

local

weighting

unnecessary

bias

of

global

function

fitting

is

reduced

higher

flexibility

is

obtained

but

desirable

properties

like

smoothness

and

statistical

analyzability

are

retained

We

have

concentrated

on

how

linear

regression

behaves

under

local

weighting

and

surveyed

the

ways

in

which

tools

from

conventional

regression

analysis

in

global

regression

can

be

used

in

locally

weighted

regression

A

major

question

has

concerned

the

notion

of

locality

what

is

a

good

choice

of

distance

metric

how

close

within

that

metric

should

points

be

and

how

can

these

decisions

be

automatically

made

from

the

data

The

field

of

local

learning

is

of

large

interest

in

the

statistics

community

and

we

have

provided

entry

points

into

that

literature

Locally

weighted

learning

is

also

rapidly

increasing

in

popularity

in

the

machine

learning

community

and

the

outlook

is

promising

for

interesting

statistical

computational

and

application-oriented

development

Acknowledgments

Support

for

C.

Atkeson

and

S.

Schaal

was

provided

by

the

ATR

Human

Information

Processing

Research

Laboratories

Support

for

C.

Atkeson

was

provided

under

Air

Force

Office

of

Scientific

Research

grant

and

by

a

National

Science

Foundation

Presidential

Young

Investigator

Award

Support

for

S.

Schaal

was

provided

by

the

German

Scholarship

Foundation

and

the

Alexander

von

Humboldt

Foundation

Support

for

A.

Moore

was

provided

by

the

U.

K.

Science

and

Engineering

Research

Council

NSF

Research

Initiation

Award

and

a

Research

Gift

from

the

Corporation

References

-LRB-

-RRB-

Ninth

National

Conference

on

Artificial

Intelligence

AAAI

Press/The

MIT

Press

Cambridge

MA

Aha

D.

W.

-LRB-

-RRB-

Incremental

instance-based

learning

of

independent

and

graded

concept

descriptions

In

Sixth

International

Machine

Learning

Workshop

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Aha

D.

W.

-LRB-

-RRB-

A

Study

of

Instance-Based

Algorithms

for

Supervised

Learning

Tasks

Mathematical

Empirical

and

Psychological

Observations

PhD

dissertation

University

of

California

Irvine

Department

of

Information

and

Computer

Science

Aha

D.

W.

-LRB-

-RRB-

Incremental

constructive

induction

An

instance-based

approach

In

Eighth

International

Machine

Learning

Workshop

Morgan

Kaufmann

San

Mateo

CA

Aha

D.

W.

and

Goldstone

R.

L.

-LRB-

-RRB-

Learning

attribute

relevance

in

context

in

instance-based

learning

algorithms

In

Annual

Conference

of

the

Cognitive

Science

Society

pages

-LCB-

Aha

D.

W.

and

Goldstone

R.

L.

-LRB-

-RRB-

Concept

learning

and

flexible

weighting

In

Annual

Conference

of

the

Cognitive

Science

Society

pages

-LCB-

Bloomington

IL

Lawrence

Erlbaum

Associates

Mahwah

NJ

Aha

D.

W.

and

Kibler

D.

-LRB-

-RRB-

Noise-tolerant

instance-based

learning

algorithms

In

Eleventh

International

Joint

Conference

on

Artificial

Intelligence

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Aha

D.

W.

and

McNulty

D.

M.

-LRB-

-RRB-

Learning

relative

attribute

weights

for

instance-based

concept

descriptions

In

Annual

Conference

of

the

Cognitive

Science

Society

pages

-LCB-

Lawrence

Erlbaum

Associates

Mahwah

NJ

Altman

N.

S.

-LRB-

-RRB-

An

introduction

to

kernel

and

nearest-neighbor

nonparametric

regression

The

American

Statistician

-LRB-

-RRB-

-LCB-

Atkeson

C.

G.

-LRB-

-RRB-

Using

local

models

to

control

movement

In

Touretzky

D.

S.

editor

Advances

In

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufman

San

Mateo

CA

Atkeson

C.

G.

-LRB-

-RRB-

Memory-based

approaches

to

approximating

continuous

functions

In

Casdagli

and

Eubank

-LRB-

-RRB-

pages

-LCB-

Proceedings

of

a

Workshop

on

Nonlinear

Modeling

and

Forecasting

September

Santa

Fe

New

Mexico

Atkeson

C.

G.

-LRB-

-RRB-

Local

learning

Atkeson

C.

G.

Moore

A.

W.

and

Schaal

S.

-LRB-

-RRB-

Locally

weighted

learning

for

control

Artificial

Intelligence

Review

in

press

Atkeson

C.

G.

and

Reinkensmeyer

D.

J.

-LRB-

-RRB-

Using

associative

content-addressable

memories

to

control

robots

In

Proceedings

of

the

IEEE

Conference

on

Decision

and

Control

volume

pages

-LCB-

Austin

Texas

IEEE

Cat

Atkeson

C.

G.

and

Reinkensmeyer

D.

J.

-LRB-

-RRB-

Using

associative

content-addressable

memories

to

control

robots

In

Proceedings

IEEE

International

Conference

on

Robotics

and

Automation

Scottsdale

Arizona

Atkeson

C.

G.

and

Schaal

S.

-LRB-

-RRB-

Memory-based

neural

networks

for

robot

learning

Neurocomputing

-LCB-

Barnhill

R.

E.

-LRB-

-RRB-

Representation

and

approximation

of

surfaces

In

Rice

J.

R.

editor

Mathematical

Software

III

pages

-LCB-

Academic

Press

New

York

NY

Batchelor

B.

G.

-LRB-

-RRB-

Practical

Approach

To

Pattern

Classification

Plenum

Press

New

York

NY

Benedetti

J.

K.

-LRB-

-RRB-

On

the

nonparametric

estimation

of

regression

functions

Journal

of

the

Royal

Statistical

Society

Series

B

-LCB-

Bentley

J.

L.

-LRB-

-RRB-

Multidimensional

binary

search

trees

used

for

associative

searching

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

Bentley

J.

L.

and

Friedman

J.

H.

-LRB-

-RRB-

Data

structures

for

range

searching

ACM

Comput

Surv.

-LRB-

-RRB-

-LCB-

Bentley

J.

L.

Weide

B.

and

Yao

A.

-LRB-

-RRB-

Optimal

expected

time

algorithms

for

closest

point

problems

ACM

Transactions

on

Mathematical

Software

-LCB-

Blyth

S.

-LRB-

-RRB-

Optimal

kernel

weights

under

a

power

criterion

Journal

of

the

American

Statistical

Association

-LRB-

-RRB-

-LCB-

Bottou

L.

and

Vapnik

V.

-LRB-

-RRB-

Local

learning

algorithms

Neural

Computation

-LRB-

-RRB-

-LCB-

Bregler

C.

and

Omohundro

S.

M.

-LRB-

-RRB-

Surface

learning

with

applications

to

lipreading

In

Cowan

et

al.

-LRB-

-RRB-

pages

-LCB-

Brockmann

M.

Gasser

T.

and

Herrmann

E.

-LRB-

-RRB-

Locally

adaptive

bandwidth

choice

for

kernel

regression

estimators

Journal

of

the

American

Statistical

Association

-LRB-

-RRB-

-LCB-

Broder

A.

J.

-LRB-

-RRB-

Strategies

for

efficient

incremental

nearest

neighbor

search

Pattern

Recognition

-LCB-

Callan

J.

P.

Fawcett

T.

E.

and

Rissland

E.

L.

-LRB-

-RRB-

CABOT

An

adaptive

approach

to

case

based

search

In

IJCAI

-LRB-

-RRB-

pages

-LCB-

Casdagli

M.

and

Eubank

S.

editors

-LRB-

-RRB-

Nonlinear

Modeling

and

Forecasting

Proceedings

Volume

XII

in

the

Santa

Fe

Institute

Studies

in

the

Sciences

of

Complexity

Addison

Wesley

New

York

NY

Proceedings

of

a

Workshop

on

Nonlinear

Modeling

and

Forecasting

September

Santa

Fe

New

Mexico

Cheng

P.

E.

-LRB-

-RRB-

Strong

consistency

of

nearest

neighbor

regression

function

estimators

Journal

of

Multivariate

Analysis

-LCB-

Cleveland

W.

S.

-LRB-

-RRB-

Robust

locally

weighted

regression

and

smoothing

scatterplots

Journal

of

the

American

Statistical

Association

-LCB-

Cleveland

W.

S.

-LRB-

-RRB-

Coplots

nonparametric

regression

and

conditionally

parametric

fits

Technical

Report

AT&T

Bell

Laboratories

Statistics

Department

Murray

Hill

NJ

Cleveland

W.

S.

-LRB-

-RRB-

Visualizing

Data

Hobart

Press

Summit

NJ

books@hobart.com

Cleveland

W.

S.

and

Devlin

S.

J.

-LRB-

-RRB-

Locally

weighted

regression

An

approach

to

regression

analysis

by

local

fitting

Journal

of

the

American

Statistical

Association

-LCB-

Cleveland

W.

S.

Devlin

S.

J.

and

Grosse

E.

-LRB-

-RRB-

Regression

by

local

fitting

Methods

properties

and

computational

algorithms

Journal

of

Econometrics

-LCB-

Cleveland

W.

S.

and

Grosse

E.

-LRB-

-RRB-

Computational

methods

for

local

regression

Statistics

and

Computing

-LRB-

-RRB-

-LCB-

ftp

Cleveland

W.

S.

Grosse

E.

and

Shyu

W.

M.

-LRB-

-RRB-

Local

regression

models

In

Chambers

J.

M.

and

Hastie

T.

J.

editors

Statistical

Models

in

S

pages

-LCB-

Wadsworth

Pacific

Grove

CA

Cleveland

W.

S.

and

Loader

C.

-LRB-

-RRB-

Computational

methods

for

local

regression

Technical

Report

AT&T

Bell

Laboratories

Statistics

Department

Murray

Hill

NJ

Cleveland

W.

S.

and

Loader

C.

-LRB-

-RRB-

Local

fitting

for

semiparametric

-LRB-

nonparametric

-RRB-

regression

Comments

on

a

paper

of

Fan

and

Marron

Technical

Report

AT&T

Bell

Laboratories

Statistics

Department

Murray

Hill

NJ

earlier

version

is

Cleveland

W.

S.

and

Loader

C.

-LRB-

-RRB-

Smoothing

by

local

regression

Principles

and

methods

Technical

Report

AT&T

Bell

Laboratories

Statistics

Department

Murray

Hill

NJ

Cleveland

W.

S.

Mallows

C.

L.

and

McRae

J.

E.

-LRB-

-RRB-

ATS

methods

Nonparametric

regression

for

non-Gaussian

data

Journal

of

the

American

Statistical

Association

-LRB-

-RRB-

-LCB-

Connell

M.

E.

and

Utgoff

P.

E.

-LRB-

-RRB-

Learning

to

control

a

dynamic

physical

system

In

Sixth

National

Conference

on

Artificial

Intelligence

pages

-LCB-

Seattle

WA

Morgan

Kaufmann

San

Mateo

CA

Cost

S.

and

Salzberg

S.

-LRB-

-RRB-

A

weighted

nearest

neighbor

algorithm

for

learning

with

symbolic

features

Machine

Learning

-LRB-

-RRB-

-LCB-

Coughran

Jr.

W.

M.

and

Grosse

E.

-LRB-

-RRB-

Seeing

and

hearing

dynamic

loess

surfaces

In

Interface

Proceedings

pages

-LCB-

Springer-Verlag

ftp

or

in

Proceedings

Interface

Cowan

J.

D.

Tesauro

G.

and

Alspector

J.

editors

-LRB-

-RRB-

Advances

In

Neural

Information

Processing

Systems

Morgan

Kaufman

San

Mateo

CA

Crain

I.

K.

and

Bhattacharyya

B.

K.

-LRB-

-RRB-

Treatment

of

nonequispaced

two

dimensional

data

with

a

digital

computer

Geoexploration

-LCB-

Deheuvels

P.

-LRB-

-RRB-

Estimation

non-param

etrique

del

la

densit?e

par

histogrammes

g?en?eralis?es

Revue

Statistique

Appliqu?e

-LCB-

Deng

K.

and

Moore

A.

W.

-LRB-

-RRB-

Multiresolution

instance-based

learning

In

Fourteenth

International

Joint

Conference

on

Artificial

Intelligence

Morgan

Kaufmann

San

Mateo

CA

Dennis

J.

E.

Gay

D.

M.

and

Welsch

R.

E.

-LRB-

-RRB-

An

adaptive

nonlinear

least-squares

algorithm

ACM

Transactions

on

Mathematical

Software

-LRB-

-RRB-

-LCB-

Devroye

L.

-LRB-

-RRB-

On

the

almost

everywhere

convergence

of

nonparametric

regression

function

estimates

The

Annals

of

Statistics

-LRB-

-RRB-

-LCB-

Diebold

F.

X.

and

Nason

J.

A.

-LRB-

-RRB-

Nonparametric

exchange

rate

prediction

Journal

of

International

Economics

-LCB-

Dietterich

T.

G.

Wettschereck

D.

Atkeson

C.

G.

and

Moore

A.

W.

-LRB-

-RRB-

Memory-based

methods

for

regression

and

classification

In

Cowan

et

al.

-LRB-

-RRB-

pages

-LCB-

Draper

N.

R.

and

Smith

H.

-LRB-

-RRB-

Applied

Regression

Analysis

John

Wiley

New

York

NY

edition

Elliot

T.

and

Scott

P.

D.

-LRB-

-RRB-

Instance-based

and

generalization-based

learning

procedures

applied

to

solving

integration

problems

In

Proceedings

of

the

Eighth

Conference

of

the

Society

for

the

Study

of

Artificial

Intelligence

pages

-LCB-

Leeds

England

Springer

Verlag

Epanechnikov

V.

A.

-LRB-

-RRB-

Nonparametric

estimation

of

a

multivariate

probability

density

Theory

of

Probability

and

Its

Applications

-LCB-

Eubank

R.

L.

-LRB-

-RRB-

Spline

Smoothing

and

Nonparametric

Regression

Marcel

Dekker

New

York

NY

Falconer

K.

J.

-LRB-

-RRB-

A

general

purpose

algorithm

for

contouring

over

scattered

data

points

Technical

Report

NAC

National

Physical

Laboratory

Teddington

Middlesex

United

Kingdon

Fan

J.

-LRB-

-RRB-

Design-adaptive

nonparametric

regression

Journal

of

the

American

Statistical

Association

-LRB-

-RRB-

-LCB-

Fan

J.

-LRB-

-RRB-

Local

linear

regression

smoothers

and

their

minimax

efficiencies

Annals

of

Statistics

-LCB-

Fan

J.

-LRB-

-RRB-

Local

modeling

EES

Update

written

for

the

Encyclopedia

of

Statistics

Science

Fan

J.

and

Gijbels

I.

-LRB-

-RRB-

Variable

bandwidth

and

local

linear

regression

smoothers

The

Annals

of

Statistics

-LRB-

-RRB-

-LCB-

Fan

J.

and

Gijbels

I.

-LRB-

-RRB-

Censored

regression

Local

linear

approximations

and

their

applications

Journal

of

the

American

Statistical

Association

-LCB-

Fan

J.

and

Gijbels

I.

-LRB-

-RRB-

Adaptive

order

polynomial

fitting

Bandwidth

robustification

and

bias

reduction

Fan

J.

and

Gijbels

I.

-LRB-

-RRB-

Data-driven

bandwidth

selection

in

local

polynomial

fitting

Variable

bandwidth

and

spatial

adaptation

Journal

of

the

Royal

Statistical

Society

B

-LCB-

Fan

J.

and

Gijbels

I.

-LRB-

-RRB-

Local

Polynomial

Modeling

and

its

Applications

Chapman

and

Hall

London

Fan

J.

and

Hall

P.

-LRB-

-RRB-

On

curve

estimation

by

minimizing

mean

absolute

deviation

and

its

implications

The

Annals

of

Statistics

-LRB-

-RRB-

-LCB-

Fan

J.

and

Kreutzberger

E.

-LRB-

-RRB-

Automatic

local

smoothing

for

spectral

densisty

estimation

ftp

Fan

J.

and

Marron

J.

S.

-LRB-

-RRB-

Comment

on

-LSB-

Hastie

and

Loader

-RSB-

Statistical

Science

-LRB-

-RRB-

-LCB-

Fan

J.

and

Marron

J.

S.

-LRB-

-RRB-

Fast

implementations

of

nonparametric

curve

estimators

Journal

of

Computational

and

Statistical

Graphics

-LCB-

Fan

J.

and

Marron

J.

S.

-LRB-

-RRB-

Rejoinder

to

discussion

of

Cleveland

and

Loader

Farmer

J.

D.

and

Sidorowich

J.

J.

-LRB-

-RRB-

Predicting

chaotic

time

series

Physical

Review

Letters

-LRB-

-RRB-

-LCB-

Farmer

J.

D.

and

Sidorowich

J.

J.

-LRB-

-RRB-

Exploiting

chaos

to

predict

the

future

and

reduce

noise

In

Lee

Y.

C.

editor

Evolution

Learning

and

Cognition

pages

-LCB-

World

Scientific

Press

NJ

also

available

as

Technical

Report

Los

Alamos

National

Laboratory

Los

Alamos

New

Mexico

Farmer

J.

D.

and

Sidorowich

J.

J.

-LRB-

-RRB-

Predicting

chaotic

dynamics

In

Kelso

J.

A.

S.

Mandell

A.

J.

and

Schlesinger

M.

F.

editors

Dynamic

Patterns

in

Complex

Systems

pages

-LCB-

World

Scientific

NJ

Farwig

R.

-LRB-

-RRB-

Multivariate

interpolation

of

scattered

data

by

moving

least

squares

methods

In

Mason

J.

C.

and

Cox

M.

G.

editors

Algorithms

for

Approximation

pages

-LCB-

Clarendon

Press

Oxford

Fedorov

V.

V.

Hackl

P.

and

Muller

W.

G.

-LRB-

-RRB-

Moving

local

regression

The

weight

function

Nonparametric

Statistics

-LRB-

-RRB-

-LCB-

Franke

R.

and

Nielson

G.

-LRB-

-RRB-

Smooth

interpolation

of

large

sets

of

scattered

data

International

Journal

for

Numerical

Methods

in

Engineering

-LCB-

Friedman

J.

H.

-LRB-

-RRB-

A

variable

span

smoother

Technical

Report

LCS

Stanford

University

Statistics

Department

Stanford

CA

Friedman

J.

H.

-LRB-

-RRB-

Flexible

metric

nearest

neighbor

classification

Friedman

J.

H.

Bentley

J.

L.

and

Finkel

R.

A.

-LRB-

-RRB-

An

algorithm

for

finding

best

matches

in

logarithmic

expected

time

ACM

Transactions

on

Mathematical

Software

-LRB-

-RRB-

-LCB-

Fritzke

B.

-LRB-

-RRB-

Incremental

learning

of

local

linear

mappings

In

Proceedings

of

the

International

Conference

on

Artificial

Neural

Networks

ICANN

pages

-LCB-

Paris

France

Fukunaga

K.

-LRB-

-RRB-

Introduction

to

Statistical

Pattern

Recognition

Academic

Press

New

York

NY

second

edition

Gasser

T.

and

Muller

H.

G.

-LRB-

-RRB-

Kernel

estimation

of

regression

functions

In

Gasser

T.

and

Rosenblatt

M.

editors

Smoothing

Techniques

for

Curve

Estimation

number

in

Lecture

Notes

in

Mathematics

pages

-LCB-

Springer-Verlag

Heidelberg

Gasser

T.

and

Muller

H.

G.

-LRB-

-RRB-

Estimating

regression

functions

and

their

derivatives

by

the

kernel

method

Scandanavian

Journal

of

Statistics

-LCB-

Gasser

T.

Muller

H.

G.

and

Mammitzsch

V.

-LRB-

-RRB-

Kernels

for

nonparametric

regression

Journal

of

the

Royal

Statistical

Society

Series

B

-LCB-

Ge

Z.

Cavinato

A.

G.

and

Callis

J.

B.

-LRB-

-RRB-

Noninvasive

spectroscopy

for

monitoring

cell

density

in

a

fermentation

process

Analytical

Chemistry

-LCB-

Goldberg

K.

Y.

and

Pearlmutter

B.

-LRB-

-RRB-

Using

a

neural

network

to

learn

the

dynamics

of

the

CMU

Direct-Drive

Arm

II

Technical

Report

Carnegie-Mellon

University

Pittsburgh

PA.

Gorinevsky

D.

and

Connolly

T.

H.

-LRB-

-RRB-

Comparison

of

some

neural

network

and

scattered

data

approximations

The

inverse

manipulator

kinematics

example

Neural

Computation

-LCB-

Goshtasby

A.

-LRB-

-RRB-

Image

registration

by

local

approximation

methods

Image

and

Vision

Computing

-LRB-

-RRB-

-LCB-

Grosse

E.

-LRB-

-RRB-

LOESS

Multivariate

smoothing

by

moving

least

squares

In

Chui

C.

K.

Schumaker

L.

L.

and

Ward

J.

D.

editors

Approximation

Theory

VI

pages

-LCB-

Academic

Press

Boston

MA

Hammond

S.

V.

-LRB-

-RRB-

Nir

analysis

of

antibiotic

fermentations

In

Murray

I.

and

Cowe

I.

A.

editors

Making

Light

Work

Advances

in

Near

Infrared

Spectroscopy

pages

-LCB-

VCH

New

York

NY

Developed

from

the

International

Conference

on

Near

Infrared

Spectroscopy

Aberdeen

Scotland

August

Hampel

F.

R.

Ronchetti

E.

M.

Rousseeuw

P.

J.

and

Stahel

W.

A.

-LRB-

-RRB-

Robust

Statistics

The

Approach

Based

On

Influence

Functions

John

Wiley

New

York

NY

Hardle

W.

-LRB-

-RRB-

Applied

Nonparametric

Regression

Cambridge

University

Press

New

York

NY

Hastie

T.

and

Loader

C.

-LRB-

-RRB-

Local

regression

Automatic

kernel

carpentry

Statistical

Science

-LRB-

-RRB-

-LCB-

Hastie

T.

J.

and

Tibshirani

R.

J.

-LRB-

-RRB-

Generalized

Additive

Regression

Chapman

Hall

London

Hastie

T.

J.

and

Tibshirani

R.

J.

-LRB-

-RRB-

Discriminant

adaptive

nearest

neighbor

classification

ftp

Z.

Higuchi

T.

Kitano

H.

Furuya

T.

ichi

Handa

K.

Takahashi

N.

and

Kokubu

A.

-LRB-

-RRB-

A

parallel

associative

processor

for

knowledge

processing

In

-LRB-

-RRB-

pages

-LCB-

Hillis

D.

-LRB-

-RRB-

The

Connection

Machine

MIT

Press

Cambridge

MA

IJCAI

-LRB-

-RRB-

Twelfth

International

Joint

Conference

on

Artificial

Intelligence

Morgan

Kaufmann

San

Mateo

CA

IJCAI

-LRB-

-RRB-

Thirteenth

International

Joint

Conference

on

Artificial

Intelligence

Morgan

Kaufmann

San

Mateo

CA

Jabbour

K.

Riveros

J.

F.

W.

Landsbergen

D.

and

Meyer

W.

-LRB-

-RRB-

ALFA

Automated

load

forecasting

assistant

In

Proceedings

of

the

IEEE

Power

Engineering

Society

Summer

Meeting

San

Francisco

CA

James

M.

-LRB-

-RRB-

Classification

Algorithms

John

Wiley

and

Sons

New

York

NY

Jones

M.

C.

Davies

S.

J.

and

Park

B.

U.

-LRB-

-RRB-

Versions

of

kernel-type

regression

estimators

Journal

of

the

American

Statistical

Association

-LRB-

-RRB-

-LCB-

Karali

c

A.

-LRB-

-RRB-

Employing

linear

regression

in

regression

tree

leaves

In

Neumann

B.

editor

ECAI

European

Conference

on

Artificial

Intelligence

pages

-LCB-

Vienna

Austria

John

Wiley

and

Sons

Katkovnik

V.

Y.

-LRB-

-RRB-

Linear

and

nonlinear

methods

of

nonparametric

regression

analysis

Soviet

Automatic

Control

-LCB-

Kazmierczak

H.

and

Steinbuch

K.

-LRB-

-RRB-

Adaptive

systems

in

pattern

recognition

IEEE

Transactions

on

Electronic

Computers

-LCB-

Kibler

D.

Aha

D.

W.

and

Albert

M.

-LRB-

-RRB-

Instance-based

prediction

of

real-valued

attributes

Computational

Intelligence

-LCB-

Kitano

H.

-LRB-

-RRB-

Challenges

of

massive

parallelism

In

IJCAI

-LRB-

-RRB-

pages

-LCB-

Kitano

H.

-LRB-

-RRB-

A

comprehensive

and

practical

model

of

memory-based

machine

translation

In

IJCAI

-LRB-

-RRB-

pages

-LCB-

Kitano

H.

and

Higuchi

T.

-LRB-

-RRB-

High

performance

memory-based

translation

on

massively

parallel

associative

memory

processor

In

-LRB-

-RRB-

pages

-LCB-

Kitano

H.

and

Higuchi

T.

-LRB-

-RRB-

Massively

parallel

memory-based

parsing

In

IJCAI

-LRB-

-RRB-

pages

-LCB-

Kitano

H.

Moldovan

D.

and

Cha

S.

-LRB-

-RRB-

High

performance

natural

language

processing

on

semantic

network

array

processor

In

IJCAI

-LRB-

-RRB-

pages

-LCB-

Kozek

A.

S.

-LRB-

-RRB-

A

new

nonparametric

estimation

method

Local

and

nonlinear

Interface

-LCB-

Lancaster

P.

-LRB-

-RRB-

Moving

weighted

least-squares

methods

In

Sahney

B.

N.

editor

Polynomial

and

Spline

Approximation

pages

-LCB-

D.

Reidel

Publishing

Boston

MA

Lancaster

P.

and

Salkauskas

K.

-LRB-

-RRB-

Surfaces

generated

by

moving

least

squares

methods

Mathematics

of

Computation

-LRB-

-RRB-

-LCB-

Lancaster

P.

and

Salkauskas

K.

-LRB-

-RRB-

Curve

And

Surface

Fitting

Academic

Press

New

York

NY

Lawrence

S.

Tsoi

A.

C.

and

Black

A.

D.

-LRB-

-RRB-

Function

approximation

with

neural

networks

and

local

methods

Bias

variance

and

smoothness

In

Australian

Conference

on

Neural

Networks

Canberra

Australia

Canberra

Australia

available

from

and

LeBaron

B.

-LRB-

-RRB-

Forecast

improvements

using

a

volatility

index

unpublished

LeBaron

B.

-LRB-

-RRB-

Nonlinear

forecasts

for

the

S&P

stock

index

In

Casdagli

and

Eubank

-LRB-

-RRB-

pages

-LCB-

Proceedings

of

a

Workshop

on

Nonlinear

Modeling

and

Forecasting

September

Santa

Fe

New

Mexico

Legg

M.

P.

C.

and

Brent

R.

P.

-LRB-

-RRB-

Automatic

contouring

In

Australian

Computer

Conference

pages

-LCB-

Lejeune

M.

-LRB-

-RRB-

Optimization

in

non-parametric

regression

In

COMPSTAT

Proceedings

in

Computational

Statistics

pages

-LCB-

Prague

Physica-Verlag

Wien

Lejeune

M.

-LRB-

-RRB-

Estimation

non-param

etrique

par

noyaux

R?egression

polyn

omial

mobile

Revue

de

Statistique

Appliqu?ee

-LRB-

-RRB-

-LCB-

Lejeune

M.

and

Sarda

P.

-LRB-

-RRB-

Smooth

estimators

of

distribution

and

density

functions

Computational

Statistics

Data

Analysis

-LCB-

Li

K.

C.

-LRB-

-RRB-

Consistency

for

cross-validated

nearest

neighbor

estimates

in

nonparametric

regression

The

Annals

of

Statistics

-LCB-

Loader

C.

-LRB-

-RRB-

Computing

nonparametric

function

estimates

Technical

Report

AT&T

Bell

Laboratories

Statistics

Department

Murray

Hill

NJ

Available

by

anonymous

FTP

from

netlib.att.com

in

Lodwick

G.

D.

and

Whittle

J.

-LRB-

-RRB-

A

technique

for

automatic

contouring

field

survey

data

Australian

Computer

Journal

-LCB-

Lowe

D.

G.

-LRB-

-RRB-

Similarity

metric

learning

for

a

variable-kernel

classifier

Neural

Computation

-LCB-

Maron

O.

and

Moore

A.

W.

-LRB-

-RRB-

A

racing

algorithm

Model

selection

for

memory

based

learners

Artificial

Intelligence

Review

in

press

Marron

J.

S.

-LRB-

-RRB-

Automatic

smoothing

parameter

selection

A

survey

Empirical

Economics

-LCB-

McIntyre

D.

B.

Pollard

D.

D.

and

Smith

R.

-LRB-

-RRB-

Computer

programs

for

automatic

contouring

Technical

Report

Kansas

Geological

Survey

Computer

Contributions

University

of

Kansas

Lawrence

KA

McLain

D.

H.

-LRB-

-RRB-

Drawing

contours

from

arbitrary

data

points

The

Computer

Journal

-LRB-

-RRB-

-LCB-

Medin

D.

L.

and

Shoben

E.

J.

-LRB-

-RRB-

Context

and

structure

in

conceptual

combination

Cognitive

Psychology

-LCB-

Meese

R.

and

Wallace

N.

-LRB-

-RRB-

Nonparametric

estimation

of

dynamic

hedonic

price

models

and

the

construction

of

residential

housing

price

indices

American

Real

Estate

and

Urban

Economics

Association

Journal

-LRB-

-RRB-

-LCB-

Meese

R.

A.

and

Rose

A.

K.

-LRB-

-RRB-

Nonlinear

nonparametric

nonessential

exchange

rate

estimation

The

American

Economic

Review

May

-LCB-

Miller

A.

J.

-LRB-

-RRB-

Subset

Selection

in

Regression

Chapman

and

Hall

London

Miller

W.

T.

Glanz

F.

H.

and

Kraft

L.

G.

-LRB-

-RRB-

Application

of

a

general

learning

algorithm

to

the

control

of

robotic

manipulators

International

Journal

of

Robotics

Research

-LCB-

Mohri

T.

and

Tanaka

H.

-LRB-

-RRB-

An

optimal

weighting

criterion

of

case

indexing

for

both

numeric

and

symbolic

attributes

In

Aha

D.

W.

editor

Workshop

Program

Case-Based

Reasoning

Working

Notes

pages

-LCB-

AAAI

Press

Seattle

WA

Moore

A.

W.

-LRB-

-RRB-

Acquisition

of

Dynamic

Control

Knowledge

for

a

Robotic

Manipulator

In

Seventh

International

Machine

Learning

Workshop

Morgan

Kaufmann

San

Mateo

CA

Moore

A.

W.

-LRB-

-RRB-

Efficient

Memory-based

Learning

for

Robot

Control

PhD

Thesis

Technical

Report

No.

Computer

Laboratory

University

of

Cambridge

Moore

A.

W.

Hill

D.

J.

and

Johnson

M.

P.

-LRB-

-RRB-

An

empirical

investigation

of

brute

force

to

choose

features

smoothers

and

function

approximators

In

Hanson

S.

Judd

S.

and

Petsche

T.

editors

Computational

Learning

Theory

and

Natural

Learning

Systems

volume

MIT

Press

Cambridge

MA

Moore

A.

W.

and

Schneider

J.

-LRB-

-RRB-

Memory-based

stochastic

optimization

To

appear

in

the

proceedings

of

Also

available

as

Technical

Report

ftp

och.ps

More

J.

J.

Garbow

B.

S.

and

Hillstrom

K.

E.

-LRB-

-RRB-

User

guide

for

Technical

Report

Argonne

National

Laboratory

Argonne

Illinois

Muller

H.-G

-LRB-

-RRB-

Weighted

local

regression

and

kernel

methods

for

nonparametric

curve

fitting

Journal

of

the

American

Statistical

Association

-LCB-

Muller

H.-G

-LRB-

-RRB-

Comment

on

-LSB-

Hastie

and

Loader

-RSB-

Statistical

Science

-LRB-

-RRB-

-LCB-

Murphy

O.

J.

and

Selkow

S.

M.

-LRB-

-RRB-

The

efficiency

of

using

k-d

trees

for

finding

nearest

neighbors

in

discrete

space

Information

Processing

Letters

-LCB-

Myers

R.

H.

-LRB-

-RRB-

Classical

and

Modern

Regression

With

Applications

PWS-KENT

Boston

MA

Nadaraya

E.

A.

-LRB-

-RRB-

On

estimating

regression

Theory

of

Probability

and

Its

Applications

-LCB-

Naes

T.

and

Isaksson

T.

-LRB-

-RRB-

Locally

weighted

regression

in

diffuse

near-infrared

transmittance

spectroscopy

Applied

Spectroscopy

-LRB-

-RRB-

-LCB-

Naes

T.

Isaksson

T.

and

Kowalski

B.

R.

-LRB-

-RRB-

Locally

weighted

regression

and

scatter

correction

for

near-infrared

reflectance

data

Analytical

Chemistry

-LRB-

-RRB-

-LCB-

Nguyen

T.

Czerwinsksi

M.

and

Lee

D.

-LRB-

-RRB-

COMPAQ

Quicksource

Providing

the

consumer

with

the

power

of

artificial

intelligence

In

Proceedings

of

the

Fifth

Annual

Conference

on

Innovative

Applications

of

Artificial

Intelligence

pages

-LCB-

Washington

DC

AAAI

Press

Nosofsky

R.

M.

Clark

S.

E.

and

Shin

H.

J.

-LRB-

-RRB-

Rules

and

exemplars

in

categorization

identification

and

recognition

Journal

of

Experimental

Psychology

Learning

Memory

and

Cognition

-LCB-

Omohundro

S.

M.

-LRB-

-RRB-

Efficient

Algorithms

with

Neural

Network

Behaviour

Journal

of

Complex

Systems

-LRB-

-RRB-

-LCB-

Omohundro

S.

M.

-LRB-

-RRB-

Bumptrees

for

Efficient

Function

Constraint

and

Classification

Learning

In

Lippmann

R.

P.

Moody

J.

E.

and

Touretzky

D.

S.

editors

Advances

in

Neural

Information

Processing

Systems

Morgan

Kaufmann

Palmer

J.

A.

B.

-LRB-

-RRB-

Automatic

mapping

In

Australian

Computer

Conference

pages

-LCB-

Pelto

C.

R.

Elkins

T.

A.

and

Boyd

H.

A.

-LRB-

-RRB-

Automatic

contouring

of

irregularly

spaced

data

Geophysics

-LCB-

Press

W.

H.

Teukolsky

S.

A.

Vetterling

W.

T.

and

Flannery

B.

P.

-LRB-

-RRB-

Numerical

Recipes

in

C.

Cambridge

University

Press

New

York

NY

Rachlin

J.

Kasif

S.

Salzberg

S.

and

Aha

D.

W.

-LRB-

-RRB-

Towards

a

better

understanding

of

memory

based

reasoning

systems

In

Cohen

W.

W.

and

Hirsh

H.

editors

Eleventh

International

Conference

on

Machine

Learning

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

now

updated

and

submitted

as

an

Artificial

Intelligence

Journal

technical

note

Kasif

S.

Salzberg

S.

Waltz

D.

Rachlin

J.

Aha

D.

W.

-LRB-

-RRB-

Towards

a

framework

for

memory-based

reasoning

Racine

J.

-LRB-

-RRB-

An

efficient

cross-validation

algorithm

for

window

width

selection

for

nonparametric

kernel

regression

Communications

in

Statistics

Simulation

and

Computation

-LRB-

-RRB-

-LCB-

Ramasubramanian

V.

and

Paliwal

K.

K.

-LRB-

-RRB-

A

generalized

optimization

of

the

k-d

tree

for

fast

nearest-neighbour

search

In

International

Conference

on

Acoustics

Speech

and

Signal

Processing

Raz

J.

Turetsky

B.

I.

and

Fein

G.

-LRB-

-RRB-

Selecting

the

smoothing

parameter

for

estimation

of

smoothly

changing

evoked

potential

signals

Biometrics

-LCB-

Renka

R.

J.

-LRB-

-RRB-

Multivariate

interpolation

of

large

sets

of

scattered

data

ACM

Transactions

on

Mathematical

Software

-LRB-

-RRB-

-LCB-

Ruppert

D.

and

Wand

M.

P.

-LRB-

-RRB-

Multivariate

locally

weighted

least

squares

regression

The

Annals

of

Statistics

-LRB-

-RRB-

-LCB-

Ruprecht

D.

and

Muller

H.

-LRB-

-RRB-

Image

warping

with

scattered

data

interpolation

methods

Technical

Report

Universitat

Dortmund

Fachbereich

Informatik

Dortmund

Germany

Available

for

anonymous

FTP

from

informatik.uni-dortmund

de

in

Z.

Ruprecht

D.

and

Muller

H.

-LRB-

-RRB-

Free

form

deformation

with

scattered

data

interpolation

methods

In

Farin

G.

Hagen

H.

and

Noltemeier

H.

editors

Geometric

Modelling

-LRB-

Computing

Suppl

-RRB-

pages

-LCB-

Springer

Verlag

Available

for

anonymous

FTP

from

informatik.uni-dortmund

de

in

Z.

Ruprecht

D.

and

Muller

H.

-LRB-

-RRB-

Deformed

cross-dissolves

for

image

interpolation

in

scientific

visualization

The

Journal

of

Visualization

and

Computer

Animation

-LRB-

-RRB-

-LCB-

Available

for

anonymous

FTP

from

informatik.uni-dortmund

de

in

Z.

Ruprecht

D.

and

Muller

H.

-LRB-

-RRB-

A

framework

for

generalized

scattered

data

interpolation

Technical

Report

Universitat

Dortmund

Fachbereich

Informatik

Dortmund

Germany

Available

for

anonymous

FTP

from

informatik.uni-dortmund

de

in

Z.

Ruprecht

D.

Nagel

R.

and

Muller

H.

-LRB-

-RRB-

Spatial

free

form

deformation

with

scattered

data

interpolation

methods

Technical

Report

Fachbereich

Informatik

der

Universitat

Dortmund

Dortmund

Germany

Accepted

for

publication

by

Computers

Graphics

Available

for

anonymous

FTP

from

informatik.uni-dortmund

de

in

Z.

Rust

R.

T.

and

Bornman

E.

O.

-LRB-

-RRB-

Distribution-free

methods

of

approximating

nonlinear

marketing

relationships

Journal

of

Marketing

Research

XIX

-LCB-

Sabin

M.

A.

-LRB-

-RRB-

Contouring

-LCB-

a

review

of

methods

for

scattered

data

In

Brodlie

K.

editor

Mathematical

Methods

in

Computer

Graphics

and

Design

pages

-LCB-

Academic

Press

New

York

NY

Saitta

L.

editor

-LRB-

-RRB-

Thirteenth

International

Conference

on

Machine

Learning

Morgan

Kaufmann

San

Mateo

CA

Samet

H.

-LRB-

-RRB-

The

Design

and

Analysis

of

Spatial

Data

Structures

Addison-Wesley

Reading

MA

Schaal

S.

and

Atkeson

C.

G.

-LRB-

-RRB-

Assessing

the

quality

of

learned

local

models

In

Cowan

et

al.

-LRB-

-RRB-

pages

-LCB-

Schaal

S.

and

Atkeson

C.

G.

-LRB-

-RRB-

From

isolation

to

cooperation

An

alternative

view

of

a

system

of

experts

proceedings

in

press

Scott

D.

W.

-LRB-

-RRB-

Multivariate

Density

Estimation

Wiley

New

York

NY

Seber

G.

A.

F.

-LRB-

-RRB-

Linear

Regression

Analysis

John

Wiley

New

York

NY

Seifert

B.

Brockmann

M.

Engel

J.

and

Gasser

T.

-LRB-

-RRB-

Fast

algorithms

for

nonparametric

curve

estimation

Journal

of

Computational

and

Graphical

Statistics

-LRB-

-RRB-

-LCB-

Seifert

B.

and

Gasser

T.

-LRB-

-RRB-

Variance

properties

of

local

polynomials

Shepard

D.

-LRB-

-RRB-

A

two-dimensional

function

for

irregularly

spaced

data

In

ACM

National

Conference

pages

-LCB-

Solow

A.

R.

-LRB-

-RRB-

Detecting

changes

through

time

in

the

variance

of

a

long-term

hemispheric

temperature

record

An

application

of

robust

locally

weighted

regression

Journal

of

Climate

-LCB-

Specht

D.

E.

-LRB-

-RRB-

A

general

regression

neural

network

IEEE

Transactions

on

Neural

Networks

-LRB-

-RRB-

-LCB-

Sproull

R.

F.

-LRB-

-RRB-

Refinements

to

nearest-neighbor

searching

in

k-d

trees

Algorithmica

-LCB-

Stanfill

C.

-LRB-

-RRB-

Memory-based

reasoning

applied

to

English

pronunciation

In

Sixth

National

Conference

on

Artificial

Intelligence

pages

-LCB-

Stanfill

C.

and

Waltz

D.

-LRB-

-RRB-

Toward

memory-based

reasoning

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

Steinbuch

K.

-LRB-

-RRB-

Die

lernmatrix

Kybernetik

-LCB-

Steinbuch

K.

and

Piske

U.

A.

W.

-LRB-

-RRB-

Learning

matrices

and

their

applications

IEEE

Transactions

on

Electronic

Computers

-LCB-

Stone

C.

J.

-LRB-

-RRB-

Nearest

neighbor

estimators

of

a

nonlinear

regression

function

In

Computer

Science

and

Statistics

Annual

Symposium

on

the

Interface

pages

-LCB-

Stone

C.

J.

-LRB-

-RRB-

Consistent

nonparametric

regression

The

Annals

of

Statistics

-LCB-

Stone

C.

J.

-LRB-

-RRB-

Optimal

rates

of

convergence

for

nonparametric

estimators

The

Annals

of

Statistics

-LCB-

Stone

C.

J.

-LRB-

-RRB-

Optimal

global

rates

of

convergence

for

nonparametric

regression

The

Annals

of

Statistics

-LRB-

-RRB-

-LCB-

Sumita

E.

Oi

K.

Furuse

O.

Iida

H.

Higuchi

T.

Takahashi

N.

and

Kitano

H.

-LRB-

-RRB-

Examplebased

machine

translation

on

massively

parallel

processors

In

IJCAI

-LRB-

-RRB-

pages

-LCB-

Tadepalli

P.

and

Ok

D.

-LRB-

-RRB-

Scaling

up

average

reward

reinforcement

learning

by

approximating

the

domain

models

and

the

value

function

In

Saitta

-LRB-

-RRB-

Tamada

T.

Maruyama

M.

Nakamura

Y.

Abe

S.

and

Maeda

K.

-LRB-

-RRB-

Water

demand

forecasting

by

memory

based

learning

Water

Science

and

Technology

-LRB-

-LCB-

-RRB-

-LCB-

Taylor

W.

K.

-LRB-

-RRB-

Pattern

recognition

by

means

of

automatic

analogue

apparatus

Proceedings

of

The

Institution

of

Electrical

Engineers

-LCB-

Taylor

W.

K.

-LRB-

-RRB-

A

parallel

analogue

reading

machine

Control

-LCB-

Thorpe

S.

-LRB-

-RRB-

Localized

versus

distributed

representations

In

Arbib

M.

A.

editor

The

Handbook

of

Brain

Theory

and

Neural

Networks

pages

-LCB-

The

MIT

Press

Cambridge

MA

Thrun

S.

-LRB-

-RRB-

Is

learning

the

n-th

thing

any

easier

than

learning

the

first

In

Advances

in

Neural

Information

Processing

Systems

-LRB-

NIPS

-RRB-

l.

Thrun

S.

and

O'Sullivan

J.

-LRB-

-RRB-

Discovering

structure

in

multiple

learning

tasks

The

TC

algorithm

In

Saitta

-LRB-

-RRB-

l.

Tibshirani

R.

and

Hastie

T.

-LRB-

-RRB-

Local

likelihood

estimation

Journal

of

the

American

Statistical

Association

-LCB-

Ting

K.

M.

and

Cameron-Jones

R.

M.

-LRB-

-RRB-

Exploring

a

framework

for

instance

based

learning

and

naive

Bayesian

classifiers

In

Proceedings

of

the

Seventh

Australian

Joint

Conference

on

Artificial

Intelligence

Armidale

Australia

World

Scientific

Tou

J.

T.

and

Gonzalez

R.

C.

-LRB-

-RRB-

Pattern

Recognition

Principles

Addison-Wesley

Reading

MA

Townshend

B.

-LRB-

-RRB-

Nonlinear

prediction

of

speech

signals

In

Casdagli

and

Eubank

-LRB-

-RRB-

pages

-LCB-

Proceedings

of

a

Workshop

on

Nonlinear

Modeling

and

Forecasting

September

Santa

Fe

New

Mexico

Tsybakov

A.

B.

-LRB-

-RRB-

Robust

reconstruction

of

functions

by

the

local

approximation

method

Problems

of

Information

Transmission

-LCB-

Tukey

J.

-LRB-

-RRB-

Exploratory

Data

Analysis

Addison-Wesley

Reading

MA

Turetsky

B.

I.

Raz

J.

and

Fein

G.

-LRB-

-RRB-

Estimation

of

trial-to-trial

variation

in

evoked

potential

signals

by

smoothing

across

trials

Psychophysiology

-LRB-

-RRB-

-LCB-

Turlach

B.

A.

and

Wand

M.

P.

-LRB-

-RRB-

Fast

computation

of

auxiliary

quantities

in

local

polynomial

regression

van

der

Smagt

P.

Groen

F.

and

van

het

Groenewoud

F.

-LRB-

-RRB-

The

locally

linear

nested

network

for

robot

manipulation

In

Proceedings

of

the

IEEE

International

Conference

on

Neural

Networks

pages

-LCB-

ftp

ftp.fwi.uva.nl

pub/computer-systems/aut-sys

ps.gz

Vapnik

V.

-LRB-

-RRB-

Principles

of

risk

minimization

for

learning

theory

In

Moody

J.

E.

Hanson

S.

J.

and

Lippmann

R.

P.

editors

Advances

In

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufman

San

Mateo

CA

Vapnik

V.

and

Bottou

L.

-LRB-

-RRB-

Local

algorithms

for

pattern

recognition

and

dependencies

estimation

Neural

Computation

-LRB-

-RRB-

-LCB-

Walden

A.

T.

and

Prescott

P.

-LRB-

-RRB-

Identification

of

trends

in

annual

maximum

sea

levels

using

robust

locally

weighted

regression

Estuarine

Coastal

and

Shelf

Science

-LCB-

Walters

R.

F.

-LRB-

-RRB-

Contouring

by

machine

A

user

's

guide

American

Association

of

Petroleum

Geologists

Bulletin

-LRB-

-RRB-

-LCB-

Waltz

D.

L.

-LRB-

-RRB-

Applications

of

the

Connection

Machine

Computer

-LRB-

-RRB-

-LCB-

Wand

M.

P.

and

Jones

M.

C.

-LRB-

-RRB-

Comparison

of

smoothing

parameterizations

in

bivariate

kernel

density

estimation

Journal

of

the

American

Statistical

Association

-LCB-

Wand

M.

P.

and

Jones

M.

C.

-LRB-

-RRB-

Kernel

Smoothing

Chapman

and

Hall

London

Wand

M.

P.

and

Schucany

W.

R.

-LRB-

-RRB-

Gaussian-based

kernels

for

curve

estimation

and

window

width

selection

Canadian

Journal

of

Statistics

-LCB-

Wang

Z.

Isaksson

T.

and

Kowalski

B.

R.

-LRB-

-RRB-

New

approach

for

distance

measurement

in

locally

weighted

regression

Analytical

Chemistry

-LRB-

-RRB-

-LCB-

Watson

G.

S.

-LRB-

-RRB-

Smooth

regression

analysis

Sankhy?a

The

Indian

Journal

of

Statistics

Series

A

-LCB-

Weisberg

S.

-LRB-

-RRB-

Applied

Linear

Regression

John

Wiley

and

Sons

Wess

S.

Althoff

K.-D.

and

Derwand

G.

-LRB-

-RRB-

Using

k-d

trees

to

improve

the

retrieval

step

in

case-based

reasoning

In

Wess

S.

Althoff

K.-D.

and

Richter

M.

M.

editors

Topics

in

CaseBased

Reasoning

pages

-LCB-

Springer-Verlag

New

York

NY

Proceedings

of

the

First

European

Workshop

Wettschereck

D.

-LRB-

-RRB-

A

Study

Of

Distance-Based

Machine

Learning

Algorithms

PhD

dissertation

Oregon

State

University

Department

of

Computer

Science

Corvalis

OR

Wijnberg

L.

and

Johnson

T.

-LRB-

-RRB-

Estimation

of

missing

values

in

lead

air

quality

data

sets

In

Johnson

T.

R.

and

Penkala

S.

J.

editors

Quality

Assurance

in

Air

Pollution

Measurements

Air

Pollution

Control

Association

Pittsburgh

PA.

Transactions

An

APCA

International

Specialty

Conference

Wolberg

G.

-LRB-

-RRB-

Digital

Image

Warping

IEEE

Computer

Society

Press

Los

Alamitos

CA

Yasunaga

M.

and

Kitano

H.

-LRB-

-RRB-

Robustness

of

the

memory-based

reasoning

implemented

by

wafer

scale

integration

IEICE

Transactions

on

Information

and

Systems

-LRB-

-RRB-

-LCB-

Zografski

Z.

-LRB-

-RRB-

Neuromorphic

Algorithmic

and

Logical

Models

for

the

Automatic

Synthesis

of

Robot

Action

PhD

dissertation

University

of

Ljubljana

Ljubljana

Slovenia

Yugoslavia

Zografski

Z.

-LRB-

-RRB-

New

methods

of

machine

learning

for

the

construction

of

integrated

neuromorphic

and

associative-memory

knowledge

bases

In

Zajc

B.

and

Solina

F.

editors

Proceedings

Mediterranean

Electrotechnical

Conference

volume

II

pages

-LCB-

Ljubljana

Slovenia

Yugoslavia

IEEE

catalog

number

Zografski

Z.

-LRB-

-RRB-

Geometric

and

neuromorphic

learning

for

nonlinear

modeling

control

and

forecasting

In

Proceedings

of

the

IEEE

International

Symposium

on

Intelligent

Control

pages

-LCB-

Glasgow

Scotland

IEEE

catalog

number

Zografski

Z.

and

Durrani

T.

-LRB-

-RRB-

Comparing

predictions

from

neural

networks

and

memory-based

learning

In

Proceedings

ICANN

International

Conference

on

Artificial

Neural

Networks

pages

-LCB-

Paris

France

distance functions
global tuning
interference
lazy learning
least commitment learning
local tuning
locally weighted regression
loess
lwr
memorybased learning
smoothing parameters
weighting functions