Locally

Weighted

Learning

for

Control

Christopher

G.

Atkeson

Andrew

W.

Moorey

and

Stefan

Schaalz

College

of

Computing

Georgia

Institute

of

Technology?z

Atlantic

Drive

Atlanta

GA

cga@cc.gatech.edu,

sschaal@cc.gatech.edu

fax

Carnegie

Mellon

Universityy

Forbes

Ave

Pittsburgh

PA

awm@cs.cmu.edu

ATR

Human

Information

Processing

Research

Laboratoriesz

Hikaridai

Seika-cho

Soraku-gun

Kyoto

Japan

June

at

Abstract

Lazy

learning

methods

provide

useful

representations

and

training

algorithms

for

learning

about

complex

phenomena

during

autonomous

adaptive

control

of

complex

systems

This

paper

surveys

ways

in

which

locally

weighted

learning

a

type

of

lazy

learning

has

been

applied

by

us

to

control

tasks

We

explain

various

forms

that

control

tasks

can

take

and

how

this

affects

the

choice

of

learning

paradigm

The

discussion

section

explores

the

interesting

impact

that

explicitly

remembering

all

previous

experiences

has

on

the

problem

of

learning

to

control

Introduction

The

necessity

for

self

improvement

in

control

systems

is

becoming

more

apparent

as

fields

such

as

robotics

factory

automation

and

autonomous

vehicles

become

impeded

by

the

complexity

of

inventing

and

programming

satisfactory

control

laws

Learned

models

of

complex

tasks

can

aid

the

design

of

appropriate

control

laws

for

these

tasks

which

often

involve

decisions

based

on

streams

of

information

from

sensors

and

actuators

where

data

is

relatively

plentiful

The

tasks

may

change

over

time

or

multiple

tasks

may

need

to

be

performed

Lazy

learning

methods

provide

an

approach

to

learning

models

of

complex

phenomena

dealing

with

large

amounts

of

data

training

quickly

and

avoiding

interference

between

multiple

tasks

during

control

of

complex

systems

-LRB-

Atkeson

et

al.

-RRB-

This

paper

describes

five

ways

in

which

lazy

learning

techniques

have

been

applied

by

us

to

control

tasks

In

learning

control

there

is

an

important

distinction

between

representational

tools

such

as

lookup

tables

neural

networks

databases

of

experiences

or

structured

representations

and

what

we

will

call

learning

paradigms

which

define

what

the

representation

is

used

for

where

training

data

comes

from

how

the

training

data

is

used

to

modify

the

representation

whether

exploratory

actions

are

performed

and

other

related

issues

It

is

difficult

to

evaluate

a

representational

tool

independently

of

the

paradigm

in

which

it

is

used

and

vice

versa

A

successful

robot

learning

algorithm

typically

is

composed

of

sophisticated

representational

tools

and

learning

paradigms

We

will

describe

using

the

same

representational

tool

locally

weighted

learning

-LRB-

Atkeson

et

al.

-RRB-

in

different

tasks

with

different

learning

paradigms

and

with

different

results

In

defining

paradigms

for

learning

to

control

complex

systems

it

is

useful

to

identify

three

separate

components

of

an

indirect

-LRB-

model-based

-RRB-

adaptive

control

system

modeling

exploration

and

policy

design

The

first

component

modeling

is

the

process

of

forming

explicit

models

of

the

task

and

the

environment

All

of

the

approaches

we

will

describe

will

form

explicit

world

models

Moore

and

Atkeson

-LRB-

-RRB-

explore

some

of

the

advantages

and

disadvantages

of

approaches

that

form

explicit

models

versus

those

that

avoid

forming

models

Often

the

modeling

process

is

equated

with

function

approximation

in

which

a

representational

tool

is

used

to

fit

a

training

data

set

Focusing

only

on

the

modeling

component

leaves

several

important

questions

unanswered

For

example

where

does

the

training

data

come

from

and

what

new

training

data

should

be

collected

are

addressed

by

the

exploration

component

The

question

how

should

the

identified

model

be

used

to

select

actions

is

addressed

by

the

policy

design

or

control

law

design

component

The

aim

of

this

paper

is

to

survey

the

implications

of

using

locally

weighted

regression

a

lazy

learning

technique

as

the

modeling

component

of

our

three

part

control

system

Lazy

modeling

techniques

can

not

be

implemented

or

discussed

without

exploring

related

issues

in

exploration

and

policy

design

Although

the

policy

design

and

exploration

components

are

not

lazy

in

the

same

sense

as

the

modeling

component

they

should

exploit

the

capabilities

of

lazy

modeling

and

make

a

lazy

modeler

's

job

easier

Why

Focus

on

Lazy

Learning

For

Learning

to

Control

We

will

not

review

lazy

learning

here

but

expect

that

our

reader

has

already

read

the

companion

paper

in

this

collection

-LRB-

Atkeson

et

al.

-RRB-

from

which

we

will

borrow

both

terminology

and

notation

In

the

form

of

lazy

learning

we

will

focus

on

locally

weighted

learning

experiences

are

explicitly

remembered

and

predictions

and

generalizations

are

performed

in

real

time

by

building

a

local

model

to

answer

any

particular

query

-LRB-

an

input

for

which

the

function

's

output

is

desired

-RRB-

The

motivation

for

focussing

on

locally

weighted

learning

was

not

that

it

is

a

more

accurate

function

approximator

than

other

methods

such

as

multi-layer

sigmoidal

neural

networks

radial

basis

functions

regression

trees

projection

pursuit

regres

sion

other

statistical

nonparametric

regression

techniques

and

global

regression

techniques

but

that

lazy

learning

techniques

avoid

negative

interference

One

of

the

primary

characteristics

of

learning

to

control

a

robot

is

that

data

comes

in

continuously

and

the

distribution

of

the

data

changes

as

the

robot

learns

and

changes

its

performance

task

Locally

weighted

learning

easily

learns

in

real

time

from

the

continuous

stream

of

training

data

It

also

avoids

the

negative

interference

exhibited

by

other

modeling

approaches

because

locally

weighted

learning

retains

all

the

training

data

as

do

many

lazy

learning

methods

-LRB-

Atkeson

et

al.

-RRB-

Our

approach

to

modeling

the

complex

functions

found

in

typical

task

or

process

dynamics

is

to

use

a

collection

of

simple

local

models

One

benefit

of

local

modeling

is

that

it

avoids

the

difficult

problem

of

finding

an

appropriate

structure

for

a

global

model

A

key

idea

in

lazy

learning

is

to

form

a

training

set

for

the

local

model

after

a

query

is

given

This

approach

allows

us

to

select

from

the

training

set

only

relevant

experiences

-LRB-

nearby

samples

-RRB-

and

to

weight

those

experiences

according

to

their

relevance

to

the

query

We

form

a

local

model

of

the

function

at

the

query

point

much

as

a

Taylor

series

models

a

function

in

the

neighborhood

of

a

point

This

local

model

is

then

used

to

predict

the

output

of

the

function

for

that

query

After

answering

the

query

the

local

model

is

discarded

A

new

local

model

is

created

to

answer

each

query

This

leads

to

another

benefit

of

lazy

modeling

for

control

we

can

delay

the

choice

of

local

model

structure

and

structural

parameters

until

a

query

must

be

answered

and

we

can

make

different

choices

for

subsequent

queries

-LRB-

Atkeson

et

al.

-RRB-

Locally

weighted

learning

can

represent

nonlinear

functions

yet

has

simple

training

rules

with

a

single

global

optimum

for

building

a

local

model

in

response

to

a

query

This

allows

complex

nonlinear

models

to

be

identified

-LRB-

trained

-RRB-

quickly

Currently

we

are

using

polynomials

as

the

local

models

Since

the

polynomial

local

models

are

linear

in

the

parameters

to

be

estimated

we

can

calculate

these

parameters

using

a

linear

regression

Fast

training

makes

continuous

learning

from

a

stream

of

new

input

data

possible

It

is

true

that

lazy

learning

transfers

the

computational

load

onto

the

lookup

process

but

our

experience

is

that

the

linear

parameter

estimation

process

during

lookup

in

locally

weighted

learning

is

still

fast

enough

for

real

time

robot

learning

-LRB-

Atkeson

et

al.

-RRB-

We

use

cross

validation

to

choose

an

appropriate

distance

metric

and

weighting

function

and

to

help

find

irrelevant

input

variables

and

terms

in

the

local

model

In

fact

performing

one

cross

validation

evaluation

in

lazy

learning

is

no

more

expensive

than

processing

a

single

query

-LRB-

Atkeson

et

al.

-RRB-

Cheap

cross

validation

makes

search

for

model

parameters

routine

and

we

have

explored

procedures

that

take

advantage

of

this

-LRB-

Atkeson

et

al.

Maron

and

Moore

Moore

et

al.

Moore

and

Lee

-RRB-

We

have

extended

the

locally

weighted

learning

approach

to

give

information

about

the

reliability

of

the

predictions

and

local

linearizations

generated

based

on

the

local

density

and

distribution

of

the

data

and

an

estimate

of

the

local

variance

-LRB-

Atkeson

et

al.

Schaal

and

Atkeson

b

-RRB-

This

allows

a

robot

to

monitor

its

own

skill

level

protect

itself

from

its

ignorance

by

designing

robust

policies

and

guide

its

exploratory

behavior

Another

attractive

feature

of

locally

weighted

learning

is

flexibility

There

are

explicit

parameters

to

control

smoothing

outlier

rejection

forgetting

and

other

processes

The

modeling

process

is

easy

to

understand

and

therefore

easy

to

adjust

or

control

-LRB-

Atkeson

et

al.

-RRB-

We

will

see

how

the

explicit

representation

of

specific

memories

can

speed

up

convergence

and

improve

the

robustness

and

autonomy

of

optimization

and

control

algorithms

-LRB-

Atkeson

et

al.

Moore

and

Schneider

-RRB-

It

is

frustrating

to

watch

a

robot

repeat

its

mistakes

with

only

a

slight

improvement

on

each

attempt

The

goal

of

the

learning

algorithms

described

here

is

to

improve

performance

as

rapidly

as

possible

using

as

little

training

data

as

possible

-LRB-

data

efficiency

-RRB-

Related

Work

Locally

weighted

learning

is

being

increasingly

used

in

control

Zografski

has

explored

the

use

of

locally

weighted

regression

in

robot

control

and

modeling

time

series

and

also

compared

LWR

to

neural

networks

and

other

methods

-LRB-

Zografski

Zografski

and

Durrani

-RRB-

Gorinevsky

and

Connolly

-LRB-

-RRB-

compared

several

different

approximation

schemes

-LRB-

neural

nets

Kohonen

maps

radial

basis

functions

and

local

polynomical

fits

-RRB-

on

simulated

robot

inverse

kinematics

with

added

noise

and

showed

that

local

polynomial

fits

were

more

accurate

than

all

other

methods

van

der

Smagt

et

al.

-LRB-

-RRB-

learned

robot

kinematics

using

local

linear

models

at

the

leaves

of

a

tree

data

structure

Tadepalli

and

Ok

-LRB-

-RRB-

apply

local

linear

regression

to

reinforcement

learning

Baird

and

Klopf

-LRB-

-RRB-

apply

nearest

neighbor

techniques

and

weighted

averaging

to

reinforcement

learning

and

Thrun

-LRB-

-RRB-

and

Thrun

and

O'Sullivan

-LRB-

-RRB-

apply

similar

techniques

to

robot

learning

Connell

and

Utgoff

-LRB-

-RRB-

interpolated

a

value

function

using

locally

weighted

averaging

to

balance

an

inverted

pendulum

-LRB-

a

pole

-RRB-

on

a

moving

cart

Peng

-LRB-

-RRB-

performed

the

cart

pole

task

using

locally

weighted

regression

to

interpolate

a

value

function

Aha

and

Salzberg

-LRB-

-RRB-

explored

nearest

neighbor

and

locally

weighted

learning

approaches

to

a

tracking

task

in

which

a

robot

pursued

and

caught

a

ball

McCallum

-LRB-

-RRB-

explored

the

use

of

lazy

learning

techniques

in

situations

where

states

were

not

completely

measured

Farmer

and

Sidorowich

-LRB-

b

-RRB-

apply

locally

weighted

regression

to

modeling

and

prediction

of

chaotic

dynamic

systems

Huang

-LRB-

-RRB-

uses

nearest

neighbor

and

weighted

averaging

techniques

to

cache

simulation

results

and

accelerate

a

movement

planner

Outline

This

article

is

organized

by

types

of

control

tasks

and

in

the

next

sections

we

will

examine

a

progression

of

control

tasks

of

increasing

complexity

We

have

chosen

these

tasks

because

we

have

implemented

lazy

learning

as

part

of

a

learning

controller

for

each

of

them

For

each

type

of

task

we

will

show

how

lazy

learning

of

models

interacts

with

other

parts

of

the

learning

control

paradigm

being

described

For

several

tasks

we

also

provide

implementation

details

The

progression

of

control

tasks

is

outlined

in

Table

Temporally

independent

tasks

include

many

forms

of

setpoint

based

process

control

and

are

of

economic

importance

We

describe

several

versions

of

temporally

dependent

tasks

which

include

trajectory

following

tasks

such

as

process

control

transients

and

vehicle

maneuvers

We

conclude

with

a

discussion

of

some

of

the

benefits

and

drawbacks

of

lazy

learning

in

this

context

Table

The

control

tasks

explored

in

this

paper

Symbols

and

mathematics

described

in

some

of

the

entries

will

be

explained

in

the

corresponding

sections

Task

Task

Specification

Goal

Example

Sec

Temporally

Independent

yd

the

desired

output

Choose

u

such

that

E

-LSB-

y

-RSB-

yd

Billiards

Deadbeat

Control

xd

or

trajectory

fxd

-LRB-

t

-RRB-

g

Choose

u

-LRB-

t

-RRB-

such

that

E

-LSB-

x

-LRB-

t

-RRB-

-RSB-

xd

-LRB-

t

-RRB-

Devil

Sticking

I

Dynamic

Regulation

xd

and

matrices

Q

and

R

Minimize

future

cost

C

ffix

-LRB-

t

-RRB-

TQffix

-LRB-

t

-RRB-

u

-LRB-

t

-RRB-

TRu

-LRB-

t

-RRB-

Devil

Sticking

II

Dynamic

Regulation

unspecified

setpoint

Q

and

R

Choose

setpoint

to

minimize

future

cost

C

Devil

Sticking

III

Nonlinear

Optimal

Control

Cost

function

G

-LRB-

x

-LRB-

t

-RRB-

u

-LRB-

t

-RRB-

t

-RRB-

Find

a

control

policy

to

minimize

the

sum

of

future

costs

Puck

Temporally

Independent

Tasks

In

the

simplest

class

of

tasks

we

will

consider

the

environment

provides

an

outcome

represented

with

a

vector

y

as

a

function

of

an

action

vector

u

which

we

can

choose

a

state

vector

x

which

we

can

observe

but

not

choose

and

random

noise

y

f

-LRB-

x

u

-RRB-

noise

-LRB-

-RRB-

The

task

is

to

choose

u

so

that

the

expected

outcome

y

is

yd

E

-LSB-

y

-RSB-

yd

where

E

is

the

expectation

operator

from

probability

theory

The

function

f

-LRB-

-RRB-

is

not

known

at

the

beginning

of

the

task

Section

will

describe

how

lazy

learning

can

be

used

to

learn

a

model

of

f

-LRB-

-RRB-

bf

-LRB-

-RRB-

Several

relationships

could

be

modeled

using

lazy

learning

techniques

including

forward

models

inverse

models

policies

and

value

functions

We

will

discuss

policies

and

value

functions

in

the

context

of

temporally

dependent

tasks

in

later

sections

The

next

sections

describe

inverse

and

forward

models

Control

Using

Inverse

Models

An

inverse

model

uses

states

and

outcomes

to

predict

the

necessary

action

-LRB-

Atkeson

Miller

-RRB-

u

bf

-LRB-

x

y

-RRB-

-LRB-

-RRB-

This

function

specifies

directly

what

action

to

take

in

each

state

but

does

not

specify

what

would

happen

given

a

state

and

an

action

A

lazy

learner

can

represent

an

inverse

model

using

a

database

of

experiences

arranged

so

that

the

input

vectors

of

each

experience

are

the

concatenation

of

state

and

outcome

vectors

-LRB-

Figure

-RRB-

The

corresponding

output

is

the

action

needed

to

produce

the

given

outcome

from

the

given

state

The

database

is

trained

by

adding

new

observed

states

actions

and

outcomes

-LRB-

x

u

y

-RRB-

A

learned

inverse

model

can

provide

a

conceptually

simple

controller

for

temporally

independent

tasks

An

action

is

chosen

by

using

the

current

state

and

desired

outcome

as

an

x

-LRB-

-RRB-

y

-LRB-

-RRB-

u

-LRB-

-RRB-

x

-LRB-

-RRB-

y

-LRB-

-RRB-

u

-LRB-

-RRB-

x

-LRB-

-RRB-

y

-LRB-

-RRB-

u

-LRB-

-RRB-

x

-LRB-

n

-RRB-

y

-LRB-

n

-RRB-

u

-LRB-

n

-RRB-

Figure

A

database

implementing

an

inverse

model

index

into

the

database

The

closest

match

in

the

database

can

be

found

or

an

interpolation

of

nearby

experiences

-LRB-

i.e.

a

weighted

average

or

locally

weighted

regression

approach

-RRB-

can

be

used

If

there

are

no

stored

experiences

close

enough

to

the

current

situation

another

method

such

as

choosing

actions

randomly

can

be

used

to

select

an

action

This

distance

threshold

is

task

dependent

and

can

be

set

by

the

user

The

strength

of

an

inverse

model

controller

in

conjunction

with

lazy

learning

is

that

the

learning

is

aggressive

during

repeated

attempts

to

achieve

the

same

goal

the

action

that

is

applied

is

not

an

incrementally

adjusted

version

of

the

previous

action

but

is

instead

the

action

that

the

lazy

learner

predicts

will

directly

achieve

the

required

outcome

Given

a

monotonic

relationship

between

u

and

y

the

sequence

of

actions

that

are

chosen

are

closely

related

to

the

Secant

method

-LRB-

Conte

and

De

Boor

-RRB-

for

numerically

finding

the

zero

of

a

function

See

-LRB-

Ortega

and

Rheinboldt

-RRB-

for

a

good

discussion

of

the

multidimensional

generalization

of

the

Secant

method

An

inverse

model

represented

using

locally

weighted

regression

and

trained

initially

with

a

feedback

learner

has

been

used

by

Atkeson

-LRB-

-RRB-

A

commonly

observed

problem

with

the

inverse

model

is

that

if

the

vector

space

of

actions

has

a

different

dimensionality

than

that

of

outcomes

then

the

inverse

model

is

not

well

defined

Problems

also

result

if

the

mapping

is

not

one

to

one

or

if

there

are

misleading

noisy

observations

Learning

can

become

stuck

in

permanent

pockets

of

inaccuracy

that

are

not

reduced

with

experience

Figure

illustrates

a

problem

where

a

non-monotonic

relation

between

actions

and

outcomes

is

misinterpreted

by

the

inverse

model

Even

if

the

inverse

model

had

interpreted

the

data

correctly

any

locally

weighted

averaging

on

u

would

have

led

to

incorrect

actions

-LRB-

Moore

Jordan

and

Rumelhart

-RRB-

In

subsequent

sections

on

temporally

dependent

tasks

we

will

discuss

how

sometimes

the

action

selected

by

the

inverse

function

is

too

aggressive

Control

Using

Forward

Models

The

forward

model

uses

states

and

actions

to

predict

outcomes

-LRB-

Miller

Mel

Moore

Jordan

and

Rumelhart

-RRB-

y

bf

-LRB-

x

u

-RRB-

-LRB-

-RRB-

This

allows

prediction

of

the

effects

of

various

actions

-LRB-

mental

simulation

-RRB-

but

does

not

prescribe

the

correct

action

to

take

Action

-LRB-

u

-RRB-

Behavior

-LRB-

y

-RRB-

yduchosenTrue

functionInverseModelprediction

Figure

The

true

relation

-LRB-

shown

as

the

thick

black

line

-RRB-

is

non-monotonic

When

an

outcome

is

desired

at

the

shown

value

yd

the

action

that

is

suggested

produces

an

outcome

that

differs

from

the

desired

one

Worse

the

new

data

point

that

is

added

-LRB-

at

the

intersection

of

the

thick

black

line

and

the

vertical

arrow

-RRB-

will

not

change

the

inverse

model

near

yd

and

the

same

mistake

will

be

repeated

indefinitely

x

-LRB-

-RRB-

u

-LRB-

-RRB-

y

-LRB-

-RRB-

x

-LRB-

-RRB-

u

-LRB-

-RRB-

y

-LRB-

-RRB-

x

-LRB-

-RRB-

u

-LRB-

-RRB-

y

-LRB-

-RRB-

x

-LRB-

n

-RRB-

u

-LRB-

n

-RRB-

y

-LRB-

n

-RRB-

Figure

A

database

implementing

a

forward

model

We

now

arrange

the

memory-base

so

that

the

input

vectors

of

each

data

point

are

the

concatenation

of

state

and

action

vectors

-LRB-

Figure

-RRB-

The

corresponding

output

is

the

actual

outcome

that

was

observed

when

the

state-action

pair

was

executed

in

the

real

world

The

forward

model

can

be

trained

from

observations

of

states

actions

and

outcomes

-LRB-

x

u

y

-RRB-

To

use

this

model

for

control

requires

more

than

a

single

lookup

Actions

are

chosen

by

on

-LCB-

line

numerical

inversion

of

the

forward

model

that

requires

searching

a

set

of

actions

to

find

one

that

is

predicted

to

achieve

the

desired

output

This

computation

is

identical

to

numerical

root

finding

over

the

empirical

model

A

number

of

root-finding

schemes

are

applicable

with

desirability

depending

on

the

dimensionality

of

the

actions

the

complexity

of

the

function

and

the

amount

of

time

available

in

which

to

perform

the

search

ffl

Grid

Search

Generate

all

available

actions

sampled

from

a

uniform

grid

over

action

space

Take

the

action

that

is

predicted

to

produce

the

closest

outcome

to

yd

ffl

Random

Search

Generate

random

actions

and

again

use

the

action

which

is

predicted

to

produce

the

closest

outcome

to

yd

ffl

First

Order

Gradient

Search

Perform

a

steepest-ascent

search

from

an

initial

candidate

action

toward

an

action

that

will

give

the

desired

output

-LRB-

Press

et

al.

-RRB-

Finding

the

local

gradient

of

the

empirical

model

is

easy

if

locally

weighted

regression

is

used

-LRB-

Atkeson

et

al.

-RRB-

Part

of

the

computation

of

the

locally

weighted

regression

model

forms

the

local

linear

map

so

it

is

already

available

We

may

write

the

prediction

local

to

x

and

u

as

bf

-LRB-

x

ffix

u

ffiu

-RRB-

ss

c

Affix

Bffiu

order

terms

-LRB-

-RRB-

where

c

is

a

vector

and

A

and

B

are

matrices

obtained

from

the

regression

such

that

c

bf

-LRB-

x

u

-RRB-

Aij

@bfi

@xj

Bij

@bfi

@uj

-LRB-

-RRB-

The

gradient

ascent

iteration

is

uk

uk

BT

-LRB-

yd

c

-RRB-

-LRB-

-RRB-

with

B

and

c

as

defined

in

Equation

This

approach

may

become

stuck

in

local

minima

so

an

initial

grid

search

or

random

search

may

provide

a

set

of

good

starting

points

for

gradient

searches

ffl

Second

Order

Gradient

Search

Use

Newton

's

method

to

iterate

towards

an

action

with

the

desired

output

-LRB-

Press

et

al.

-RRB-

If

uk

is

an

approximate

solution

Newton

's

method

gives

uk

as

a

better

solution

where

uk

uk

B

-LRB-

yd

c

-RRB-

-LRB-

-RRB-

with

B

and

c

as

defined

in

Equation

Newton

's

method

is

less

stable

than

first

order

gradient

search

but

if

a

good

approximate

solution

is

available

perhaps

from

one

of

the

other

search

methods

and

the

local

linear

model

structure

is

correct

in

a

region

including

the

current

action

and

the

best

action

it

produces

a

good

estimate

of

the

best

action

in

only

two

or

three

iterations

If

the

partial

derivative

matrix

B

is

singular

or

the

action

space

and

state

space

differ

in

dimensionality

then

robust

matrix

techniques

based

on

the

pseudo-inverse

can

be

applied

to

invert

B

-LRB-

Press

et

al.

-RRB-

The

forward

model

can

be

used

to

minimize

a

criterion

C

that

penalizes

large

commands

as

well

as

errors

which

also

makes

this

search

more

robust

C

-LRB-

yd

c

-RRB-

TQ

-LRB-

yd

c

-RRB-

uTRu

-LRB-

-RRB-

The

matrices

Q

and

R

allow

the

user

to

control

which

components

of

the

error

are

most

important

Combining

Forward

and

Inverse

models

The

inverse

model

can

provide

a

good

initial

starting

point

for

a

search

using

the

forward

model

bf

-LRB-

x

yd

-RRB-

can

be

evaluated

using

a

lazy

forward

model

with

the

same

data

by

bf

-LRB-

x

-RRB-

Provided

by

is

close

to

yd

Newton

's

method

can

then

be

used

for

further

refinement

If

by

is

not

close

to

yd

the

local

linear

model

may

not

be

a

good

fit

and

the

aggressive

Newton

step

may

move

away

from

the

goal

Exploration

in

Temporally

Independent

Learning

A

nice

feature

of

the

approaches

described

so

far

is

that

in

normal

operation

they

perform

their

own

exploration

reducing

the

need

for

human

supervision

or

external

guidance

The

experiments

are

chosen

greedily

at

the

exact

points

where

the

desired

output

is

predicted

to

be

which

for

the

forward

model

is

guaranteed

to

provide

useful

data

If

an

action

is

wrongly

predicted

to

succeed

the

resulting

new

data

point

will

change

the

prediction

of

the

forward

model

for

that

state

and

action

helping

to

prevent

the

error

from

being

repeated

In

the

early

stages

of

learning

however

there

may

be

no

action

that

is

predicted

to

give

the

desired

outcome

A

simple

experiment

design

strategy

is

to

choose

actions

at

random

It

is

more

effective

to

choose

data

points

which

given

the

uncertainty

inherent

in

the

prediction

are

considered

most

likely

to

achieve

the

desired

outcome

This

can

considerably

reduce

the

exploration

required

-LRB-

Moore

Cohn

et

al.

-RRB-

A

Temporally

Independent

Task

Billiards

In

order

to

explore

the

efficacy

of

lazy

learning

methods

for

the

control

of

temporally

independent

tasks

the

previously

described

approaches

were

implemented

on

the

billiards

robot

shown

in

Figure

-LRB-

Moore

Moore

et

al.

-RRB-

The

equipment

consists

of

a

small

-LRB-

m

m

-RRB-

pool

table

a

spring

actuated

cue

with

a

rotary

joint

under

the

control

of

a

stepper

motor

and

two

cameras

attached

to

a

Datacube

image

processing

system

All

sensing

is

visual

one

camera

looks

along

the

cue

stick

and

the

other

looks

down

at

the

table

The

cue

stick

swivels

around

the

cue

ball

which

in

this

implementation

has

to

start

each

shot

at

the

same

position

A

shot

proceeds

as

follows

At

the

start

of

each

attempt

the

object

ball

-LRB-

i.e.

the

ball

we

want

to

sink

in

a

pocket

-RRB-

is

placed

at

a

random

position

in

the

half

of

the

table

opposite

the

cue

stick

This

random

position

is

selected

by

the

computer

to

avoid

human

bias

The

camera

above

the

table

obtains

the

centroid

image

coordinates

of

the

object

ball

-LRB-

xabove

object

yabove

object

-RRB-

which

constitute

the

state

x.

The

controller

then

uses

an

inverse

model

followed

by

search

over

a

forward

model

to

find

an

action

u

that

is

predicted

to

sink

the

object

ball

into

the

nearer

of

the

two

pockets

at

the

far

end

of

the

table

The

action

is

specified

by

what

we

wish

the

view

from

the

cue

to

be

just

prior

to

shooting

Figure

shows

a

view

from

the

cue

camera

during

this

process

The

cue

swivels

until

the

centroid

of

the

object

ball

's

image

-LRB-

shown

by

the

vertical

line

-RRB-

coincides

with

the

chosen

action

xcue

object

shown

by

the

cross

Figure

The

billiards

robot

In

the

foreground

is

the

cue

stick

which

attempts

to

sink

balls

in

the

far

pockets

Figure

The

view

from

the

cue

camera

during

aiming

The

cue

swivels

until

the

centroid

of

the

object

ball

's

image

-LRB-

shown

by

the

vertical

line

-RRB-

coincides

with

the

chosen

action

xcue

object

shown

by

the

cross

The

shot

is

then

performed

and

observed

by

the

overhead

camera

The

image

after

a

shot

overlaid

with

the

tracking

of

both

balls

is

shown

in

Figure

The

outcome

is

defined

as

the

cushion

and

position

on

the

cushion

where

the

object

ball

first

collides

In

Figure

it

is

the

point

b.

Independent

of

success

or

failure

the

memory-base

is

updated

with

the

new

observation

-LRB-

xabove

object

yabove

object

xcue

object

-RRB-

b

As

time

progresses

the

database

of

experiences

increases

hopefully

converging

to

expertise

in

the

two-dimensional

manifold

of

state-space

corresponding

to

sinking

balls

placed

in

arbitrary

positions

Before

learning

begins

there

is

no

explicit

knowledge

or

calibration

of

the

robot

pool

table

or

cameras

beyond

having

the

object

ball

in

view

of

the

overhead

camera

and

the

assumption

that

the

relationship

between

state

action

and

outcome

is

reasonably

repeatable

In

this

implementation

the

representation

used

for

both

forward

and

inverse

models

was

locally

weighted

regression

using

outlier

removal

and

cross

validation

for

choosing

the

kernel

width

-LRB-

Atkeson

et

al.

-RRB-

Inverse

and

forward

models

were

used

together

the

forward

model

was

searched

with

steepest

ascent

Early

shots

-LRB-

when

no

success

was

predicted

-RRB-

were

uncertainty-based

-LRB-

Moore

-RRB-

After

shots

control

choice

running

on

a

was

taking

seconds

This

implementation

demonstrates

several

important

points

The

first

is

the

precision

required

of

the

modeling

component

The

cue-action

must

be

extremely

precise

for

success

Locally

weighted

regression

provided

the

needed

precision

A

graph

of

the

number

of

successes

against

trial

number

-LRB-

Figure

-RRB-

shows

the

performance

of

the

robot

against

time

b

60 80 100 120

Shot

Number

Recent

Success

Figure

The

trajectory

of

both

balls

is

tracked

using

the

overhead

camera

b

indicates

the

cushion

and

position

on

the

cushion

where

the

object

ball

first

collides

In

this

shot

the

pocket

was

missed

Figure

Frequency

of

successes

versus

control

cycle

for

the

billiards

task

The

number

of

successes

averaged

over

the

twenty

previous

shots

is

shown

Sinking

the

ball

requires

better

than

accuracy

in

the

choice

of

action

the

world

contains

discontinuities

and

there

are

random

outliers

in

the

data

due

to

visual

tracking

errors

and

so

it

is

encouraging

that

within

less

than

experiences

the

robot

had

reached

a

success

rate

An

informal

assessment

of

this

performance

is

that

its

success

rate

is

as

high

as

possible

-LRB-

given

that

the

ball

is

placed

at

random

positions

some

of

which

are

virtually

impossibly

difficult

-RRB-

Unfortunately

the

only

evidence

for

this

is

anecdotal

the

students

who

built

the

robot

-LRB-

one

of

whom

was

an

MIT

billiards

champion

-RRB-

could

not

do

any

better

A

second

point

is

the

non-uniformity

of

the

training

data

distribution

due

to

the

implicit

exploration

process

Although

the

function

being

learned

is

only

inputs

output

it

is

perhaps

surprising

that

it

achieved

sufficient

accuracy

in

only

data

points

The

reason

is

the

aggressive

non-uniformity

of

the

training

data

distribution

almost

all

the

training

data

was

clustered

around

state-action

pairs

which

get

the

ball

in

or

close

to

a

pocket

The

lazy

learner

did

not

expend

many

resources

on

exploring

or

representing

how

to

make

bad

shots

Optimizing

a

Performance

Criterion

Often

a

goal

in

temporally

independent

learning

is

to

optimize

a

particular

criterion

rather

than

achieve

a

particular

outcome

Lazy

learning

can

be

used

to

represent

the

cost

function

directly

and

to

speed

the

search

for

maxima

or

minima

-LRB-

Moore

and

Schneider

-RRB-

A

linear

local

model

can

be

used

to

estimate

the

first

derivatives

-LRB-

gradient

-RRB-

and

a

quadratic

local

model

can

be

used

to

estimate

the

second

derivatives

-LRB-

Hessian

-RRB-

of

the

cost

function

at

the

current

point

in

the

optimization

procedure

These

estimates

can

be

used

in

first

order

gradient

search

or

in

a

Newton

search

that

uses

estimates

of

second

derivatives

Constraints

on

the

output

can

be

included

in

this

optimization

process

Temporal

Dependence

in

Temporally

Independent

Tasks

It

is

considerably

easier

to

choose

actions

for

temporally

independent

than

temporally

dependent

tasks

because

the

choice

of

action

has

no

effect

on

future

states

There

is

no

need

to

consider

the

effects

of

the

current

action

on

future

states

and

indirectly

on

future

performance

In

Section

we

will

consider

temporally

dependent

tasks

where

there

is

an

opportunity

to

choose

suboptimal

actions

in

the

short-term

to

obtain

more

desirable

states

and

thereby

improve

performance

in

the

long-term

However

temporally

independent

tasks

do

provide

an

opportunity

to

increase

the

knowledge

available

to

the

controller

in

order

to

improve

future

performance

They

differ

from

batch

learning

tasks

because

new

training

data

becomes

available

after

each

action

and

the

choice

of

action

which

depends

on

inferences

from

earlier

training

data

affects

the

training

data

available

to

future

decisions

Modifying

actions

to

increase

knowledge

rather

than

greedily

pursue

a

desired

outcome

is

the

responsibility

of

the

exploration

component

of

the

controller

Temporally

Dependent

Tasks

A

more

complex

class

of

learning

control

tasks

occur

when

the

assumption

of

temporal

independence

is

removed

x

-LRB-

t

-RRB-

may

now

be

influenced

by

x

-LRB-

t

-RRB-

A

useful

case

to

explore

is

when

the

outcome

is

the

next

state

x

-LRB-

t

-RRB-

f

-LRB-

x

-LRB-

t

-RRB-

u

-LRB-

t

-RRB-

-RRB-

-LRB-

-RRB-

The

task

may

be

to

regulate

the

state

to

a

predefined

desired

value

called

a

setpoint

xd

or

to

a

sequence

or

trajectory

of

states

xd

-LRB-

-RRB-

xd

-LRB-

-RRB-

xd

-LRB-

-RRB-

Deadbeat

Control

One

approach

to

performing

temporally

dependent

tasks

is

to

use

the

successful

techniques

from

the

previous

section

and

ignore

the

temporal

dependence

One-step

deadbeat

control

chooses

actions

to

-LRB-

in

expectation

-RRB-

cause

the

immediate

next

state

to

be

the

desired

next

state

-LRB-

Stengel

-RRB-

Assuming

the

next

state

is

always

attainable

in

one

step

the

action

may

be

chosen

without

paying

attention

to

future

states

decisions

or

performance

An

Implementation

of

Deadbeat

Control

Devil

Sticking

I

Deadbeat

control

using

lazy

learning

models

was

explored

by

implementing

it

for

a

juggling

task

known

as

devil

sticking

-LRB-

Schaal

and

Atkeson

b

-RRB-

A

center

stick

is

batted

back

and

forth

between

two

handsticks

Figure

shows

a

sketch

of

our

devil

sticking

robot

The

juggling

robot

uses

its

top

two

joints

to

perform

planar

devil

sticking

Hand

sticks

are

mounted

on

the

robot

with

springs

and

dampers

This

implements

a

passive

catch

The

center

stick

does

not

bounce

when

it

hits

the

hand

stick

and

therefore

requires

an

active

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

An

illustration

of

devil

sticking

-LRB-

b

-RRB-

A

sketch

of

our

devil

sticking

robot

A

position

change

due

to

movement

of

joint

and

respectively

is

indicated

in

the

small

sketches

throwing

motion

by

the

robot

To

simplify

the

problem

the

center

stick

is

constrained

by

a

boom

to

move

on

the

surface

of

a

sphere

For

small

movements

the

center

stick

movements

are

approximately

planar

The

boom

also

provides

a

way

to

measure

the

current

state

of

the

center

stick

The

task

state

is

the

predicted

location

of

the

center

stick

when

it

hits

the

hand

stick

held

in

a

nominal

position

Standard

ballistics

equations

for

the

flight

of

the

center

stick

are

used

to

map

flight

trajectory

measurements

into

a

task

state

The

dynamics

of

throwing

the

devil

stick

are

parameterized

by

five

state

and

five

action

variables

resulting

in

a

input/output

model

for

each

hand

Every

time

the

robot

catches

and

throws

the

devil

stick

it

generates

an

experience

of

the

form

-LRB-

xk

uk

xk

-RRB-

where

xk

is

the

current

state

uk

is

the

action

performed

by

the

robot

and

xk

is

the

state

of

the

center

stick

that

results

Initially

we

explored

learning

an

inverse

model

of

the

task

using

deadbeat

control

to

attempt

to

eliminate

all

error

on

each

hit

Each

hand

had

its

own

inverse

model

of

the

form

buk

bf

-LRB-

xk

xk

-RRB-

-LRB-

-RRB-

Before

each

hit

the

system

looked

up

a

command

with

the

predicted

nominal

impact

state

and

the

desired

result

state

xd

buk

bf

-LRB-

xk

xd

-RRB-

-LRB-

-RRB-

Inverse

model

learning

using

lazy

learning

-LRB-

locally

weighted

regression

-RRB-

was

successfully

used

to

train

the

system

to

perform

the

devil

sticking

task

Juggling

runs

up

to

hits

were

achieved

The

system

incorporated

new

data

in

real

time

and

used

databases

of

several

hundred

hits

Lookups

took

less

than

milliseconds

and

therefore

several

lookups

could

be

performed

before

the

end

of

the

flight

of

the

center

stick

-LRB-

the

flight

duration

was

approximately

s

-RRB-

Later

queries

incorporated

more

measurements

of

the

flight

of

the

center

stick

and

therefore

more

accurate

predictions

of

the

state

of

the

task

However

the

system

required

substantial

structure

in

the

initial

training

to

achieve

this

performance

The

system

was

started

with

a

manually

generated

command

that

was

appropriate

for

open

loop

performance

of

the

task

Each

control

parameter

was

varied

systematically

to

explore

the

space

near

the

default

command

A

global

linear

model

was

made

of

this

initial

data

and

a

linear

controller

based

on

this

model

was

used

to

generate

an

initial

training

set

for

the

locally

weighted

system

-LRB-

of

approximately

hits

-RRB-

Learning

with

small

amounts

of

initial

data

was

not

possible

Furthermore

learning

based

on

just

an

inverse

model

was

prone

to

get

stuck

at

poor

levels

of

performance

and

to

repeat

the

same

mistakes

for

reasons

discussed

in

the

previous

section

To

eliminate

these

problems

we

also

experimented

with

learning

based

on

both

inverse

and

forward

models

After

a

command

is

generated

by

the

inverse

model

it

can

be

evaluated

using

a

forward

model

based

on

the

same

data

bxk

bf

-LRB-

xk

buk

-RRB-

-LRB-

-RRB-

Because

it

produces

a

local

linear

model

the

locally

weighted

regression

procedure

will

produce

estimates

of

the

derivatives

of

the

forward

model

with

respect

to

the

commands

as

part

of

the

estimated

parameter

vector

These

derivatives

can

be

used

to

find

a

correction

to

the

command

vector

that

reduces

errors

in

the

predicted

outcome

based

on

the

forward

model

@bf

@u

buk

bxk

xd

-LRB-

-RRB-

This

process

of

command

refinement

can

be

repeated

until

the

forward

model

no

longer

produces

accurate

predictions

of

the

outcome

which

will

happen

when

the

query

to

the

forward

model

requires

significant

extrapolation

from

the

current

database

The

distance

to

the

nearest

stored

data

point

can

be

used

as

a

crude

measure

of

the

validity

of

the

forward

model

estimate

We

investigated

this

method

for

incremental

learning

of

devil

sticking

in

simulations

However

the

outcome

did

not

meet

expectations

without

sufficient

initial

data

around

the

setpoint

the

algorithm

did

not

work

We

see

two

reasons

for

this

First

similar

to

the

pure

inverse

model

approach

the

inverse-forward

model

acts

as

a

one-step

deadbeat

controller

in

that

it

tries

to

eliminate

all

error

in

one

time

step

One-step

deadbeat

control

applies

large

commands

to

correct

for

deviations

from

the

setpoint

especially

in

the

presence

of

state

measurement

errors

The

workspace

bounds

and

command

bounds

of

our

devil

sticking

robot

limit

the

size

of

allowable

commands

Large

control

actions

may

also

be

less

accurate

or

robust

This

was

the

case

in

devil

sticking

where

a

large

control

action

tended

to

cause

the

center

stick

to

fly

in

a

random

direction

and

nothing

was

learned

from

that

hit

Second

the

ten

dimensional

input

space

is

large

and

even

if

experiences

are

uniformly

randomly

distributed

in

the

space

there

is

often

not

enough

data

near

a

particular

point

to

make

a

robust

inverse

or

forward

model

Thus

two

ingredients

had

to

be

added

to

the

devil

sticking

controller

First

the

controller

should

not

be

deadbeat

It

should

plan

to

attain

the

goal

using

multiple

control

actions

We

discuss

control

approaches

that

keep

commands

small

in

the

next

section

Second

the

control

must

increase

the

data

density

in

the

current

region

of

the

state-action

space

in

order

to

arrive

at

the

desired

goal

state

We

discuss

control

approaches

that

are

more

tightly

coupled

to

exploration

in

a

Section

Dynamic

Regulation

In

this

section

we

discuss

a

reformulation

of

temporally

dependent

control

tasks

to

avoid

the

problems

encountered

by

the

first

implementation

of

a

lazy

learner

for

robot

control

which

used

deadbeat

control

From

a

theoretical

point

of

view

it

is

often

not

possible

to

return

to

the

desired

setpoint

or

trajectory

in

one

step

an

attempt

to

do

so

would

require

actions

of

infinite

magnitude

or

cause

the

size

of

the

required

actions

to

grow

without

limit

One

step

deadbeat

control

will

fail

on

some

non-minimum

phase

systems

of

which

pole

balancing

is

one

example

-LRB-

Cannon

-RRB-

In

these

systems

one

must

move

away

from

the

goal

to

approach

it

later

In

the

case

of

the

cart-pole

system

the

cart

must

initially

move

away

from

the

target

position

so

that

the

pole

leans

in

the

direction

of

future

cart

motion

towards

the

target

This

maneuvering

avoids

having

the

pole

fall

backwards

as

the

cart

moves

toward

the

target

A

controller

can

perform

more

robustly

if

it

uses

smaller

magnitude

actions

and

returns

to

the

correct

state

or

trajectory

in

a

larger

number

of

steps

This

idea

is

posed

precisely

in

the

language

of

linear

quadratic

regulation

-LRB-

LQR

-RRB-

in

which

a

long

term

quadratic

cost

criterion

C

is

minimized

that

penalizes

both

state-errors

and

action

magnitudes

-LRB-

Stengel

-RRB-

C

X

t

-LRB-

x

-LRB-

t

-RRB-

xd

-RRB-

TQ

-LRB-

x

-LRB-

t

-RRB-

xd

-RRB-

uT

-LRB-

t

-RRB-

Ru

-LRB-

t

-RRB-

X

t

ffixT

-LRB-

t

-RRB-

Qffix

-LRB-

t

-RRB-

uT

-LRB-

t

-RRB-

Ru

-LRB-

t

-RRB-

-LRB-

-RRB-

where

Q

and

R

are

matrices

whose

elements

set

the

tradeoff

between

the

size

of

the

action

components

and

the

error

components

If

for

example

Q

and

R

were

identity

matrices

then

the

sum

of

squared

state

errors

and

the

sum

of

the

squared

action

components

would

be

minimized

Not

using

deadbeat

control

laws

implies

some

amount

of

lookahead

LQR

control

assumes

a

time

invariant

task

and

performs

an

infinite

amount

of

lookahead

Predictive

or

Receding

Horizon

control

design

techniques

look

N

steps

ahead

every

time

an

action

is

chosen

All

of

these

techniques

will

allow

larger

state

errors

to

reduce

the

size

of

the

control

signals

when

compared

to

deadbeat

methods

The

Linear

part

of

the

LQR

approach

is

a

local

linearization

of

the

forward

dynamics

of

the

task

We

can

take

advantage

of

the

locally

linear

state-transition

function

provided

by

locally

weighted

regression

-LRB-

Equation

-RRB-

x

-LRB-

t

-RRB-

xd

ffix

-LRB-

t

-RRB-

ss

bf

-LRB-

xd

ffix

-LRB-

t

-RRB-

u

-LRB-

t

-RRB-

-RRB-

ss

bf

-LRB-

xd

-RRB-

Affix

-LRB-

t

-RRB-

Bu

-LRB-

t

-RRB-

-LRB-

-RRB-

We

will

assume

that

-LRB-

xd

-RRB-

is

an

equilibrium

point

so

xd

bf

-LRB-

xd

-RRB-

and

we

have

the

following

linear

dynamics

ffix

-LRB-

t

-RRB-

Affix

-LRB-

t

-RRB-

Bu

-LRB-

t

-RRB-

-LRB-

-RRB-

The

optimal

action

with

respect

to

the

criteria

in

Equation

and

linear

dynamics

in

Equation

can

be

obtained

by

solution

of

a

matrix

equation

called

the

Ricatti

equation

-LRB-

Stengel

-RRB-

Assuming

the

locally

linear

model

provided

by

the

locally

weighted

regression

is

correct

the

optimal

action

u

is

u

-LRB-

R

BTPB

-RRB-

-LRB-

-RRB-

where

P

is

obtained

by

initialliy

setting

P

Q

and

then

running

the

following

iteration

to

convergence

P

Q+ATP

-LSB-

I

BR

-RSB-

-LRB-

-RRB-

This

rather

inscrutable

result

is

not

obvious

from

visual

inspection

but

follows

from

reasonably

elementary

algebra

and

calculus

that

can

be

found

in

almost

any

introductory

controls

text

We

recommend

-LRB-

Stengel

-RRB-

We

also

provide

a

very

simplified

self-contained

derivation

in

Appendix

A

The

long

term

cost

starting

from

state

xd

ffix

turns

out

to

be

ffixTPffix

Note

that

u

is

a

linear

function

of

the

state

x

in

Equation

u

Kffix

-LRB-

-RRB-

Linear

quadratic

regulation

has

useful

robustness

when

compared

to

deadbeat

controllers

even

if

the

underlying

linear

models

are

imprecise

-LRB-

Stengel

-RRB-

Implementation

of

Dynamic

Regulation

Devil

Sticking

II

Linear

quadratic

regulation

controller

design

permitted

successful

devil

sticking

It

did

require

manual

generation

of

training

data

to

estimate

the

matrices

of

the

local

linear

model

A

and

B.

However

once

the

local

linear

model

was

reliable

the

robot

had

a

complete

policy

-LRB-

i.e.

a

control

law

-RRB-

for

the

vicinity

of

the

local

linear

model

The

aggressiveness

of

the

control

law

could

be

controlled

by

choosing

Q

and

R

These

matrices

were

set

once

by

us

and

then

not

adjusted

during

learning

One

drawback

of

our

LQR

implementation

was

the

need

for

the

manual

search

for

an

equilibrium

point

The

robot

needed

to

be

told

a

nominal

hit

that

would

actually

send

the

devil

stick

to

the

other

hand

There

is

a

continuum

of

reasonable

equilibrium

points

but

our

formulation

required

the

arbitrary

selection

of

only

one

Furthermore

the

experimenter

did

not

know

in

advance

where

the

set

of

equilibrium

points

were

for

the

actual

machine

so

manual

search

for

equilibrium

points

was

a

difficult

task

given

the

five

dimensional

action

space

The

next

section

describes

a

new

procedure

to

search

for

equilibrium

points

Dynamic

Regulation

With

An

Unspecified

Setpoint

The

learning

task

is

considerably

harder

if

the

desired

setpoint

is

not

known

in

advance

and

instead

must

itself

be

optimized

to

achieve

some

higher

level

task

description

However

the

setpoint

of

the

task

can

be

manipulated

during

learning

to

improve

exploration

This

is

done

by

the

shifting

setpoint

algorithm

-LRB-

SSA

-RRB-

-LRB-

Schaal

and

Atkeson

-RRB-

SSA

attempts

to

decompose

the

control

problem

into

two

separate

control

tasks

on

different

time

scales

At

the

fast

time

scale

it

acts

as

a

dynamic

regulator

by

trying

to

keep

the

controlled

system

at

a

chosen

setpoint

On

a

slower

time

scale

the

setpoint

is

shifted

to

accomplish

a

desired

goal

SSA

uses

local

models

from

lazy

learning

and

can

be

viewed

as

an

approach

to

exploration

in

these

regulation

tasks

based

on

information

on

the

quality

of

predictions

provided

by

lazy

learning

Experiment

Design

with

Shifting

Setpoints

The

major

ingredient

of

the

SSA

is

a

statistical

self-monitoring

process

Whenever

the

current

location

in

input

space

has

obtained

a

sufficient

amount

of

experience

such

that

a

measure

of

confidence

rises

above

a

threshold

the

setpoint

is

shifted

in

the

direction

of

the

goal

until

the

confidence

falls

below

a

minimum

confidence

level

At

this

new

setpoint

location

the

learning

system

collects

new

experiences

The

shifting

process

is

repeated

until

the

goal

is

reached

In

this

way

the

SSA

builds

a

narrow

tube

of

data

support

in

which

it

knows

the

world

This

data

builds

the

basis

for

the

first

success

of

the

regulator

controller

Subsequently

the

learned

model

can

be

used

for

more

sophisticated

control

algorithms

for

planning

or

for

further

exploration

Dynamic

Regulation

With

An

Unspecified

Setpoint

Devil

Sticking

III

The

SSA

method

was

tested

on

the

devil

sticking

juggling

task

-LRB-

Schaal

and

Atkeson

b

-RRB-

In

this

case

it

had

the

following

steps

Regardless

of

the

poor

juggling

quality

of

the

robot

-LRB-

i.e.

at

most

two

or

three

hits

per

trial

-RRB-

the

SSA

made

the

robot

repeat

these

initial

actions

with

small

random

perturbations

until

a

cloud

of

data

was

collected

somewhere

in

the

state-action

space

for

each

hand

An

abstract

illustration

for

this

is

given

in

Figure

Each

point

in

the

data

cloud

of

each

hand

was

used

as

a

candidate

for

a

setpoint

of

the

corresponding

hand

by

trying

to

predict

its

output

from

its

input

with

locally

weighted

regression

The

point

achieving

the

narrowest

local

confidence

interval

became

the

setpoint

of

the

hand

and

a

linear

quadratic

regulator

was

calculated

for

its

local

linear

model

estimated

using

locally

weighted

regression

By

means

of

these

controllers

the

amount

of

data

around

the

setpoints

could

quickly

be

increased

until

the

quality

of

the

local

models

exceeded

a

statistical

threshold

-LRB-

Figure

-RRB-

-LRB-

Atkeson

et

al.

-RRB-

At

this

point

the

setpoints

were

gradually

shifted

towards

the

goal

setpoints

until

the

statistical

confidence

in

the

predictions

made

by

the

local

model

again

fell

below

a

threshold

-LRB-

Figure

-RRB-

The

SSA

iterated

by

collecting

data

in

the

new

regions

of

the

workspace

until

the

setpoints

could

be

shifted

again

The

procedure

terminated

when

the

goal

was

reached

leaving

a

ridge

of

data

in

the

state-action

space

-LRB-

Figure

-RRB-

r

ightleftD

at

a

DensityTh

row

to

leftT

hrow

to

rightGlG

oal

S

tate

of

R

ig

ht

HandGrGrGlr

ightleft

Jugg

ling

f

ails

since

no

local

model

in

t

his

reg

ionGlr

ightleftGrGlr

ightleftGr

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

-LRB-

d

-RRB-

G

oal

S

tate

of

Le

f

t

Hand

Figure

Abstract

illustration

on

how

the

SSA

algorithm

collects

data

in

space

-LRB-

a

-RRB-

sparse

data

after

the

first

few

hits

-LRB-

b

-RRB-

high

local

data

density

due

to

local

control

in

this

region

-LRB-

c

-RRB-

increased

data

density

on

the

way

to

the

goals

due

to

shifting

the

setpoints

-LRB-

d

-RRB-

ridge

of

data

density

after

the

goal

was

reached

The

SSA

was

tested

in

a

noise

corrupted

simulation

and

on

the

real

robot

Each

attempt

to

juggle

the

devil

stick

is

called

a

trial

which

consists

of

a

series

of

left

and

right

handed

hits

Each

series

of

trials

that

begins

with

the

lazy

learning

system

in

its

initial

state

is

referred

to

as

a

run

Our

measure

of

performance

is

the

number

of

hits

per

trial

In

the

simulation

it

takes

on

average

trials

before

the

setpoint

of

each

hand

has

moved

close

enough

to

the

other

hand

's

setpoint

This

is

slightly

better

performance

than

with

the

real

robot

At

that

point

a

breakthrough

occurs

and

afterwards

the

simulated

robot

rarely

drops

the

devilstick

At

this

time

about

data

points

-LRB-

hits

-RRB-

have

been

collected

in

memory

The

real

robot

's

learning

performance

is

qualitatively

the

same

as

that

of

the

simulated

robot

Due

to

stronger

nonlinearities

and

unknown

noise

sources

the

actual

robot

takes

more

trials

to

accomplish

a

steady

juggling

pattern

We

show

three

typical

learning

runs

for

the

actual

robot

in

Figure

We

do

not

show

averages

of

these

learning

runs

because

averaged

runs

show

a

gradual

increase

in

performance

which

is

unlike

any

individual

learning

run

which

show

sudden

increases

in

performance

Peak

performance

of

the

robot

was

more

than

consecutive

hits

-LRB-

minutes

of

continuous

juggling

-RRB-

Limits

For

Linear

Quadratic

Regulation

Control

laws

based

on

linear

quadratic

regulator

designs

are

not

useful

if

the

task

requires

operation

outside

a

locally

linear

region

The

LQR

controller

may

actually

be

unstable

For

of

Hits

per

TrialTrial

Number

Run

IRun

IIRun

III

Figure

Learning

curves

of

devil

sticking

for

three

runs

example

the

following

one

dimensional

system

with

a

one

dimensional

action

xk

uk

-LRB-

-RRB-

has

a

local

linear

model

at

the

origin

-LRB-

x

-RRB-

of

A

and

B

-LRB-

all

matrices

are

for

this

one

dimensional

problem

-RRB-

For

the

optimization

criteria

Q

and

R

and

the

Ricatti

equation

-LRB-

Equations

and

-RRB-

gives

K

For

a

goal

of

moving

to

the

origin

-LRB-

xd

-RRB-

this

linear

control

law

is

unstable

for

x

larger

than

because

the

actions

u

are

too

large

This

means

that

the

LQR

optimal

action

actually

increases

the

error

ffix

if

the

error

is

already

larger

than

This

limitation

of

linear

quadratic

regulation

motivates

us

to

explore

full

dynamic

programming

based

policy

design

approaches

which

are

described

in

the

next

section

Figure

compares

the

LQR

based

control

law

and

the

control

law

based

on

full

dynamic

programming

using

the

same

model

and

optimization

criteria

Note

that

the

shifting

setpoint

algorithm

can

provide

the

initial

training

data

for

these

more

complex

approaches

Nonlinear

Optimal

Control

In

more

general

control

design

we

must

accommodate

a

more

general

formulation

of

the

cost

function

or

criterion

to

optimize

and

also

move

from

local

control

laws

based

on

a

small

number

of

local

models

to

more

global

control

laws

based

on

many

local

models

We

now

need

to

learn

not

just

a

local

model

of

the

task

but

many

local

models

of

the

task

distributed

throughout

the

task

space

We

will

first

discuss

a

more

general

formulation

of

cost

functions

We

are

given

a

cost

function

for

each

step

which

is

known

by

the

controller

g

-LRB-

t

-RRB-

G

-LRB-

x

-LRB-

t

-RRB-

u

-LRB-

t

-RRB-

t

-RRB-

-LRB-

-RRB-

The

task

is

to

minimize

one

of

the

following

expressions

X

t

g

-LRB-

t

-RRB-

or

tmax

X

t

g

-LRB-

t

-RRB-

or

X

t

tg

-LRB-

t

-RRB-

where

or

lim

n

n

nX

t

g

-LRB-

t

-RRB-

The

attractive

aspect

of

these

formulations

is

their

generality

All

of

the

previously

described

control

formulations

are

special

cases

of

at

least

one

of

these

For

example

the

quadratic

one

step

cost

defined

by

Q

and

R

can

be

viewed

as

a

local

quadratic

model

of

g

-LRB-

t

-RRB-

State

x

Command

u

Figure

Solid

line

optimal

action

based

on

dynamic

programming

-LRB-

DP

-RRB-

using

the

nonlinear

model

dashed

line

optimal

command

based

on

a

LQR

design

using

a

single

linear

forward

model

at

the

origin

Although

in

both

cases

the

optimization

criterion

is

the

same

and

the

LQR

and

DP-based

control

laws

agree

for

small

x

the

LQR

control

law

is

linear

and

does

not

take

into

account

the

nonlinear

dynamics

of

the

task

for

large

x

The

delayed

rewards

nature

of

these

tasks

means

that

actions

we

choose

at

time

t

do

not

only

affect

the

quality

of

the

immediate

reward

but

also

affect

the

next

and

all

subsequent

states

and

in

so

doing

affect

the

future

rewards

attainable

This

leads

to

computational

difficulties

in

the

general

case

A

large

literature

on

such

learning

control

problems

has

sprung

up

in

recent

years

with

the

general

name

of

reinforcement

learning

Overviews

may

be

found

in

-LRB-

Sutton

Barto

et

al.

Watkins

Barto

et

al.

Moore

and

Atkeson

-RRB-

In

this

paper

we

will

restrict

discussion

to

the

applications

of

lazy

learning

to

these

problems

Again

we

proceed

by

learning

an

empirical

forward

model

bxk

bf

-LRB-

xk

buk

-RRB-

A

generalpurpose

solution

can

be

obtained

by

discretizing

state-space

into

a

multidimensional

array

of

small

cells

and

performing

a

dynamic

programming

method

-LRB-

Bellman

Bertsekas

and

Tsitsiklis

-RRB-

such

as

value

iteration

or

policy

iteration

to

produce

two

things

A

value

function

V

-LRB-

x

-RRB-

mapping

cells

onto

the

minimum

possible

sum

of

future

costs

if

one

starts

in

that

cell

A

policy

u

-LRB-

x

-RRB-

mapping

cells

onto

the

optimal

action

to

take

in

that

cell

Value

iteration

can

be

used

in

conjunction

with

learning

a

world

model

However

it

is

extremely

computationally

expensive

For

a

fixed

quantization

level

the

cost

is

exponential

in

the

dimensionality

of

the

state

variables

For

a

D

dimensional

state

space

and

action

space

and

a

grid

resolution

of

R

for

both

states

and

actions

one

value

iteration

pass

would

require

evaluations

of

the

forward

model

The

most

computationally

intensive

version

would

perform

several

cycles

of

value

iteration

after

every

update

of

the

memory

base

Less

expensive

forms

of

dynamic

programming

would

normally

perform

value

iteration

only

at

the

end

of

each

trial

-LRB-

as

we

do

in

the

example

in

Section

-RRB-

or

as

an

incremental

parallel

process

-LRB-

Sutton

Moore

and

Atkeson

Peng

and

Williams

-RRB-

A

Simulation

Example

The

Puck

We

illustrate

this

form

of

learning

by

means

of

a

simple

simulated

example

Figure

depicts

a

frictionless

puck

on

a

bumpy

surface

whose

objective

is

to

drive

itself

up

the

hill

to

a

goal

region

in

the

minimum

number

of

time

steps

The

state

x

-LRB-

x

x

-RRB-

is

two-dimensional

and

must

lie

in

the

region

x

x

x

denotes

the

horizontal

position

of

the

puck

in

Figure

The

action

u

a

is

one-dimensional

and

represents

the

horizontal

force

applied

to

the

puck

Actions

are

constrained

such

that

a

The

goal

region

is

the

rectangle

x

x

The

surface

upon

which

the

puck

slides

has

the

following

height

as

a

function

of

x

H

-LRB-

x

-RRB-

-LRB-

x

if

x

x

if

x

-LRB-

-RRB-

The

puck

's

dynamics

are

given

by

x

a

M

q

-LRB-

H

-LRB-

x

-RRB-

-RRB-

gH

-LRB-

x

-RRB-

-LRB-

H

-LRB-

x

-RRB-

-RRB-

-LRB-

-RRB-

where

M

and

g

This

equation

is

integrated

using

x

-LRB-

t

-RRB-

x

-LRB-

t

-RRB-

h

x

-LRB-

t

-RRB-

-LRB-

t

-RRB-

x

-LRB-

t

-RRB-

x

-LRB-

t

-RRB-

h?x

-LRB-

t

-RRB-

-LRB-

-RRB-

where

h

is

the

simulation

time

step

Because

of

gravity

there

is

a

region

near

the

center

of

the

hill

at

which

the

maximum

rightward

thrust

is

insufficient

to

accelerate

up

the

slope

If

the

goal

region

is

at

the

hill-top

a

strategy

that

proceeded

by

greedily

choosing

actions

to

thrust

towards

the

goal

would

get

stuck

This

is

made

clearer

in

Figure

a

state

transition

diagram

The

puck

's

state

has

two

components

the

position

and

velocity

The

hairs

show

the

next

state

of

the

puck

if

it

were

to

thrust

rightwards

with

the

maximum

legal

force

of

Newtons

for

s

At

the

center

of

state-space

even

when

this

thrust

is

applied

the

puck

velocity

decreases

and

it

eventually

slides

leftwards

The

optimal

solution

for

the

puck

task

depicted

in

Figure

is

to

initially

thrust

away

from

the

goal

gaining

negative

velocity

until

it

is

on

the

far

left

of

the

diagram

Then

it

thrusts

hard

right

to

build

up

sufficient

energy

to

reach

the

top

of

the

hill

We

explored

two

implementations

of

adaptive

controllers

one

of

which

used

lazy

learning

techniques

ffl

Implementation

-LRB-

Grid

Based

-RRB-

Conventional

Discretization

This

used

the

conventional

reinforcement

learning

strategy

of

discretizing

state

space

into

a

grid

of

cells

for

the

forward

model

and

value

function

The

reinforcement

learning

N

mg

F

m

F

Goal

Position

-LRB-

x

-RRB-

Figure

A

frictionless

puck

acted

on

by

gravity

and

a

horizontal

thruster

The

puck

must

get

to

the

goal

as

quickly

as

possible

There

are

bounds

on

the

maximum

thrust

Goal

Position

Velocity

Figure

The

state

transition

diagram

for

a

puck

that

constantly

thrusts

right

with

maximum

thrust

GoalStart

Position

Velocity

Figure

The

minimum-time

path

from

start

to

goal

for

the

puck

on

the

hill

The

optimal

value

function

is

shown

by

the

background

dots

The

shorter

the

time

to

goal

the

larger

the

black

dot

Notice

the

discontinuity

at

the

escape

velocity

Trial

Trial

Trial

Trial

Trial

Grid

LWR

Figure

The

first

five

trials

for

both

implementations

of

the

puck

controller

algorithm

was

chosen

to

be

as

efficient

as

possible

-LRB-

i.e.

in

terms

of

data

needed

for

convergence

-RRB-

given

that

we

were

working

with

a

fixed

discretization

All

transitions

between

cells

experienced

by

the

system

were

remembered

in

a

discrete

state

transition

model

A

learning

algorithm

similar

to

Dyna

-LRB-

Sutton

-RRB-

was

used

with

full

value

iteration

carried

out

on

the

discrete

model

every

time-step

Exploration

was

achieved

by

assuming

any

unvisited

state

had

a

future

cost

of

zero

The

action

which

is

onedimensional

was

discretized

to

five

levels

f

N

ffl

Implementation

-LRB-

LWR

-RRB-

Lazy

Forward

Model

The

second

implementation

was

the

same

as

the

first

except

that

transitions

between

cells

were

filled

in

by

predictions

from

a

locally

weighted

regression

forward

model

x

-LRB-

t

-RRB-

bf

-LRB-

x

-LRB-

t

-RRB-

u

-LRB-

t

-RRB-

-RRB-

Thus

unlike

implementation

many

discrete

transitions

that

had

not

been

physically

experienced

were

stored

in

the

transition

table

by

extrapolation

from

the

actual

experiences

Also

the

lazy

model

supported

a

higher

resolution

representation

in

areas

where

many

experiences

had

been

collected

The

value

function

was

represented

by

a

table

in

both

implementations

The

experimental

domain

is

a

simple

one

but

its

empirical

behavior

demonstrates

an

important

point

A

lazy

forward

model

in

combination

with

value

iteration

can

dramatically

reduce

the

amount

of

actual

data

needed

during

learning

The

graphs

of

the

first

five

trajectories

of

the

two

experiments

are

shown

in

Figure

The

steps

per

trial

for

both

implementations

are

shown

in

Figure

The

best

possible

number

of

steps

per

trial

is

The

implementation

using

the

locally

weighted

regression

forward

model

learns

much

faster

in

terms

of

trials

than

the

implementation

using

the

grid

The

lazy

model

based

implementation

also

requires

approximately

two

orders

of

magnitude

fewer

steps

in

order

to

reach

optimal

performance

For

example

after

trial

the

grid

based

implementation

has

executed

total

steps

more

than

the

optimal

required

when

all

trials

are

combined

while

the

lazy

forward

model

based

implementation

has

executed

only

suboptimal

steps

Since

we

did

not

include

any

random

noise

in

this

simulation

these

numbers

are

deterministic

The

spikes

in

Figure

are

due

to

the

severe

nonlinearity

of

this

problem

where

small

errors

in

the

policy

may

lead

to

the

puck

failing

to

have

enough

energy

to

get

to

the

goal

In

this

case

the

puck

slides

back

down

and

must

perform

another

orbit

of

the

start

point

in

state

space

before

reaching

the

goal

The

lack

of

random

sensor

or

actuator

noise

Forward

ModelTrialSteps

Figure

Top

Steps

per

trial

for

a

grid

based

forward

model

Bottom

Steps

per

trial

for

an

LWR

based

forward

model

Note

the

difference

in

vertical

scales

makes

the

problem

unrealistically

easy

for

both

approaches

We

expect

the

benefits

of

a

lazy

model

over

the

standard

grid

model

to

carry

over

to

the

stochastic

case

The

computational

costs

of

this

kind

of

control

are

considerable

Although

it

is

not

necessary

to

gather

data

from

every

part

of

the

state

space

when

generalization

occurs

with

a

model

the

simple

form

of

value

iteration

requires

a

multidimensional

discretization

for

computing

the

value

function

Several

researchers

are

investigating

methods

for

reducing

the

cost

of

value

iteration

when

a

model

has

been

learned

-LRB-

e.g.

-LRB-

Moore

Mahadevan

Atkeson

-RRB-

-RRB-

Exploration

The

approach

we

have

described

does

not

explicitly

explore

If

the

learned

model

contains

serious

errors

a

part

of

state

space

that

wrongly

looks

unrewarding

will

never

be

visited

by

the

real

system

so

the

model

will

never

be

updated

On

the

other

hand

we

do

not

want

the

system

to

explore

every

part

of

state

space

explicitly

the

supposed

advantage

of

lazy

learning

based

function

approximation

is

the

ability

to

generalize

parts

of

the

model

without

explicitly

performing

an

action

To

resolve

this

dilemma

a

number

of

useful

exploration

heuristics

can

be

used

all

based

on

the

idea

that

it

is

worth

exploring

only

where

there

is

little

confidence

in

the

empirical

model

-LRB-

Sutton

Kaelbling

Moore

and

Atkeson

Cohn

et

al.

-RRB-

Lazy

Learning

of

Models

Pros

and

Cons

Lazy

learning

of

models

leads

to

new

forms

of

autonomous

control

The

control

algorithms

explicitly

perform

empirical

nonlinear

modeling

as

well

as

simultaneously

designing

policies

without

a

strong

commitment

to

a

model

structure

or

controller

structure

in

advance

Parametric

modeling

approaches

such

as

polynomial

regression

multi-layer

sigmoidal

neural

networks

and

projection

pursuit

regression

all

make

a

strong

commitment

to

a

model

structure

and

new

training

data

has

a

global

effect

on

the

learned

function

Locally

weighted

learning

only

assumes

local

smoothness

This

section

discusses

the

strengths

and

weaknesses

of

a

local

and

lazy

modeling

approach

in

the

context

of

control

Stanfill

and

Waltz

-LRB-

-RRB-

provide

a

similar

discussion

for

lazy

approaches

to

classification

Benefits

of

Lazy

Learning

of

Models

ffl

Automatic

empirical

local

linear

models

Locally

weighted

linear

regression

returns

a

local

linear

map

It

performs

the

job

of

an

engineer

who

is

trying

to

empirically

linearize

the

system

around

a

region

of

interest

It

is

not

difficult

for

neural

net

representations

to

provide

a

local

linear

map

too

but

other

approximators

such

as

straightforward

nearest

neighbor

or

the

original

version

of

CMAC

-LRB-

Albus

Miller

-RRB-

are

less

reliable

in

their

estimation

of

local

gradients

because

predicted

surfaces

are

not

smooth

Additionally

if

the

input

data

distribution

is

not

too

non-uniform

it

can

be

shown

that

the

linearizations

returned

by

locally

weighted

learning

accomplish

a

low-bias

estimate

of

the

true

gradient

with

fewer

data

points

than

required

for

a

low-bias

prediction

of

a

query

-LRB-

Hastie

and

Loader

-RRB-

ffl

Automatic

confidence

estimations

Locally

weighted

regression

can

also

be

modified

to

return

a

confidence

interval

along

with

its

prediction

This

can

be

done

heuristically

with

the

local

density

of

the

data

providing

an

uncertainty

estimate

-LRB-

Moore

-RRB-

or

by

making

sensible

statistical

assumptions

-LRB-

Schaal

and

Atkeson

Cohn

et

al.

-RRB-

In

either

case

this

has

been

shown

empirically

to

dramatically

reduce

the

amount

of

exploration

needed

when

the

uncertainty

estimates

guide

the

experiment

design

The

cost

of

estimating

uncertainty

with

locally

weighted

methods

is

small

Nonlinear

parametric

representations

such

as

multi-layer

sigmoidal

neural

networks

can

also

be

adapted

to

return

confidence

intervals

-LRB-

MacKay

Pomerleau

-RRB-

but

approximations

are

required

and

the

computational

cost

is

larger

Worse

parametric

models

-LRB-

e.g.

global

polynomial

regression

-RRB-

that

predict

confidence

statistically

are

typically

assuming

that

the

true

world

can

be

perfectly

modeled

by

at

least

one

set

of

parameter

values

If

this

assumption

is

violated

then

the

confidence

intervals

are

difficult

to

interpret

ffl

Adding

new

data

to

a

lazy

model

is

cheap

For

a

lazy

model

adding

a

new

data

point

means

simply

inserting

it

into

the

data

base

ffl

One-shot

learning

Lazy

models

do

not

need

to

be

repeatedly

exposed

to

the

same

data

to

learn

it

A

consequence

of

this

rapid

learning

is

that

errors

are

not

repeated

and

can

be

eliminated

much

more

quickly

than

approaches

that

incrementally

update

parameters

Nonlinear

parametric

models

can

be

trained

by

-RRB-

exposing

the

model

to

a

new

data

point

only

once

-LRB-

e.g.

-LRB-

Jordan

and

Jacobs

Kuperstein

-RRB-

-RRB-

or

-RRB-

by

storing

the

data

in

a

database

and

cycling

through

the

training

data

repeatedly

In

case

much

more

data

must

be

collected

since

the

training

effect

of

each

data

point

is

small

This

leads

to

slower

learning

since

real

robot

movements

take

time

and

to

increased

wear-and-tear

on

the

robot

or

industrial

process

that

is

to

be

controlled

In

case

a

lazy

learning

approach

has

been

adopted

and

one

must

then

evaluate

the

relative

benefits

of

complex

and

simple

local

models

ffl

Non-linear

yet

no

danger

of

local

minima

in

function

approximation

Locally

weighted

regression

can

fit

a

wide

range

of

complex

non-linear

functions

and

finds

the

best

fit

directly

without

requiring

any

gradient

descent

There

are

no

dangers

of

the

model

learner

becoming

stuck

in

a

local

optimum

In

contrast

training

nonlinear

parametric

models

can

get

stuck

in

local

minima

However

some

of

the

control

law

design

algorithms

we

have

surveyed

can

become

stuck

-LRB-

Moore

Jordan

and

Rumelhart

-RRB-

The

inverse-model

method

can

become

stuck

with

non-monotonic

or

highly

noisy

systems

The

shifting

setpoint

algorithm

can

become

stuck

in

principle

although

this

has

not

yet

occurred

in

practice

ffl

Avoids

interference

Lazy

modeling

is

insensitive

to

what

task

it

is

currently

learning

or

if

the

data

distribution

changes

In

contrast

nonlinear

parametric

models

trained

incrementally

with

gradient

descent

eventually

forget

old

experiences

and

concentrate

representational

power

on

new

experiences

Drawbacks

of

Lazy

Learning

of

Models

Here

we

consider

the

disadvantages

of

lazy

learning

that

may

be

encountered

under

some

circumstances

and

we

also

point

out

promising

directions

for

addressing

them

ffl

Lookup

costs

increase

with

the

amount

of

training

data

Memory

and

computation

costs

increase

with

the

amount

of

data

Memory

costs

increase

linearly

with

the

amount

of

data

and

are

not

generally

a

problem

Any

algorithm

that

avoids

storing

redundant

data

would

greatly

reduce

the

amount

of

memory

needed

and

one

can

also

discard

data

perhaps

selected

according

to

predictive

usefulness

redundancy

or

age

-LRB-

Atkeson

et

al.

-RRB-

Computational

costs

are

more

serious

For

a

fixed

amount

of

computation

a

single

processor

can

process

a

limited

number

of

training

data

points

There

are

several

solutions

to

this

problem

-LRB-

Atkeson

et

al.

-RRB-

The

database

can

be

structured

so

that

the

most

relevant

data

points

are

accessed

first

or

so

that

close

approximations

to

the

output

predicted

by

locally

weighted

regression

can

be

obtained

without

explicitly

visiting

every

point

in

the

database

There

are

a

surprisingly

large

number

of

algorithms

available

for

doing

this

mostly

based

on

kd-trees

-LRB-

Preparata

and

Shamos

Omohundro

Moore

Grosse

Quinlan

Omohundro

Deng

and

Moore

-RRB-

ffl

Is

the

curse

of

dimensionality

a

problem

for

lazy

learning

for

control

The

curse

of

dimensionality

is

the

exponential

dependence

of

needed

resources

on

dimensionality

found

in

many

learning

and

planning

approaches

The

methods

we

have

discussed

so

far

can

handle

a

wide

class

of

problems

On

the

other

hand

it

is

well

known

that

without

strong

constraints

on

the

class

of

functions

being

approximated

learning

with

many

input

dimensions

will

not

successfully

approximate

a

particular

function

over

the

entire

space

of

potential

inputs

unless

the

data

set

is

unrealistically

large

This

is

an

apparently

serious

problem

for

multivariate

control

using

locally

weighted

learning

and

raises

the

question

as

to

why

the

examples

given

in

this

paper

worked

Happily

it

is

actually

quite

difficult

to

think

of

useful

tasks

that

require

the

system

to

have

an

accurate

model

over

the

entire

input

space

-LRB-

Albus

-RRB-

Indeed

for

a

robot

of

more

than

say

eight

degrees

of

freedom

it

will

not

be

possible

for

it

to

get

into

every

significantly

different

configuration

even

once

in

its

entire

lifetime

Many

tasks

require

high

accuracy

only

in

low-dimensional

manifolds

of

input

space

or

thin

slices

around

those

manifolds

In

some

cases

these

may

be

clumps

around

the

desired

goal

value

of

stationary

tasks

For

example

in

devil

sticking

the

robot

needs

to

gain

highly

accurate

expertise

only

in

the

vicinity

of

stable

juggling

patterns

Another

common

task

involves

the

system

spending

most

of

its

life

traveling

along

a

number

of

important

trajectories

highways

through

state

space

in

which

case

expertise

need

only

be

clustered

in

these

regions

In

general

the

curse

of

dimensionality

may

not

be

dangerous

for

tasks

whose

solution

lies

in

a

low-dimensional

manifold

or

a

thin

slice

even

if

the

number

of

state

variables

and

control

inputs

is

several

times

larger

In

any

event

we

expect

the

performance

of

locally

weighted

regression

to

be

as

good

as

any

other

method

as

the

dimensionality

of

the

problem

increases

as

locally

weighted

learning

can

become

global

if

necessary

to

emulate

global

models

and

can

become

global

or

local

in

particular

directions

to

emulate

projection

pursuit

models

-LRB-

e.g.

the

distance

function

can

be

set

to

choose

a

projection

direction

for

example

but

for

multiple

projection

directions

multiple

distance

functions

must

be

used

in

additive

locally

weighted

fits

-RRB-

-LRB-

Friedman

and

Stuetzle

-RRB-

We

expect

locally

weighted

learning

to

degrade

gracefully

as

the

problem

dimensionality

increases

ffl

Lazy

learning

depends

on

having

good

representations

already

selected

Good

representational

choices

-LRB-

i.e.

choices

of

the

elements

of

the

state

and

control

vectors

etc.

-RRB-

can

dramatically

speed

up

learning

or

make

learning

possible

at

all

Feature

selection

and

scaling

algorithms

are

a

crude

form

of

choosing

new

representations

-LRB-

Atkeson

et

al.

-RRB-

However

we

have

not

solved

the

representation

problem

and

locally

weighted

learning

and

all

other

machine

learning

approaches

depend

on

prior

representational

decisions

Conclusions

This

paper

has

explored

methods

for

using

lazy

learning

to

learn

task

models

for

control

emphasizing

how

forward

and

inverse

learned

models

can

be

used

The

implementations

all

used

lazy

models

The

last

section

discussed

in

more

detail

the

pros

and

cons

of

lazy

learning

as

the

specific

choice

of

model

learner

There

is

little

doubt

that

these

advances

can

be

converted

into

general

purpose

software

packages

for

the

benefit

of

robotics

and

process

control

But

it

should

also

be

understood

that

we

are

still

a

considerable

way

from

full

autonomy

A

human

programmer

has

to

decide

what

the

state

and

action

variables

are

for

a

problem

how

the

task

should

be

specified

and

what

class

of

control

task

it

is

The

engineering

of

real-time

systems

sensors

and

actuators

is

still

required

A

human

must

take

responsibility

for

safety

and

supervision

of

the

system

Thus

at

this

stage

if

we

are

given

a

problem

the

relative

effectiveness

of

learning

control

measured

as

the

proportion

of

human

effort

eliminated

is

heavily

dependent

on

problemspecific

issues

Appendix

A

Simple

Linear

Quadratic

Regulator

deriva

tion

This

appendix

provides

a

simplified

self-contained

introduction

to

LQR

control

for

readers

who

wish

to

understand

the

ideas

behind

Equations

and

Assume

a

scalar

state

and

action

and

assume

that

desired

state

and

action

are

zero

-LRB-

xd

ud

-RRB-

Assume

linear

dynamics

xk

axk

buk

-LRB-

-RRB-

where

a

and

b

are

constants

Define

V

k

-LRB-

x

-RRB-

to

be

the

minimum

possible

sum

of

future

costs

starting

from

state

x

assuming

we

are

at

time-step

k.

Assume

the

system

stops

at

time

k

N

and

the

stopping

cost

is

For

all

other

steps

-LRB-

i.e.

k

N

-RRB-

the

cost

is

V

k

-LRB-

x

-RRB-

N

X

j

k

assuming

uk

uk

uN

chosen

optimally

-LRB-

-RRB-

V

k

-LRB-

x

-RRB-

can

be

defined

inductively

V

N

-LRB-

x

-RRB-

-LRB-

-RRB-

V

k

-LRB-

x

-RRB-

arg

min

uk

V

k

-LRB-

xk

-RRB-

-LRB-

-RRB-

by

the

principal

of

optimality

which

says

that

your

best

bet

for

minimal

costs

is

to

minimize

over

your

first

step

for

the

cost

of

that

step

plus

the

minimum

possible

costs

of

future

steps

We

will

now

prove

by

induction

that

V

k

-LRB-

x

-RRB-

is

a

quadratic

in

x

with

the

quadratic

coefficient

dependent

on

k

V

k

-LRB-

x

-RRB-

for

some

pN

ffl

Base

case

pN

q

from

Equation

ffl

Inductive

step

Assume

V

k

-LRB-

x

-RRB-

pk

we

'll

prove

V

k

-LRB-

x

-RRB-

for

some

pk

From

here

on

all

that

remains

is

algebra

We

begin

with

Equation

in

which

we

replace

xk

with

axk

buk

from

Equation

V

k

-LRB-

x

-RRB-

arg

min

uk

V

k

-LRB-

axk

buk

-RRB-

-LRB-

-RRB-

Then

we

use

the

inductive

assumption

V

k

-LRB-

x

-RRB-

pk

V

k

-LRB-

x

-RRB-

arg

min

uk

pk

-LRB-

axk

buk

-RRB-

-LRB-

-RRB-

Next

we

simplify

with

three

new

variables

ff

V

k

-LRB-

x

-RRB-

arg

min

uk

xkuk

where

-LRB-

-RRB-

ff

q

pk

-LRB-

-RRB-

pk

ab

-LRB-

-RRB-

r

pk

-LRB-

-RRB-

To

minimize

Equation

with

respect

to

u

we

differentiate

and

set

to

zero

the

bracketed

expression

giving

x

u?k

-LRB-

-RRB-

where

u?k

is

the

optimal

action

Thus

u?k

-LRB-

fl

-RRB-

xk

-LRB-

-RRB-

Since

u?k

minimizes

Equation

we

have

V

k

-LRB-

x

-RRB-

xu?k

-LRB-

u?k

-RRB-

-LRB-

-RRB-

So

from

Equation

V

k

-LRB-

x

-RRB-

x

-LRB-

fl

-RRB-

x

-LRB-

fl

-RRB-

ff

fl

fl

ff

fl

-LRB-

-RRB-

so

that

we

have

shown

V

k

-LRB-

x

-RRB-

where

pk

ff

fl

-LRB-

-RRB-

Inserting

back

the

substitutions

of

Equations

into

Equations

and

u?k

pk

ab

r

pk

xk

-LRB-

-RRB-

V

k

-LRB-

x

-RRB-

pk

q

pk

r

pk

-LRB-

-RRB-

Assuming

that

there

are

N

k

steps

remaining

to

compute

the

cost-to-go

from

state

x

we

set

p

q

and

then

iterate

the

assignment

p

q

-LRB-

r

-RRB-

a

total

of

N

k

times

As

N

k

becomes

large

p

converges

to

a

constant

value

-LRB-

not

proven

here

-RRB-

This

gives

the

cost-to-go

value

function

of

assuming

that

the

system

will

run

forever

Acknowledgments

Support

for

C.

Atkeson

and

S.

Schaal

was

provided

by

the

ATR

Human

Information

Processing

Research

Laboratories

Support

for

C.

Atkeson

was

provided

under

Air

Force

Office

of

Scientific

Research

grant

and

by

a

National

Science

Foundation

Presidential

Young

Investigator

Award

Support

for

S.

Schaal

was

provided

by

the

German

Scholarship

Foundation

and

the

Alexander

von

Humboldt

Foundation

Support

for

A.

Moore

was

provided

by

the

U.

K.

Science

and

Engineering

Research

Council

NSF

Research

Initiation

Award

and

a

Research

Gift

from

the

Corporation

References

Aha

D.

W.

and

Salzberg

S.

L.

-LRB-

-RRB-

Learning

to

catch

Applying

nearest

neighbor

algorithms

to

dynamic

control

tasks

In

Proceedings

of

the

Fourth

International

Workshop

on

Artificial

Intelligence

and

Statistics

pages

-LCB-

Ft.

Lauderdale

FL

Albus

J.

S.

-LRB-

-RRB-

Brains

Behaviour

and

Robotics

BYTE

Books

McGraw-Hill

Atkeson

C.

G.

-LRB-

-RRB-

Using

local

models

to

control

movement

In

Touretzky

D.

S.

editor

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Atkeson

C.

G.

-LRB-

-RRB-

Using

local

trajectory

optimizers

to

speed

up

global

optimization

in

dynamic

programming

In

Hanson

S.

J.

Cowan

J.

D.

and

Giles

C.

L.

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Atkeson

C.

G.

Moore

A.

W.

and

Schaal

S.

-LRB-

-RRB-

Locally

weighted

learning

Submitted

to

the

Artificial

Intelligence

Review

special

issue

on

Lazy

Learning

Baird

L.

C.

and

Klopf

A.

H.

-LRB-

-RRB-

Reinforcement

learning

with

high-dimensional

continuous

actions

Technical

Report

Wright

Laboratory

Wright-Patterson

Air

Force

Base

Ohio

Barto

A.

G.

Bradtke

S.

J.

and

Singh

S.

P.

-LRB-

-RRB-

Learning

to

act

using

real-time

dynamic

programming

Artificial

Intelligence

-LRB-

-RRB-

-LCB-

Barto

A.

G.

Sutton

R.

S.

and

Watkins

C.

J.

C.

H.

-LRB-

-RRB-

Learning

and

Sequential

Decision

Making

In

Gabriel

M.

and

Moore

J.

W.

editors

Learning

and

Computational

Neuroscience

pages

-LCB-

MIT

Press

Cambridge

MA

Bellman

R.

E.

-LRB-

-RRB-

Dynamic

Programming

Princeton

University

Press

Princeton

NJ

Bertsekas

D.

P.

and

Tsitsiklis

J.

N.

-LRB-

-RRB-

Parallel

and

Distributed

Computation

Prentice

Hall

Cannon

R.

H.

-LRB-

-RRB-

Dynamics

of

Physical

Systems

McGraw-Hill

Cohn

D.

A.

Ghahramani

Z.

and

Jordan

M.

I.

-LRB-

-RRB-

Active

learning

with

statistical

models

In

Tesauro

G.

Touretzky

D.

and

Leen

T.

editors

Advances

in

Neural

Information

Processing

Systems

MIT

Press

Connell

M.

E.

and

Utgoff

P.

E.

-LRB-

-RRB-

Learning

to

control

a

dynamic

physical

system

In

Sixth

National

Conference

on

Artificial

Intelligence

pages

-LCB-

Seattle

WA

Morgan

Kaufmann

San

Mateo

CA

Conte

S.

D.

and

De

Boor

C.

-LRB-

-RRB-

Elementary

Numerical

Analysis

McGraw

Hill

Deng

K.

and

Moore

A.

W.

-LRB-

-RRB-

Multiresolution

Instance-based

Learning

In

Proceedings

of

the

International

Joint

Conference

on

Artificial

Intelligence

Montreal

Morgan

Kaufmann

Farmer

J.

D.

and

Sidorowich

J.

J.

-LRB-

-RRB-

Predicting

chaotic

time

series

Physical

Review

Letters

-LRB-

-RRB-

-LCB-

Farmer

J.

D.

and

Sidorowich

J.

J.

-LRB-

-RRB-

Exploiting

chaos

to

predict

the

future

and

reduce

noise

In

Lee

Y.

C.

editor

Evolution

Learning

and

Cognition

pages

-LCB-

World

Scientific

Press

NJ

also

available

as

Technical

Report

Los

Alamos

National

Laboratory

Los

Alamos

New

Mexico

Farmer

J.

D.

and

Sidorowich

J.

J.

-LRB-

-RRB-

Predicting

chaotic

dynamics

In

Kelso

J.

A.

S.

Mandell

A.

J.

and

Schlesinger

M.

F.

editors

Dynamic

Patterns

in

Complex

Systems

pages

-LCB-

World

Scientific

NJ

Friedman

J.

H.

and

Stuetzle

W.

-LRB-

-RRB-

Projection

Pursuit

Regression

Journal

of

the

American

Statistical

Association

-LRB-

-RRB-

-LCB-

Gorinevsky

D.

and

Connolly

T.

H.

-LRB-

-RRB-

Comparison

of

some

neural

network

and

scattered

data

approximations

The

inverse

manipulator

kinematics

example

Neural

Computation

-LCB-

Grosse

E.

-LRB-

-RRB-

LOESS

Multivariate

Smoothing

by

Moving

Least

Squares

In

C.

K.

Chul

L.

L.

S.

and

Ward

J.

D.

editors

Approximation

Theory

VI

Academic

Press

Hastie

T.

and

Loader

C.

-LRB-

-RRB-

Local

regression

Automatic

kernel

carpentry

Statistical

Science

-LRB-

-RRB-

-LCB-

Huang

P.

S.

-LRB-

-RRB-

Planning

For

Dynamic

Motions

Using

A

Search

Tree

MS

thesis

University

of

Toronto

Graduate

Department

of

Computer

Science

Jordan

M.

I.

and

Jacobs

R.

A.

-LRB-

-RRB-

Learning

to

control

an

unstable

system

with

forward

modeling

In

Touretzky

D.

editor

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Jordan

M.

I.

and

Rumelhart

D.

E.

-LRB-

-RRB-

Forward

Models

Supervised

Learning

with

a

Distal

Teacher

Cognitive

Science

-LCB-

Kaelbling

L.

P.

-LRB-

-RRB-

Learning

in

Embedded

Systems

MIT

Press

Cambridge

MA

Kuperstein

M.

-LRB-

-RRB-

Neural

Model

of

Adaptive

Hand-Eye

Coordination

for

Single

Postures

Science

-LCB-

MacKay

D.

J.

C.

-LRB-

-RRB-

Bayesian

Model

Comparison

and

Backprop

Nets

In

Moody

J.

E.

Hanson

S.

J.

and

Lippman

R.

P.

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Mahadevan

S.

-LRB-

-RRB-

Enhancing

Transfer

in

Reinforcement

Learning

by

Building

Stochastic

Models

of

Robot

Actions

In

Machine

Learning

Proceedings

of

the

Ninth

International

Workshop

Morgan

Kaufmann

Maron

O.

and

Moore

A.

-LRB-

-RRB-

Hoeffding

Races

Accelerating

Model

Selection

Search

for

Classification

and

Function

Approximation

In

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

McCallum

R.

A.

-LRB-

-RRB-

Instance-based

utile

distinctions

for

reinforcement

learning

with

hidden

state

In

Prieditis

and

Russell

-LRB-

-RRB-

pages

-LCB-

Mel

B.

W.

-LRB-

-RRB-

MURPHY

A

Connectionist

Approach

to

Vision-Based

Robot

Motion

Planning

Technical

Report

University

of

Illinois

at

Urbana-Champaign

Miller

W.

T.

-LRB-

-RRB-

Real-Time

Application

of

Neural

Networks

for

Sensor-Based

Control

of

Robots

with

Vision

IEEE

Transactions

on

Systems

Man

and

Cybernetics

-LRB-

-RRB-

-LCB-

Moore

A.

W.

-LRB-

-RRB-

Acquisition

of

Dynamic

Control

Knowledge

for

a

Robotic

Manipulator

In

Proceedings

of

the

International

Conference

on

Machine

Learning

Morgan

Kaufmann

Moore

A.

W.

-LRB-

-RRB-

Knowledge

of

Knowledge

and

Intelligent

Experimentation

for

Learning

Control

In

Proceedings

of

the

Seattle

International

Joint

Conference

on

Neural

Networks

Moore

A.

W.

-LRB-

-RRB-

Variable

Resolution

Dynamic

Programming

Efficiently

Learning

Action

Maps

in

Multivariate

Real-valued

State-spaces

In

Birnbaum

L.

and

Collins

G.

editors

Machine

Learning

Proceedings

of

the

Eighth

International

Workshop

Morgan

Kaufmann

Moore

A.

W.

-LRB-

-RRB-

Fast

Robust

Adaptive

Control

by

Learning

only

Forward

Models

In

Moody

J.

E.

Hanson

S.

J.

and

Lippman

R.

P.

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Moore

A.

W.

and

Atkeson

C.

G.

-LRB-

-RRB-

Prioritized

Sweeping

Reinforcement

Learning

with

Less

Data

and

Less

Real

Time

Machine

Learning

-LCB-

Moore

A.

W.

Hill

D.

J.

and

Johnson

M.

P.

-LRB-

-RRB-

An

Empirical

Investigation

of

Brute

Force

to

Choose

Features

Smoothers

and

Function

Approximators

In

Hanson

S.

Judd

S.

and

Petsche

T.

editors

Computational

Learning

Theory

and

Natural

Learning

Systems

Volume

MIT

Press

Moore

A.

W.

and

Lee

M.

S.

-LRB-

-RRB-

Efficient

Algorithms

for

Minimizing

Cross

Validation

Error

In

Cohen

W.

W.

and

Hirsh

H.

editors

Proceedings

of

the

International

Conference

on

Machine

Learning

Morgan

Kaufmann

Moore

A.

W.

and

Schneider

J.

-LRB-

-RRB-

Memory-Based

Stochastic

Optimization

In

Proceedings

of

Neural

Information

Processing

Systems

Conference

Omohundro

S.

M.

-LRB-

-RRB-

Efficient

Algorithms

with

Neural

Network

Behaviour

Journal

of

Complex

Systems

-LRB-

-RRB-

-LCB-

Omohundro

S.

M.

-LRB-

-RRB-

Bumptrees

for

Efficient

Function

Constraint

and

Classification

Learning

In

Lippmann

R.

P.

Moody

J.

E.

and

Touretzky

D.

S.

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Ortega

J.

M.

and

Rheinboldt

W.

C.

-LRB-

-RRB-

Iterative

Solution

of

Nonlinear

Equations

in

Several

Variables

Academic

Press

Peng

J.

-LRB-

-RRB-

Efficient

memory-based

dynamic

programming

In

Prieditis

and

Russell

-LRB-

-RRB-

pages

-LCB-

Peng

J.

and

Williams

R.

J.

-LRB-

-RRB-

Efficient

Learning

and

Planning

Within

the

Dyna

Framework

In

Proceedings

of

the

Second

International

Conference

on

Simulation

of

Adaptive

Behavior

MIT

Press

Pomerleau

D.

-LRB-

-RRB-

Reliability

estimation

for

neural

network

based

autonomous

driving

Robotics

and

Autonomous

Systems

Preparata

F.

P.

and

Shamos

M.

-LRB-

-RRB-

Computational

Geometry

Springer-Verlag

Press

W.

H.

Teukolsky

S.

A.

Vetterling

W.

T.

and

Flannery

B.

P.

-LRB-

-RRB-

Numerical

Recipes

in

C.

Cambridge

University

Press

New

York

NY

Prieditis

A.

and

Russell

S.

editors

-LRB-

-RRB-

Twelfth

International

Conference

on

Machine

Learning

Tahoe

City

CA

Morgan

Kaufmann

San

Mateo

CA

Quinlan

J.

R.

-LRB-

-RRB-

Combining

Instance-Based

and

Model-Based

Learning

In

Machine

Learning

Proceedings

of

the

Tenth

International

Conference

Saitta

L.

editor

-LRB-

-RRB-

Thirteenth

International

Conference

on

Machine

Learning

Morgan

Kaufmann

San

Mateo

CA

Schaal

S.

and

Atkeson

C.

-LRB-

-RRB-

Robot

Juggling

An

Implementation

of

Memory-based

Learning

Control

Systems

Magazine

-LRB-

-RRB-

-LCB-

Schaal

S.

and

Atkeson

C.

G.

-LRB-

-RRB-

Assessing

the

Quality

of

Local

Linear

Models

In

Cowan

J.

D.

Tesauro

G.

and

Alspector

J.

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

Stanfill

C.

and

Waltz

D.

-LRB-

-RRB-

Towards

Memory-Based

Reasoning

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

Stengel

R.

F.

-LRB-

-RRB-

Stochastic

Optimal

Control

John

Wiley

and

Sons

Sutton

R.

S.

-LRB-

-RRB-

Learning

to

Predict

by

the

Methods

of

Temporal

Differences

Machine

Learning

-LCB-

Sutton

R.

S.

-LRB-

-RRB-

Integrated

Architecture

for

Learning

Planning

and

Reacting

Based

on

Approximating

Dynamic

Programming

In

Proceedings

of

the

International

Conference

on

Machine

Learning

pages

-LCB-

Morgan

Kaufmann

Tadepalli

P.

and

Ok

D.

-LRB-

-RRB-

Scaling

up

average

reward

reinforcement

learning

by

approximating

the

domain

models

and

the

value

function

In

Saitta

-LRB-

-RRB-

Thrun

S.

-LRB-

-RRB-

Is

learning

the

n-th

thing

any

easier

than

learning

the

first

In

Advances

in

Neural

Information

Processing

Systems

-LRB-

NIPS

-RRB-

l.

Thrun

S.

and

O'Sullivan

J.

-LRB-

-RRB-

Discovering

structure

in

multiple

learning

tasks

The

TC

algorithm

In

Saitta

-LRB-

-RRB-

l.

van

der

Smagt

P.

Groen

F.

and

van

het

Groenewoud

F.

-LRB-

-RRB-

The

locally

linear

nested

network

for

robot

manipulation

In

Proceedings

of

the

IEEE

International

Conference

on

Neural

Networks

pages

-LCB-

ftp

ftp.fwi.uva.nl

pub/computer-systems/aut-sys

ps.gz

Watkins

C.

J.

C.

H.

-LRB-

-RRB-

Learning

from

Delayed

Rewards

PhD

Thesis

King

's

College

University

of

Cambridge

Zografski

Z.

-LRB-

-RRB-

Neuromorphic

Algorithmic

and

Logical

Models

for

the

Automatic

Synthesis

of

Robot

Action

PhD

dissertation

University

of

Ljubljana

Ljubljana

Slovenia

Yugoslavia

Zografski

Z.

-LRB-

-RRB-

New

methods

of

machine

learning

for

the

construction

of

integrated

neuromorphic

and

associative-memory

knowledge

bases

In

Zajc

B.

and

Solina

F.

editors

Proceedings

Mediterranean

Electrotechnical

Conference

volume

II

pages

-LCB-

Ljubljana

Slovenia

Yugoslavia

IEEE

catalog

number

Zografski

Z.

-LRB-

-RRB-

Geometric

and

neuromorphic

learning

for

nonlinear

modeling

control

and

forecasting

In

Proceedings

of

the

IEEE

International

Symposium

on

Intelligent

Control

pages

-LCB-

Glasgow

Scotland

IEEE

catalog

number

Zografski

Z.

and

Durrani

T.

-LRB-

-RRB-

Comparing

predictions

from

neural

networks

and

memory-based

learning

In

Proceedings

ICANN

International

Conference

on

Artificial

Neural

Networks

pages

-LCB-

Paris

France

dynamic programming
forward models
inverse models
lazy learning
linear quadratic regulation
locally weighted regression
loess
lqr
lwr
shifting setpoint algorithm

least commitment learning
memory-based learning