Deictic

Codes

for

the

Embodiment

of

Cognition

Dana

H.

Ballard

Mary

M.

Hayhoe

and

Polly

K.

Pook

Computer

Science

Department

University

of

Rochester

Rochester

NY

USA

-LRB-

dana@cs.rochester.edu,

mary@cvs.rochester.edu,

pook@cs.rochester.edu

-RRB-

Short

Abstract

To

describe

phenomena

that

occur

at

different

time

scales

computational

models

of

the

brain

necessarily

must

incorporate

different

levels

of

abstraction

We

argue

that

at

time

scales

of

approximately

one-third

of

a

second

orienting

movements

of

the

body

play

a

crucial

role

in

cognition

and

form

a

useful

computational

level

termed

the

embodiment

level

At

this

level

the

constraints

of

the

body

determine

the

nature

of

cognitive

operations

since

the

natural

sequentiality

of

body

movements

can

be

matched

to

the

natural

computational

economies

of

sequential

decision

systems

The

way

this

is

done

is

through

a

system

of

implicit

reference

termed

deictic

whereby

pointing

movements

are

used

to

bind

objects

in

the

world

to

cognitive

programs

We

show

how

deictic

bindings

enable

the

solution

of

natural

tasks

and

argue

that

one

of

the

central

features

of

cognition

working

memory

can

be

related

to

moment-by-moment

dispositions

of

body

features

such

as

eye

movements

and

hand

movements

Long

Abstract

To

describe

phenomena

that

occur

at

different

time

scales

computational

models

of

the

brain

necessarily

must

incorporate

different

levels

of

abstraction

We

argue

that

at

time

scales

of

approximately

one-third

of

a

second

orienting

movements

of

the

body

play

a

crucial

role

in

cognition

and

form

a

useful

computational

level

This

level

is

more

abstract

than

that

used

to

capture

neural

phenomema

yet

is

framed

at

a

level

of

abstraction

below

that

traditionally

used

to

study

high-level

cognitive

processes

such

as

reasoning

We

term

this

level

the

embodiment

level

At

the

embodiment

level

the

constraints

of

the

physical

system

determine

the

nature

of

cognitive

operations

The

key

synergy

is

that

at

time

scales

of

about

one-third

second

the

natural

sequentiality

of

body

movements

can

be

matched

to

the

natural

computational

economies

of

sequential

decision

systems

The

way

this

is

done

is

through

a

system

of

implicit

reference

termed

deictic

whereby

pointing

movements

are

used

to

bind

objects

in

the

world

to

cognitive

programs

The

focus

of

this

paper

is

to

study

how

deictic

bindings

enable

the

solution

of

natural

tasks

We

show

how

deictic

computation

provides

a

mechanism

for

representing

the

essential

features

that

link

external

sensory

data

with

internal

cognitive

programs

and

motor

actions

In

particular

we

argue

that

one

of

the

central

features

of

cognition

working

memory

can

be

related

to

moment-by-moment

dispositions

of

body

features

such

as

eye

movements

and

hand

movements

Embodiment

This

paper

is

an

attempt

to

describe

the

cognitive

functioning

of

the

brain

in

terms

of

its

interactions

with

the

rest

of

the

body

Our

central

thesis

is

that

intelligence

has

to

relate

to

interactions

with

the

physical

world

and

that

means

that

the

particular

form

of

the

human

body

is

a

vital

constraint

in

delimiting

many

aspects

of

intelligent

behavior

On

first

consideration

the

assertion

that

the

aspects

of

body

movements

play

a

vital

role

in

cognition

might

seem

unusual

The

tenets

of

logic

and

reason

demand

that

these

formalisms

can

exist

independently

of

body

aspects

and

that

intelligence

can

be

described

in

purely

computational

terms

without

recourse

to

any

particular

embodiment

From

this

perspective

the

special

features

of

the

human

body

and

its

particular

ways

of

interacting

in

the

world

are

seen

as

secondary

to

the

fundamental

problems

of

intelligence

However

the

world

of

formal

logic

is

often

freed

from

the

constraints

of

process

When

the

production

of

intelligent

behavior

by

the

body-brain

system

is

taken

into

account

the

constraints

of

time

and

space

intervene

to

greatly

limit

what

is

possible

We

will

argue

that

at

time

scales

of

approximately

one-third

of

a

second

the

momentary

disposition

of

the

body

plays

an

essential

role

in

the

brain

's

symbolic

computations

Body

actions

this

time

scale

provide

an

essential

link

between

processes

underlying

elemental

perceptual

events

and

those

involved

in

symbol

manipulation

and

the

organization

of

complex

behaviors

To

understand

this

role

for

body

movements

one

first

must

understand

the

time

scale

of

human

movements

and

how

it

relates

to

computation

in

the

brain

Since

the

brain

is

a

physical

system

communicating

over

long

distances

is

costly

in

time

and

space

and

thus

local

computation

is

the

most

efficient

Local

computation

can

be

utilized

effectively

by

organizing

systems

hierarchically

Hierarchical

structure

allows

one

to

tailor

local

effects

to

the

most

appropriate

temporal

and

spatial

scales

In

addition

a

hierarchical

organization

may

be

necessary

for

a

complex

system

to

achieve

stability

-LSB-

Simon

-RSB-

Newell

has

pointed

out

-LSB-

Newell

-RSB-

that

whenever

a

system

is

constructed

of

units

that

are

composed

of

simpler

primitives

the

more

abstract

primitives

are

necessarily

larger

and

slower

This

is

because

within

each

level

in

a

hierarchical

system

there

will

be

sequential

units

of

computation

that

must

be

composed

to

form

a

primitive

result

at

the

next

level

In

fact

with

increasing

levels

of

abstraction

the

more

abstract

components

run

slower

at

geometric

rates

This

constraint

provides

a

context

for

understanding

the

functioning

of

the

brain

and

the

organization

of

behavior

by

allowing

us

to

separate

processes

that

occur

at

different

time

scales

and

different

levels

of

abstraction

Consider

first

the

communication

system

between

neurons

Almost

all

neurons

communicate

by

sending

electrical

spikes

that

take

about

millisecond

to

generate

This

means

that

the

circuitry

that

uses

these

spikes

for

computation

has

to

run

slower

than

this

rate

Let

us

use

Newell

's

can

have

systems

for

which

local

effects

can

be

of

great

consequence

at

long

scales

Such

systems

are

termed

chaotic

-LSB-

Baker

and

Gollub

-RSB-

but

these

systems

can

not

be

easily

utilized

in

goal-directed

computation

The

bottom

line

is

that

for

any

physical

system

to

be

manageable

it

must

be

organized

hierarchically

Abstraction

Level

Temporal

Scale

Primitive

Example

Cognitive

sec

Unit

Task

Dialing

a

phone

number

Embodiment

sec

Physical

Act

Eye

movement

Attentive

ms

Deliberate

Act

Noticing

a

stimulus

Neural

ms

Neural

Circuit

Neural

ms

Neuron

Spike

Table

The

organization

of

human

computation

into

temporal

bands

-LRB-

after

-LSB-

Newell

-RSB-

but

with

some

time

scales

adjusted

to

account

for

experimental

observations

-RRB-

assumption

that

about

ten

operations

are

composed

at

each

level

Then

local

cortical

circuitry

will

require

ms.

These

operations

are

in

turn

composed

for

the

fastest

deliberate

act

in

Newell

's

terminology

A

primitive

deliberate

act

is

then

ms.

A

deliberate

act

would

correspond

to

any

kind

of

perceptual

decision

e.g.

recognizing

a

pattern

a

visual

search

operation

or

an

attentional

shift

The

next

level

is

the

physical

act

Examples

of

primitive

physical

acts

would

include

an

eye

movement

a

hand

movement

or

a

spoken

word

Composing

these

results

is

a

primitive

task

which

defines

a

new

level

Examples

of

this

level

would

be

uttering

a

sentence

or

any

actions

requiring

a

sequence

of

movements

such

as

making

a

cup

of

coffee

or

dialing

a

telephone

number

Another

example

would

be

a

chess

move

Speed

chess

is

played

at

about

seconds

per

move

Experimental

observations

show

that

Newell

's

ten-operations

rule

is

very

close

to

experimental

observations

Simple

perceptual

acts

such

as

an

attentional

shift

or

pattern

classification

take

several

tens

of

msec

so

the

ms

estimate

is

in

the

ballpark

Speeds

of

body

movements

show

that

eye

movements

take

about

sec

to

generate

which

is

about

times

the

duration

of

a

perceptual

decision

One

can

see

that

the

composition

of

tasks

by

primitive

acts

requires

the

persistence

of

the

information

in

time

Thus

the

demands

of

task

composition

require

something

like

a

working

memory

Human

working

memory

has

a

natural

decay

constant

of

a

few

sec

so

this

is

also

consistent

with

a

hierarchical

structure

Table

shows

these

relations

Our

focus

is

the

one-third

second

time

scale

We

argue

that

this

time

scale

defines

a

special

level

of

abstraction

which

we

term

the

embodiment

level

At

this

level

the

appropriate

model

of

computation

is

very

different

from

one

that

might

be

used

at

shorter

or

longer

time

scales

Computation

at

this

level

governs

the

rapid

deployment

of

the

body

's

sensors

and

effectors

in

order

to

bind

variables

in

behavioral

programs

As

such

this

computation

provides

a

language

which

reason

that

the

modeling

methods

here

are

different

than

the

traditional

symbol

manipulation

used

in

AI

is

that

the

time

scales

are

much

shorter

too

short

to

form

a

symbol

represents

the

essential

features

that

link

external

sensory

data

with

internal

cognitive

programs

and

motor

actions

In

addition

this

language

provides

an

interface

between

lower-level

neural

deliberate

acts

and

higher-level

symbolic

programs

The

ramifications

of

this

view

are

several

ffl

Cognitive

and

perceptual

processes

can

not

be

easily

separated

and

in

fact

are

interlocked

for

reasons

of

computational

economy

The

products

of

perception

are

integrated

into

distinct

serial

sensory-motor

primitives

each

taking

a

fraction

of

a

second

This

viewpoint

is

very

compatible

with

Arbib

's

perception-action

cycle

-LSB-

Arbib

Arbib

et

al.

Fuster

-RSB-

but

with

the

emphasis

on

-LRB-

a

-RRB-

the

sec

time

scale

and

-LRB-

b

-RRB-

sensory

motor

primitives

For

problems

that

take

on

the

order

of

many

seconds

to

minutes

to

solve

many

of

these

sensorymotor

primitives

must

be

synthesized

into

the

solution

ffl

The

key

constraint

is

the

number

of

degrees

of

freedom

needed

to

define

the

ongoing

cognitive

programs

We

argue

that

this

is

a

useful

interpretation

of

the

role

of

working

memory

ffl

The

function

of

the

sensory-motor

primitives

is

to

load

or

bind

the

items

in

working

memory

This

can

be

done

by

accessing

the

external

environment

or

an

internal

program

or

item

from

long-term

memory

Deictic

Sensory-Motor

Primitives

A

ubiquitous

example

of

a

rapid

sensory-motor

primitive

is

the

saccadic

eye

movement

system

Saccadic

eye

movements

are

typically

made

at

the

rate

of

about

three

per

second

and

we

make

on

the

order

of

saccades

per

day

Eye

fixations

are

at

the

boundary

of

perception

and

cognition

they

are

the

overt

indicator

that

information

is

being

represented

in

cognitive

programs

Attempts

to

understand

the

cognitive

role

of

eye

movements

have

focused

either

on

the

eye

movement

patterns

as

did

Noton

and

Stark

in

their

study

of

scanpaths

-LSB-

Noton

and

Stark

-RSB-

and

Simon

and

Chase

in

their

study

of

eye

movement

patterns

in

chess

-LSB-

Chase

and

Simon

-RSB-

or

on

the

duration

of

fixation

patterns

themselves

-LRB-

e.g.

-LSB-

Just

and

Carpenter

-RSB-

-RRB-

But

as

Viviani

points

out

-LSB-

Viviani

-RSB-

the

crux

of

the

matter

is

that

one

has

to

have

a

way

of

accessing

cognitive

state

in

addition

to

the

underlying

overt

structure

of

the

eye

scanning

patterns

For

that

reason

studies

of

reading

have

been

the

most

successful

-LSB-

Pollatsek

and

Rayner

-RSB-

but

these

results

do

not

carry

over

to

general

visual

behaviors

Viviani

's

point

is

crucial

one

needs

to

be

able

to

relate

the

actions

of

the

physical

system

to

the

internal

cognitive

state

One

way

to

start

to

do

this

is

to

posit

a

general

role

for

such

movements

irrespective

of

the

particular

behavioral

program

The

role

we

posit

here

is

variable

binding

and

it

is

best

illustrated

with

the

eye

movement

system

Since

humans

can

fixate

an

environmental

point

their

visual

system

can

directly

sample

portions

of

three-dimensional

space

as

shown

in

Figure

and

as

a

consequence

the

brain

's

internal

representations

are

implicitly

referred

to

an

external

point

Thus

neurons

tuned

to

zero-disparity

at

the

fovea

refer

to

the

instantaneous

exocentric

three-dimensional

fixation

point

The

ability

Figure

Biological

and

psychophysical

data

argue

for

deictic

frames

These

frames

are

selected

by

the

observer

to

suit

information-gathering

goals

to

use

an

external

frame

of

reference

centered

at

the

fixation

point

that

can

be

rapidly

moved

to

different

locations

leads

to

great

simplifications

in

algorithmic

complexity

-LSB-

Ballard

-RSB-

For

example

an

object

is

usually

grasped

by

first

looking

at

it

and

then

directing

the

hand

to

the

center

of

the

fixation

coordinate

frame

For

the

terminal

phase

of

the

movement

the

hand

can

be

servoed

in

depth

relative

to

the

horopter

by

using

binocular

cues

Placing

a

grasped

object

can

be

done

in

a

similar

manner

The

location

can

be

selected

using

an

eye

fixation

and

then

that

fixation

can

be

used

to

guide

the

hand

movement

Informally

we

refer

to

these

behaviors

as

do-it-where-I

'm

looking

strategies

but

more

technically

they

are

referred

to

as

deictic

strategies

after

-LSB-

Agre

and

Chapman

-RSB-

building

on

work

by

-LSB-

Ullman

-RSB-

The

word

deictic

means

pointing

or

showing

Deictic

primitives

dynamically

refer

to

points

in

the

world

with

respect

to

their

crucial

describing

features

-LRB-

e.g.

color

or

shape

-RRB-

Aspects

of

the

scene

can

be

selectively

referred

to

by

denoting

that

part

of

the

scene

with

a

special

referent

that

we

will

term

a

marker

The

dynamic

nature

of

the

referent

also

captures

the

agent

's

momentary

intentions

In

contrast

a

non-deictic

system

might

construct

a

representation

of

all

the

positions

and

properties

of

a

set

of

objects

in

viewer-centered

coordinates

and

there

would

be

no

notion

of

current

goals

Vision

is

not

the

only

sense

that

can

be

modeled

as

a

deictic

pointing

device

Haptic

manipulation

which

can

be

used

for

grasping

or

pointing

and

audition

which

can

be

used

for

localization

is

a

very

different

assertion

than

that

of

-LSB-

Marr

-RSB-

who

emphasized

that

vision

calculations

were

initially

in

viewer-centered

coordinates

and

did

not

address

the

functional

role

of

eye

movements

-LSB-

Agre

and

Chapman

-RSB-

can

also

be

modeled

as

localization

devices

We

can

think

of

fixation

and

grasping

as

mechanical

pointing

devices

and

localization

by

attention

as

a

neural

device

Thus

we

can

think

of

vision

as

having

either

mechanical

or

neural

deictic

devices

fixation

and

attention

In

this

paper

we

focus

on

vision

but

the

arguments

are

not

restricted

to

this

The

Computational

Role

of

Deictic

Reference

Although

the

human

brain

is

radically

different

in

many

ways

from

conventional

silicon

computers

they

both

have

to

address

many

of

the

same

problems

Thus

it

is

sometimes

useful

to

look

at

how

problems

are

handled

by

silicon

computers

One

major

problem

is

that

of

variable

binding

For

symbolic

computation

it

is

often

necessary

to

have

a

symbol

denote

a

very

large

number

of

bits

and

then

modify

this

reference

during

the

course

of

a

computation

Let

us

examine

how

this

is

done

using

an

artificial

example

Table

shows

a

hypothetical

portion

of

memory

for

a

computer

video

where

a

penguin

has

to

battle

bees

The

most

important

bee

is

the

closest

so

that

bee

is

marked

with

a

special

symbol

the-bee-chasing-me

The

properties

of

the

lead

bee

are

associated

with

a

pointer

That

is

conjoined

with

the

symbol

name

is

an

address

in

the

next

word

of

memory

that

locates

the

properties

of

the

lead

bee

In

the

table

this

is

the

contents

of

location

which

is

itself

an

address

pointing

to

the

location

of

beeA

's

properties

the

three

contiguous

entries

starting

at

location

Now

suppose

that

the

lead

bee

is

no

longer

beeA

but

now

beeB

The

use

of

pointers

vastly

simplifies

the

bookkeeping

in

this

case

To

change

the

referent

's

properties

only

the

contents

of

one

address

have

to

be

changed

in

this

case

location

its

contents

are

changed

to

instead

of

Thus

changing

one

memory

location

accomplishes

the

change

of

reference

Consider

the

alternative

which

is

to

have

all

of

beeA

's

properties

associated

in

contiguous

addresses

In

that

case

to

switch

to

beeB

all

of

the

latter

's

properties

have

to

be

copied

into

the

locations

currently

occupied

by

beeA

Using

pointers

avoids

the

copying

problem

It

should

be

apparent

now

how

deictic

reference

as

exemplified

by

eye

fixations

can

act

as

a

pointer

system

Here

the

external

world

is

analogous

to

computer

memory

When

fixating

a

location

the

neurons

that

are

linked

to

the

fovea

refer

to

information

computed

from

that

location

Changing

gaze

is

analogous

to

changing

the

memory

reference

in

a

silicon

computer

Physical

pointing

with

fixation

is

a

technique

that

works

as

long

as

the

embodying

physical

system

the

gaze

control

system

is

maintaining

fixation

In

a

similar

way

we

might

think

of

the

attentional

system

as

a

neural

way

of

pointing

The

center

of

gaze

does

not

have

to

be

moved

but

the

idea

is

the

same

to

create

a

momentary

reference

to

a

point

in

space

so

that

the

properties

of

the

referent

can

be

used

as

a

unit

in

computation

Both

the

gaze

control

system

and

neural

attentional

mechanisms

each

dedicate

themselves

to

processing

a

single

token

If

behaviors

require

additional

variables

these

must

be

kept

in

a

technical

details

see

-LSB-

Agre

and

Chapman

-RSB-

Address

Contents

the-bee-chasing-me

beeA

's

weight

beeA

's

speed

beeA

's

of

stripes

beeB

's

weight

beeB

's

speed

beeB

's

of

stripes

Address

Contents

the-bee-chasing-me

beeA

's

weight

beeA

's

speed

beeA

's

of

stripes

beeB

's

weight

beeB

's

speed

beeB

's

of

stripes

Table

A

portion

of

computer

memory

illustrating

the

use

of

pointers

Left

Reference

is

to

beeA

Right

Reference

is

to

beeB

The

change

in

reference

can

be

accomplished

by

changing

a

single

memory

cell

separate

system

termed

working

memory

-LSB-

Baddeley

Broadbent

-RSB-

Although

the

brain

works

on

very

different

principles

than

a

computer

the

problem

faced

is

the

same

In

working

memory

the

references

to

the

items

therein

have

to

be

changed

with

the

requirements

of

the

ongoing

computation

The

strategy

of

copying

that

was

used

as

a

straw

man

in

the

silicon

example

is

even

more

implausible

here

as

most

neurons

in

the

cortex

exhibit

a

form

of

place

coding

-LSB-

Barlow

Ballard

-RSB-

that

can

not

be

easily

changed

Thus

it

seems

that

at

the

one-third

second

time

scale

ways

of

temporarily

binding

huge

numbers

of

neurons

and

changing

those

bindings

must

exist

That

is

the

brain

must

have

some

kind

of

pointer

mechanism

Outline

The

focus

of

this

paper

is

to

explain

why

deictic

codes

are

a

good

model

for

behavior

at

the

embodiment

level

The

presentation

is

organized

into

three

main

sections

ffl

Section

develops

the

computational

role

of

deictic

codes

which

is

to

represent

the

essential

degrees

of

freedom

used

to

characterize

behavioral

programs

The

essential

degrees

of

mechanisms

for

such

operations

have

been

proposed

-LSB-

Koch

and

Crick

Buhmann

et

al.

Shastri

-RSB-

but

as

of

yet

there

is

not

sufficient

data

to

resolve

the

issue

freedom

can

have

perceptual

cognitive

and

motor

components

The

perceptual

component

deictically

indicates

parts

of

the

world

that

define

the

momentary

external

context

for

the

current

behavioral

program

The

cognitive

component

is

the

momentary

definition

of

state

by

deictic

variables

in

working

memory

The

motor

component

deictically

indicates

the

essential

internal

motor

context

necessary

for

the

current

behavioral

program

We

argue

that

the

complexity

of

behavioral

programs

scales

exponentially

with

the

number

of

markers

The

main

consequence

is

that

the

only

computations

that

are

practical

use

a

small

number

of

variables

ffl

Section

discusses

the

psychological

evidence

in

favor

of

deictic

strategies

Studying

a

simple

sensory-motor

task

provides

evidence

that

working

memory

is

intimately

involved

in

describing

the

task

and

that

working

memory

is

reset

from

moment

to

moment

with

deictic

actions

ffl

Section

discusses

the

implications

of

deictic

computation

in

understanding

cortical

circuitry

A

consequence

of

complex

programs

being

composed

of

simpler

primitives

each

of

which

involves

sensory-motor

operations

is

that

many

disparate

areas

of

the

brain

must

interact

in

distinct

ways

to

achieve

special

functions

Some

of

these

operations

bind

parts

of

the

sensorium

and

others

use

these

bindings

to

select

the

next

action

Deictic

Representation

Deictic

representation

is

a

system

of

implicit

reference

whereby

the

body

's

pointing

movements

are

used

to

bind

objects

in

the

world

to

cognitive

programs

This

section

develops

the

computational

role

of

deictic

pointing

which

is

to

represent

the

essential

degrees

of

freedom

used

to

characterize

behavioral

programs

Our

intent

is

to

show

that

distilling

the

degrees

of

freedom

down

to

the

minimum

allows

simple

decision

making

The

essential

degrees

of

freedom

can

have

perceptual

cognitive

and

motor

components

The

perceptual

component

uses

deictic

pointing

to

define

the

context

for

the

current

behavioral

program

The

cognitive

component

maintains

this

context

as

variables

in

working

memory

The

motor

component

uses

the

working

memory

variables

to

mediate

the

action

of

effectors

We

argue

that

the

complexity

of

behavioral

programs

scales

exponentially

with

the

number

of

variables

with

the

consequence

that

the

only

computations

that

are

practical

use

a

small

number

of

variables

Deictic

Models

of

Sensory

Processing

The

primary

example

of

a

deictic

sensory

action

is

fixation

There

are

a

number

of

indications

from

human

vision

that

fixation

might

have

theoretical

significance

Fixations

result

in

a

high-resolution

apple

apple

Figure

A

schematic

of

Kowler

and

Anton

's

experiment

Subjects

reading

text

normally

fixate

words

only

once

but

when

the

letters

are

reversed

each

letter

is

fixated

-LSB-

Kowler

and

Anton

-RSB-

spatial

context

in

a

local

region

since

the

human

eye

has

much

better

resolution

in

a

small

region

near

the

optical

axis

i.e.

the

fovea

Over

a

region

of

approximately

one

to

two

degrees

of

visual

angle

the

resolution

is

better

by

an

order

of

magnitude

than

in

the

periphery

One

feature

of

this

design

is

the

representation

of

local

high

acuity

within

a

larger

field

of

view

This

makes

it

ideal

as

a

pointing

device

to

denote

the

relevant

parts

of

the

visible

environment

Given

the

high-resolution

fovea

one

might

be

tempted

to

conclude

that

the

primary

purpose

of

fixation

is

to

obtain

better

spatial

resolution

That

certainly

is

an

important

consequence

of

fixation

but

is

almost

certainly

not

its

only

role

An

indication

of

its

computational

role

is

given

in

a

study

by

-LSB-

Kowler

and

Anton

-RSB-

who

measured

fixation

patterns

while

reading

text

of

reversed

letters

-LRB-

see

Figure

-RRB-

In

normal

text

individual

words

are

contained

within

the

fovea

and

are

fixated

only

once

With

the

reversed

letters

however

individual

letters

were

fixated

resulting

in

several

fixations

per

word

Note

that

in

this

case

the

function

of

fixation

can

not

be

for

increased

resolution

as

individual

words

can

be

resolved

within

a

single

fixation

It

must

be

the

case

that

fixation

is

serving

some

technical

function

in

recognizing

the

reversed

letters

beyond

that

of

improving

spatial

resolution

In

this

case

the

letter

is

the

appropriate

pattern

recognition

unit

so

the

fixations

are

indicating

what

information

is

relevant

for

the

task

The

Kowler

and

Anton

data

suggest

that

sequences

of

deictic

primitives

can

create

complex

behaviors

where

each

sensory-motor

primitive

defines

the

context

for

its

successor

using

only

the

information

immediately

fixated

or

attended

This

idea

was

tested

in

computer

simulations

where

an

abstract

hand-eye

robot

learned

simple

block

manipulations

For

example

consider

the

problem

of

picking

up

a

green

block

that

has

another

block

stacked

on

top

of

it

shown

in

Figure

-LRB-

from

-LSB-

Whitehead

and

Ballard

-RSB-

-RRB-

This

problem

is

solvable

by

computer

simulation

using

reinforcement

learning

-LSB-

Whitehead

and

Ballard

-RSB-

The

result

is

the

following

program

the

three

current

models

of

neural

computation

reinforcement

learning

-LSB-

Barto

et

al.

-RSB-

neural

networks

-LSB-

Hertz

et

al.

-RSB-

and

genetic

algorithms

-LSB-

Koza

Goldberg

-RSB-

only

reinforcement

learning

explicitly

captures

discrete

sequential

structure

as

a

primitive

As

such

it

is

a

good

model

for

integrating

cognitive

portions

of

a

behavioral

program

with

observed

characteristics

of

the

brain

-LSB-

Woodward

et

al.

Schultz

et

al.

-RSB-

Fixate

-LRB-

Blue

-RRB-

Fixate

-LRB-

Green

-RRB-

Fixate

-LRB-

Top

-RRB-

PickUp

Fixate

-LRB-

Table

-RRB-

PutDown

Fixate

-LRB-

Green

-RRB-

PickUp

Figure

A

graphical

display

from

the

output

of

a

program

that

has

learned

the

Pick

up

the

green

block

task

The

steps

in

the

program

use

deictic

references

rather

than

geometrical

coordinates

For

each

stage

in

the

solution

the

plus

symbol

shows

the

location

of

the

fixation

point

Fixate

-LRB-

green

-RRB-

Fixate

-LRB-

Top-of-where-I

'm

looking

-RRB-

Pickup

Fixate

-LRB-

Somewhere-on-the-table

-RRB-

Putdown

Fixate

-LRB-

green

-RRB-

Pickup

To

work

this

program

needs

more

than

just

the

ability

to

fixate

The

reason

is

that

the

deictic

representation

only

uses

the

fixated

objects

to

make

decisions

Thus

when

fixating

a

blue

block

it

is

not

clear

what

to

do

If

the

block

is

at

the

top

of

the

stack

the

next

operation

should

be

PickUp

but

if

it

is

on

the

table

the

next

instruction

should

be

Fixate

-LRB-

Green

-RRB-

-LSB-

Whitehead

and

Ballard

-RSB-

have

shown

that

this

problem

can

be

resolved

by

using

an

additional

deictic

mechanism

in

the

form

of

a

visual

focus

of

attention

This

allows

the

program

to

keep

track

of

the

necessary

context

as

the

attended

object

can

be

different

in

the

two

different

cases

This

in

turn

allows

the

task

to

be

successfully

completed

In

the

program

it

is

assumed

that

the

instruction

Fixate

-LRB-

image

feature

-RRB-

will

orient

the

center

of

gaze

to

point

to

a

place

in

the

image

with

that

feature

the

details

of

how

this

could

be

done

are

described

in

Section

These

actions

are

context

sensitive

For

example

Pickup

and

Putdown

are

assumed

to

act

at

the

center

of

the

fixation

frame

Fixate

-LRB-

Top-of-where-I

'm

looking

-RRB-

will

transfer

the

gaze

to

the

top

of

the

stack

currently

fixated

Adding

Memory

to

Deictic

Models

The

central

representational

core

of

cognitive

programs

is

working

memory

-LSB-

Baddeley

-RSB-

The

very

limited

capacity

of

working

memory

is

one

of

the

most

prominent

features

of

human

cognition

and

has

been

much

studied

The

earliest

of

these

studies

stressed

delimiting

the

upper

bounds

of

this

memory

-LSB-

Miller

-RSB-

but

more

current

thinking

recognizes

the

role

of

working

memory

in

the

ongoing

cognitive

process

-LSB-

Baddeley

-RSB-

However

these

studies

have

not

yet

related

the

process

of

memory

use

to

the

detailed

interactions

between

the

body

and

the

world

on

the

all-important

time

scales

of

sec

or

less

Perhaps

as

a

consequence

few

studies

have

posited

a

detailed

computational

role

for

working

memory

in

cognition

The

role

posited

for

working

memory

is

inspired

by

robot

models

of

behavior

that

use

deictic

actions

Such

models

simplify

behavioral

programs

by

minimizing

representation

-LSB-

Brooks

Agre

and

Chapman

Ballard

-RSB-

To

make

this

work

there

must

be

mechanisms

for

temporarily

denoting

that

part

of

the

scene

with

a

special

referent

or

pointer

Agre

and

Chapman

define

this

operation

as

marking

properties

of

the

scene

Once

marked

the

properties

can

be

accessed

via

the

marker

for

as

long

as

the

marker

is

bound

to

those

properties

The

referent

of

marked

items

can

be

large

as

in

iconic

items

-LSB-

Nakayama

-RSB-

but

the

total

number

of

such

items

typically

is

small

The

notion

of

marking

parts

of

the

scene

was

introduced

when

we

described

how

fixation

bound

parts

of

the

world

in

the

green

block

task

In

that

task

only

two

such

markers

were

required

fixation

and

attention

For

more

complicated

tasks

however

more

markers

may

be

needed

Let

us

illustrate

this

with

Chapman

's

example

of

copying

a

tower

of

blocks

each

identified

with

a

color

as

shown

in

Figure

To

solve

this

problem

one

marker

keeps

track

of

the

point

in

the

tower

being

copied

another

keeps

track

of

the

point

in

the

copy

and

a

third

is

used

for

manipulating

the

new

block

that

is

part

of

the

copy

Figure

shows

how

this

can

be

done

The

markers

provide

a

dynamic

referent

for

the

blocks

that

is

action

specific

A

non-deictic

way

to

solve

this

task

would

be

to

process

the

scene

so

that

each

block

is

catalogued

and

has

a

unique

identification

code

Then

the

copying

task

could

be

solved

by

the

space

of

all

possible

relationships

amoung

coded

items

for

the

right

ones

The

problem

is

that

this

strategy

is

very

expensive

as

the

number

of

possible

relationships

among

different

configurations

of

blocks

can

be

prohibitively

large

For

all

possible

configurations

of

just

blocks

billion

relationships

are

needed

In

contrast

a

deictic

strategy

avoids

costly

descriptions

by

using

markers

that

can

be

re-assigned

to

blocks

dynamically

using

visual

routines

In

contrast

the

key

advantage

of

the

marker

strategy

is

that

it

scales

well

Only

three

markers

are

needed

regardless

of

the

tower

height

This

is

the

important

claim

of

marker-based

behavioral

Mark

-LRB-

RED

TOWER

Plus

-RRB-

Mark

-LRB-

RED

notPlus

Times

-RRB-

PutOn

-LRB-

Square

Times

-RRB-

Move

-LRB-

Plus

UP

-RRB-

Move

-LRB-

Times

UP

-RRB-

Mark

-LRB-

Color

-LRB-

Plus

-RRB-

notPlus

Square

-RRB-

PutOn

-LRB-

Square

Times

-RRB-

Markers

SquareTimesPlus

Move

-LRB-

Plus

UP

-RRB-

Mark

-LRB-

Color

-LRB-

Plus

-RRB-

notPlus

Square

-RRB-

Figure

A

tower

of

blocks

can

be

copied

using

three

markers

At

any

instant

the

access

to

the

sensorium

is

limited

to

the

marked

objects

The

number

of

markers

is

minimized

by

re-using

markers

during

the

course

of

the

behavior

programs

Complex

tasks

can

be

decomposed

into

small

behavioral

primitives

where

the

necessary

state

needed

to

keep

track

of

the

process

can

be

represented

with

just

the

temporal

history

of

a

handful

of

markers

In

other

words

our

claim

is

that

almost

all

complex

behaviors

can

be

performed

with

just

a

few

markers

Let

us

now

make

the

connection

between

the

computational

and

psychological

domains

If

a

task

can

be

solved

using

only

one

or

two

markers

then

this

can

be

handled

by

explicit

pointing

such

as

fixation

or

neural

pointing

such

as

attention

However

additional

markers

require

some

additional

representational

mechanism

A

plausible

psychological

mechanism

is

that

of

working

memory

Working

memory

items

may

be

thought

of

as

corresponding

to

computational

markers

A

marker

allows

access

to

the

contents

of

an

item

of

memory

The

marker

notation

raises

the

issue

of

binding

or

setting

the

marker

referent

This

is

because

markers

are

general

variables

that

can

be

re-used

for

other

computations

When

are

markers

bound

For

a

visual

marker

one

possible

indication

that

it

is

being

set

could

be

fixation

Looking

directly

at

a

part

of

the

scene

provides

special

access

to

the

features

immediate

to

the

fixation

point

and

these

could

be

bound

to

a

marker

during

the

fixation

period

In

all

likelihood

binding

can

take

place

faster

than

this

say

by

using

an

attentional

process

but

using

fixation

as

a

lower

bound

would

allow

us

to

bind

several

markers

per

second

with

a

capacity

determined

by

the

decay

rate

of

the

activated

items

Even

though

a

problem

can

be

solved

with

a

small

number

of

markers

why

should

there

be

pressure

to

use

the

minimal-marker

solution

One

argument

for

minimal-marker

programs

can

be

made

in

terms

of

the

cost

of

finding

alternate

solutions

which

is

often

characterized

as

the

credit

assignment

problem

As

an

example

of

this

problem

suppose

that

the

tower

problem

is

changed

so

that

the

colors

must

be

copied

in

reverse

order

and

the

problem

of

picking

up

a

green

block

is

changed

to

that

of

picking

up

a

yellow

block

Comparing

the

two

problems

what

makes

the

new

solution

of

the

second

easier

to

discover

is

that

the

possibilities

scale

as

a

function

of

the

number

of

required

markers

If

we

assume

a

sequential

search

model

such

as

that

postulated

in

reinforcement

learning

models

then

the

cost

of

searching

for

an

alternative

solution

to

a

problem

would

scale

as

-LRB-

MV

-RRB-

s

where

M

is

the

number

of

markers

V

is

the

number

of

visual/manual

routines

and

s

is

the

number

of

steps

in

the

program

Thus

the

central

problem

may

be

just

that

the

cost

of

searching

alternatives

scales

badly

with

an

increasing

number

of

markers

This

may

result

in

a

tremendous

pressure

on

finding

behavioral

programs

that

only

require

small

numbers

of

markers

A

second

reason

for

a

small

number

of

markers

may

be

that

it

reflects

a

balance

between

the

natural

decay

rate

of

the

marked

items

and

the

temporal

demands

of

the

task

In

Section

we

described

the

composition

of

cognitive

tasks

from

components

at

a

lower

level

which

we

called

physical

acts

To

compose

new

behaviors

in

this

way

there

must

be

some

way

of

keeping

the

components

active

long

enough

to

compose

the

new

behavior

At

the

same

time

it

seems

likely

that

if

the

component

items

are

active

for

too

long

they

would

no

longer

be

relevant

for

the

current

task

demands

and

may

interfere

with

the

execution

of

the

next

task

-LRB-

The

extraordinary

case

of

Luria

's

patient

S

whose

sensory

memory

was

excessively

persistent

attests

to

such

interference

-LSB-

Luria

-RSB-

-RRB-

Thus

we

can

see

that

the

capacity

of

working

memory

will

be

a

consequence

of

the

natural

decay

rate

of

marked

-LRB-

activated

-RRB-

items

and

should

reflect

the

dynamic

demands

of

the

task

Deictic

Motor

Routines

Deictic

variables

define

a

relative

coordinate

frames

for

successive

motor

behaviors

Such

variables

can

avoid

world-centered

geometry

that

varies

with

robot

movement

For

fixation

the

reference

is

gaze-centered

To

open

a

door

for

instance

looking

at

the

doorknob

defines

a

relative

servo

target

-LSB-

Crisman

and

Cleary

-RSB-

demonstrate

the

computational

advantage

of

target-centered

frames

for

mobile

robot

navigation

Similarly

body

position

creates

a

reference

for

motor

actions

Studies

of

human

hand-eye

coordination

have

found

head

movements

often

to

be

closely

associated

with

hand

movements

-LSB-

Pelz

et

al.

-RSB-

found

that

most

subjects

square

their

head

with

respect

to

the

site

of

a

complex

manipulation

action

and

then

hold

it

very

still

during

the

manipulation

even

when

the

eyes

are

looking

elsewhere

This

fits

the

deictic

hypothesis

if

head

position

is

a

reference

frame

for

relative

hand

motion

Additionally

bi-manual

animals

often

use

one

hand

as

a

vise

and

the

other

for

dexterous

manipulation

In

these

cases

the

vise

hand

may

mark

a

reference

frame

allowing

dexterous

motions

to

be

made

relative

to

that

frame

Homunculus

Cerebrum

Spinal

Reflexes

Muscles

Muscles

Operator

Programs

Effectors

a

b

c

Figure

The

teleassistance

model

allows

the

study

of

brain

function

by

emulating

the

distinction

between

abstract

codes

for

motor

programs

and

pre-programmed

routines

-LRB-

a

-RRB-

The

original

homunculus

concept

-LRB-

b

-RRB-

Bernsteins

's

hierarchical

division

of

function

-LSB-

Bernstein

-RSB-

-LRB-

c

-RRB-

The

teleassistance

model

The

advantages

of

deictic

strategies

can

be

illustrated

with

robot

hardware

using

the

concept

of

teleassistance

shown

in

Figure

-LSB-

Pook

and

Ballard

-RSB-

In

teleassistance

a

human

operator

is

the

brain

to

an

otherwise

autonomous

dexterous

robot

manipulator

The

operator

does

not

directly

control

the

robot

but

rather

communicates

symbolically

via

a

deictic

sign

language

shown

in

Table

A

sign

selects

the

next

motor

program

to

perform

and

tunes

it

with

hand-centered

markers

This

example

illustrates

a

way

of

decoupling

the

human

's

link

between

motor

program

and

reflexes

Here

the

output

of

the

human

subject

is

a

deictic

code

for

a

motor

program

that

a

robot

then

carries

out

This

allows

the

study

of

the

use

and

properties

of

the

deictic

code

The

sign

language

is

very

simple

To

assist

a

robot

to

open

a

door

requires

only

the

three

signs

shown

in

Table

Pointing

to

the

door

handle

prompts

the

robot

to

reach

toward

it

and

provides

the

axis

along

which

to

reach

A

finite

state

machine

-LRB-

FSM

-RRB-

for

the

task

specifies

the

flow

of

control

This

embeds

sign

recognition

and

motor

response

within

the

overall

task

context

Pointing

and

preshaping

the

hand

create

hand-centered

spatial

frames

Pointing

defines

a

relative

axis

for

subsequent

motion

In

the

case

of

preshaping

the

relative

frame

attaches

within

the

opposition

space

-LSB-

Arbib

et

al.

-RSB-

of

the

robot

fingers

With

adequate

dexterity

and

compliance

simply

flexing

the

fingers

toward

the

origin

of

that

frame

coupled

with

a

force

control

loop

suffices

to

morphology

determines

much

of

how

hands

are

used

the

domain

knowledge

inherent

in

the

shape

and

frame

position

can

be

exploited

For

example

a

wrap

grasp

defines

a

coordinate

system

relative

to

the

palm

Sign

Meaning

POINT

While

the

operator

points

the

robot

moves

in

the

direction

of

the

pointing

axis

independently

of

world

coordinates

Thus

the

robot

reach

is

made

relative

to

a

deictic

axis

that

the

teleoperator

can

easily

adjust

PRESHAPE

A

grasp

preshape

defines

a

new

spatial

frame

centered

on

the

palm

of

the

hand

The

operator

preshapes

her

hand

to

define

a

grasp

form

and

a

new

spatial

frame

centered

on

the

palm

HALT

Halting

is

used

to

punctuate

the

motor

program

Table

Signs

used

in

teleassistance

experiment

form

a

stable

grasp

Since

the

motor

action

is

bound

to

the

local

context

the

same

grasping

action

can

be

applied

to

different

objects

a

spatula

a

mug

a

doorknob

by

changing

the

preshape

Robot

control

by

deictic

sign

language

can

be

compared

to

two

other

conditions

as

shown

in

Figure

The

first

can

be

thought

of

a

homunculus

strategy

The

strategy

allows

the

human

operator

to

directly

control

all

robot

motion

The

operator

is

connected

to

the

robot

effectors

in

a

servo-loop

that

is

closed

in

real-time

The

second

strategy

is

to

not

use

the

human

operator

at

all

but

have

all

the

operations

done

by

autonomous

low-level

robot

routines

The

homunculus

strategy

results

in

error-free

performance

but

has

two

glaring

disadvantages

First

it

requires

monitoring

since

the

operator

is

part

of

the

low-level

feedback

loops

Second

control

is

much

slower

owing

to

the

delays

introduced

by

this

closed-loop

strategy

The

autonomous

strategy

is

the

fastest

but

suffers

by

having

little

error

tolerance

Current

real-time

robots

can

not

readily

accommodate

even

simple

changes

in

their

environments

If

the

door

handle

is

different

than

expected

the

door-opening

behavior

fails

By

way

of

contrast

the

teleassistance

strategy

which

models

a

layered

control

structure

that

uses

autonomous

routines

directed

by

the

deictic

signs

is

significantly

faster

than

the

homunculus

strategy

only

requires

of

the

total

time

for

executive

control

-LRB-

indicated

by

the

extent

of

the

dark

shaded

areas

in

Figure

-RRB-

and

can

better

accommodate

natural

variations

in

the

task

TELEOPERATION

Avg

-LRB-

sec

-RRB-

humanreachgraspopenreleasepointpreshapehalthumanreachgraspopenre

leaserobotreachgraspopenreleaserobotTELEASSISTANCE

Avg

CaseAUTONOMOUS

SERVOING

Avg

Case

Figure

Results

of

simulating

different

forms

of

control

in

the

door-opening

task

Bars

show

the

time

spent

in

each

subtask

-LRB-

a

-RRB-

The

original

homunculus

concept

models

the

direct

connection

of

the

human

operator

to

effectors

-LRB-

b

-RRB-

Autonomous

control

models

the

execution

of

the

task

exclusively

by

low-level

routines

-LRB-

c

-RRB-

The

teleassistance

model

shows

the

interaction

between

deictic

commands

that

signal

the

low-level

routines

and

the

routines

themselves

Deictic

Strategies

and

the

What/Where

Dichotomy

The

success

of

deictic

sensory

strategies

may

be

an

indication

that

the

general

problem

of

associating

many

internal

models

to

many

parts

of

an

image

simultaneously

is

too

difficult

for

the

brain

to

solve

Deictic

strategies

may

make

it

computationally

tractable

by

simplifying

the

general

task

into

a

series

of

two

kinds

of

simpler

tasks

what

and

where

-LSB-

Ungerleider

and

Mishkin

-RSB-

These

tasks

either

find

information

about

location

-LRB-

using

only

one

internal

model

-RRB-

or

identification

-LRB-

of

only

one

world

object

whose

location

is

known

-RRB-

Table

summarizes

this

view

A

location

task

is

greatly

simplified

by

only

having

to

find

the

image

coordinates

of

a

single

model

In

this

task

the

image

periphery

must

be

searched

one

can

assume

that

the

model

has

been

chosen

a

priori

An

identification

task

is

greatly

simplified

by

having

to

identify

only

the

foveated

part

of

the

image

In

this

task

one

can

assume

that

the

location

of

the

material

to

be

established

is

at

the

fixation

point

only

the

internal

model

data

base

must

be

searched

Experimental

models

that

test

what/where

primitives

on

real

image

data

confirm

that

this

dichotomy

leads

to

dramatically

faster

algorithms

for

each

of

the

specialized

tasks

-LSB-

Swain

and

Ballard

-RSB-

Thus

we

think

of

the

eye

movements

as

solving

a

succession

of

location

and

identification

subtasks

in

the

process

of

meeting

some

larger

cognitive

goal

This

suggestion

is

compatible

with

the

extensive

data

that

supports

the

hypothesis

of

two

parallel

pathways

in

visual

cortex

-LSB-

Ungerleider

and

Mishkin

-RSB-

and

addresses

the

issue

of

why

they

are

separate

-LRB-

sequentialization

simplifies

the

computational

problem

-RRB-

and

work

together

-LRB-

the

separate

operations

have

to

be

composed

to

form

behaviors

-RRB-

The

need

for

composing

is

neglected

by

Goodale

-LSB-

Goodale

and

Milner

-RSB-

who

sees

the

parallel

pathways

as

reflecting

different

behavioral

abilities

In

Goodale

's

model

the

where

pathway

is

characterized

in

terms

of

motoric

function

as

a

how

pathway

Thus

Goodale

's

bifurcation

is

asymmetric

the

what

pathway

still

performs

a

sub-function

but

the

how

pathway

is

integrated

into

a

complete

motor

system

In

contrast

our

what/where

distinctions

are

based

on

symmetries

in

the

underlying

computational

problem

and

have

the

same

ontological

status

as

sub-functions

in

a

larger

program

In

Section

we

show

that

human

performance

of

a

sensory-motor

task

appears

to

be

broken

down

into

just

such

a

sequence

of

primitive

what/where

operations

The

concept

of

markers

changes

the

conceptual

focus

of

computation

from

continuous

processing

to

discrete

processing

centered

around

the

momentary

disposition

of

markers

Deictic

actions

can

be

described

in

terms

of

markers

Some

change

the

location

of

a

marker

and

others

compute

properties

or

initiate

movements

with

respect

to

marked

locations

Thus

we

can

re-interpret

the

what/where

dichotomy

in

Table

in

terms

of

marker

operations

Manipulation

can

be

characterized

as

applying

effectors

to

a

marked

object

Location

can

be

interpreted

as

marking

an

object

Identification

can

be

interpreted

as

applying

a

lower-level

primitive

routine

to

a

marked

object

Image

Parts

Models

One

Many

One

I.

Manipulation

trying

to

do

something

with

an

object

whose

identity

and

location

are

known

II

Identification

trying

to

identify

an

object

whose

location

can

be

fixated

Many

III

Location

trying

to

find

a

known

object

that

may

not

be

in

view

Too

difficult

Table

The

organization

of

visual

computation

into

what/where

modules

may

have

a

basis

in

complexity

Trying

to

match

a

large

number

of

image

segments

to

a

large

number

of

models

at

once

may

be

too

difficult

Evidence

for

Deictic

Strategies

in

Behavior

We

began

by

positing

the

role

of

deictic

actions

as

binding

variables

in

deictic

programs

Next

we

introduced

markers

as

a

general

term

to

describe

both

variables

in

spatial

working

memory

and

current

deictic

variables

for

acquisition

of

visual

information

and

initiation

of

motor

routines

We

now

go

on

to

examine

whether

in

fact

this

conceptualization

is

appropriate

for

human

behavior

Because

the

eyes

allow

a

natural

implementation

of

deictic

strategies

the

question

immediately

raised

is

how

humans

in

fact

use

their

eye

movements

in

the

context

of

natural

behaviors

We

designed

a

series

of

experiments

to

test

the

use

of

deictic

strategies

in

the

course

of

a

simple

task

involving

movements

of

the

eyes

and

hand

and

also

visual

memory

The

task

was

to

copy

a

pattern

of

colored

blocks

This

task

was

chosen

to

reflect

the

basic

sensory

and

motor

operations

involved

in

a

wide

range

of

human

performance

An

important

feature

of

the

task

is

that

subjects

have

the

freedom

to

choose

their

own

task

parameters

The

task

involves

a

series

of

steps

that

require

coordination

of

eye

and

hand

movements

and

memory

but

the

subjects

organize

the

steps

to

compose

the

behavior

as

in

any

natural

behavior

An

advantage

of

this

simple

task

is

that

the

underlying

cognitive

operations

are

quite

clearly

defined

by

the

implicit

physical

constraints

of

the

task

as

will

become

evident

in

what

follows

This

is

important

because

definitive

statements

about

the

role

of

fixation

in

cognition

are

impossible

when

the

internal

cognitive

state

is

undefined

-LSB-

Viviani

-RSB-

Serialized

Representations

The

block

copying

task

is

shown

in

Figure

A

display

of

colored

blocks

was

divided

up

into

three

areas

the

model

source

and

workspace

The

model

area

contains

the

block

configuration

to

be

copied

the

source

contains

the

blocks

to

be

used

and

the

workspace

is

the

area

where

the

copy

is

assembled

Note

that

the

colored

blocks

are

random

and

difficult

to

group

into

larger

shapes

so

they

have

to

be

handled

individually

This

allows

the

separation

of

perceptual

and

motor

components

of

the

task

The

task

has

been

studied

using

both

real

blocks

and

simulated

blocks

on

a

computer

screen

that

are

moved

with

a

cursor

A

striking

feature

of

task

performance

is

that

subjects

behaved

in

a

very

similar

stereotypical

way

characterized

by

frequent

eye

movements

to

the

model

pattern

Observations

of

individual

eye

movements

suggest

that

information

is

acquired

incrementally

during

the

task

and

even

modest

demands

on

visual

memory

are

avoided

For

example

if

the

subject

memorized

and

copied

four

sub-patterns

of

two

blocks

which

is

well

within

visual

memory

limitations

one

would

expect

a

total

of

four

looks

into

the

model

area

Instead

subjects

sometimes

made

as

many

as

fixations

in

the

model

area

in

the

course

of

copying

the

pattern

and

did

not

appear

to

memorize

more

than

the

immediately

relevant

information

from

the

model

Indeed

they

commonly

made

more

than

one

fixation

in

the

model

area

while

copying

a

single

block

Thus

subjects

choose

to

serialize

the

task

by

adding

many

more

eye

fixations

than

might

be

expected

These

fixations

allow

subjects

to

postpone

the

gathering

of

task-relevant

information

until

just

before

it

is

required

Figure

shows

an

example

of

the

eye

and

hand

-LRB-

mouse

-RRB-

movements

involved

in

moving

a

single

block

by

one

of

the

subjects

Following

placement

of

the

second

block

the

eye

moves

up

to

the

model

area

while

at

the

same

time

the

hand

moves

toward

the

blocks

in

the

resource

During

the

fixation

in

the

model

area

the

subject

presumably

is

acquiring

the

color

of

the

next

block

Following

a

visual

search

operation

a

saccade

is

then

programmed

and

the

eye

moves

to

the

resource

at

the

location

of

block

three

-LRB-

black

-RRB-

and

is

used

to

guide

the

hand

for

a

pickup

action

The

eye

then

goes

back

to

the

model

while

the

cursor

is

moved

to

the

workspace

for

putting

down

the

block

This

second

fixation

in

the

model

area

is

presumably

for

the

purpose

of

acquiring

positional

information

for

block

placement

The

eye

then

moves

to

the

drop-off

location

to

facilitate

the

release

of

the

block

The

basic

cycle

from

the

point

just

after

a

block

is

dropped

off

to

the

point

where

the

next

block

is

dropped

off

allows

us

to

explore

the

different

sequences

of

primitive

movements

made

in

putting

the

blocks

into

place

A

way

of

coding

these

subtasks

is

to

summarize

the

eye

fixations

Thus

the

sequence

in

Figure

can

be

encoded

as

model-pickup-model-drop

with

the

understanding

that

the

pickup

occurs

in

the

source

area

and

the

drop

in

the

workspace

area

Four

principal

sequences

of

eye

movements

can

be

identified

as

shown

in

Figure

Since

the

decisive

information

is

the

color

and

relative

location

of

each

block

the

observed

sequences

can

be

understood

in

terms

of

whether

the

subject

has

remembered

either

the

color

and/or

the

location

of

the

block

currently

needed

The

necessary

assumption

is

that

the

information

is

most

conveniently

obtained

by

explicitly

fixating

Model

Workspace

Resource

eye

hand

Figure

Copying

a

single

block

within

the

task

The

eye

position

trace

is

shown

by

the

cross

and

the

dotted

line

The

cursor

trace

is

shown

by

the

arrow

and

dark

line

The

numbers

indicate

corresponding

points

in

time

for

the

eye

and

hand

traces

Strategy

Time

-LRB-

Sec

-RRB-

Memory

Items

MPMD

PMD

color

MPD

offset

PD

color

and

offset

Table

Speed

vs.

memory

tradeoffs

observed

in

the

block-copying

task

the

appropriate

locations

in

the

model

and

that

the

main

preference

is

to

acquire

color

or

location

information

just

before

it

is

required

If

both

the

color

and

location

are

needed

that

is

have

not

been

previously

remembered

a

model-pickup-model-drop

sequence

should

result

If

the

color

is

known

a

pickup-model-drop

sequence

should

result

if

the

location

is

known

a

model-pickupdrop

sequence

should

result

If

both

are

known

a

pickup-drop

sequence

should

result

In

the

data

pickup-drop

sequences

were

invariably

the

last

one

or

sometimes

two

blocks

in

the

sequence

Thus

with

respect

to

color

and

location

the

model-pickup-model-drop

sequences

are

memoryless

and

model-pickup-drop

pickup-model-drop

and

pickup-drop

sequences

can

be

explained

if

the

subjects

are

sometimes

able

to

remember

an

extra

location

and/or

color

when

they

fixate

the

model

area

Summary

data

for

seven

subjects

is

shown

as

the

dark

bars

in

Figure

The

lowest-memory

model-pickup-model-drop

strategy

is

the

most

frequently

used

by

all

the

subjects

far

outweighing

the

others

Note

that

if

subjects

were

able

to

complete

the

task

from

memory

then

a

sequence

composed

exclusively

of

pickup-drops

could

have

been

observed

but

instead

the

pickup-drop

strategy

is

usually

used

only

near

the

end

of

the

construction

The

frequent

access

to

the

model

area

during

the

construction

of

the

copy

we

take

as

direct

evidence

of

incremental

access

to

information

in

the

world

in

the

process

of

performing

the

task

As

the

task

progresses

the

markers

color

and

location

in

this

case

are

reset

as

the

new

information

is

called

for

Minimal

Memory

Strategies

Another

interesting

result

is

the

time

required

for

each

strategy

when

the

target

is

in

view

The

time

tallies

are

shown

in

Table

along

with

the

putative

memory

load

for

color

and

location

for

each

strategy

What

is

seen

is

that

the

lower

memory

strategies

take

longer

This

is

not

too

surprising

as

the

number

of

fixations

goes

down

if

items

can

be

memorized

However

it

is

surprising

that

subjects

choose

minimal

memory

strategies

in

view

of

their

temporal

cost

particularly

as

they

have

been

instructed

to

complete

the

task

as

quickly

as

possible

and

memorization

saves

time

The

reluctance

to

use

working

memory

to

capacity

can

be

explained

if

such

memory

is

expensive

M-P-M-D

P-M-D

M-P-D

P-D

Other

Eye

Fixation

Category

Relative

Frequency

M-P-M-D

M-P-D

P-DP-M-D

a

-RRB-

b

-RRB-

Figure

-LRB-

a

-RRB-

The

codes

M

means

that

the

eyes

are

directed

to

the

model

P

and

D

mean

that

the

eyes

and

cursor

are

coincident

at

the

pickup

point

and

drop-off

point

respectively

Thus

for

the

PMD

strategy

the

eye

goes

directly

to

the

source

for

pickup

then

to

the

model

area

and

then

to

the

source

for

drop-off

-LRB-

b

-RRB-

The

relative

frequency

of

the

different

strategies

for

four

subjects

to

use

with

respect

to

the

cost

of

the

serializing

strategy

Our

experiments

support

the

hypothesis

that

working

memory

is

expensive

compared

to

the

cost

of

acquiring

the

information

on-line

for

the

technical

reasons

discussed

in

Section

so

that

low

memory

strategies

are

preferred

This

hypothesis

would

predict

that

if

the

cost

of

the

on-line

acquisition

of

information

could

be

increased

relative

to

that

of

memorizing

the

balance

of

effort

should

shift

in

the

direction

of

increased

memorization

To

test

this

the

cost

of

on-line

acquisition

was

increased

by

moving

the

model

and

copy

from

their

previous

position

underneath

one

another

to

eccentric

positions

separated

by

degrees

Under

these

conditions

subjects

use

memory

more

as

reflected

in

fewer

eye

movements

to

the

model

area

The

number

of

eye

movements

decreases

from

an

average

of

per

block

to

per

block

The

analysis

makes

the

assumption

that

the

individual

blocks

are

primitives

for

the

task

This

implies

that

the

eye

movements

back

to

the

model

primarily

serve

to

obtain

properties

of

individual

blocks

An

alternate

explanation

is

that

the

extra

movements

to

the

model

area

are

in

fact

not

essential

but

instead

appear

because

of

some

other

effect

that

is

not

being

modeled

-LRB-

One

such

explanation

is

that

the

eyes

move

faster

than

the

hand

so

that

there

will

be

extra

time

to

check

the

model

in

a

way

that

is

unrelated

to

the

properties

of

individual

blocks

-RRB-

However

a

control

in

which

all

the

blocks

were

a

single

color

argues

that

this

is

unlikely

In

this

control

experiment

conditions

were

identical

to

the

standard

experiment

with

the

exception

that

all

the

blocks

were

one

color

There

was

a

dramatic

decrease

in

the

number

of

eye

movements

used

to

inspect

the

model

area

per

block

in

the

monochrome

case

versus

per

block

in

the

multicolored

case

A

closer

inspection

of

the

individual

trials

in

the

monochrome

case

reveals

that

subjects

copy

subpatterns

without

reference

to

the

model

suggesting

that

they

are

able

to

recognize

these

composite

shapes

In

this

case

subjects

do

abandon

eye

movements

to

the

model

Thus

we

conclude

that

the

movements

to

the

model

in

the

standard

case

are

necessary

and

related

to

the

individual

properties

of

blocks

Experiments

using

Saccade-Contingent

Display

Changes

Performance

in

the

blocks

task

provides

plausible

evidence

that

subjects

use

fixation

as

a

deictic

pointing

device

to

serialize

the

task

and

allow

incremental

access

to

the

immediately

task-relevant

information

However

it

is

important

to

attempt

a

more

direct

verification

One

way

to

do

this

is

to

explore

exactly

what

visual

information

is

retained

in

visual

memory

from

prior

fixations

by

changing

various

aspects

of

the

display

during

task

performance

Changing

information

that

is

critical

for

the

task

should

disrupt

performance

in

some

way

In

one

experiment

we

changed

the

color

of

one

of

the

uncopied

blocks

while

the

subject

was

making

a

saccade

to

the

model

area

following

a

block

placement

The

changes

were

made

when

the

subject

's

eyes

crossed

the

boundary

between

the

workspace

and

model

area

Fixations

in

the

workspace

area

are

almost

invariably

for

the

purpose

of

guiding

the

placement

of

a

block

in

the

partially

completed

copy

If

the

S

follows

this

placement

with

a

saccade

to

the

model

area

the

implication

is

that

the

S

currently

has

no

color

information

or

location

information

in

memory

and

is

fixating

the

model

to

acquire

this

Model

Resource

Workspace

Model

Resource

Workspace

a

b

Figure

Two

different

experimental

conditions

for

the

color-changing

experiment

In

-LRB-

a

-RRB-

the

color

of

an

uncopied

model

block

is

changed

during

a

workspace-to-model

saccade

In

-LRB-

b

-RRB-

the

color

of

an

uncopied

model

block

is

changed

during

a

resource-to-model

saccade

information

It

is

not

clear

on

what

basis

the

saccade

to

the

model

area

is

programmed

although

they

tend

to

be

close

to

the

previous

fixation

In

the

first

condition

illustrated

in

Figure

the

color

of

a

block

in

the

model

was

changed

during

the

saccade

to

the

model

following

block

placement

in

the

workspace

when

the

S

was

beginning

to

work

on

the

next

block

This

is

indicated

by

the

zig-zag

The

small

arrow

indicates

the

color

change

in

the

block

In

another

condition

shown

in

Figure

the

change

was

made

after

the

S

had

picked

up

a

block

and

was

returning

to

the

model

presumably

to

check

its

location

In

both

conditions

the

changed

block

was

chosen

randomly

Data

for

one

subject

are

shown

in

Figure

We

measured

the

total

time

the

subject

spent

fixating

in

the

model

area

under

different

conditions

For

Figure

these

conditions

were

as

follows

-LRB-

-RRB-

The

TARGET

condition

the

subject

was

making

a

saccade

from

the

workspace

area

to

the

model

area

and

the

block

fixated

at

the

end

of

the

saccade

had

just

undergone

a

color

change

during

the

saccade

-LRB-

-RRB-

The

NON

TARGET

condition

there

was

a

color

change

when

the

subject

made

a

saccade

but

the

block

fixated

at

the

end

of

the

saccade

was

not

the

one

that

had

changed

color

-LRB-

-RRB-

The

CONTROL

NO

SWAP

condition

the

subject

made

a

saccade

to

the

model

area

but

no

blocks

changed

color

There

is

no

effect

of

the

color

swap

on

the

subsequent

fixation

even

if

this

block

is

the

target

of

the

saccade

This

suggests

that

the

target

selection

involved

in

programming

the

saccade

into

the

model

does

not

involve

the

acquisition

of

color

information

at

that

location

and

that

this

function

occurs

during

the

fixation

in

the

model

area

This

implies

that

rather

minimal

information

is

retained

from

the

immediately

prior

fixation

and

is

consistent

with

the

suggestion

that

fixation

is

used

for

acquiring

information

just

prior

to

its

use

A

validation

of

fixation

duration

as

a

sensitive

measure

is

given

by

the

data

in

Figure

This

Figure

Results

of

the

color-changing

experiments

The

leftmost

bar

shows

the

fixation

time

when

the

target

of

the

fixation

was

the

next

block

moved

The

middle

bar

shows

the

case

when

it

was

not

The

rightmost

bar

is

the

no-change

control

For

the

condition

in

Figure

there

is

no

change

in

fixation

time

The

condition

in

Figure

shows

a

dramatic

increase

in

fixation

time

when

the

target

block

's

color

has

changed

shows

the

same

manipulation

at

a

different

stage

of

task

performance

when

the

subject

had

just

picked

up

a

block

and

was

returning

to

the

model

area

perhaps

to

get

location

information

In

this

case

changes

in

color

of

the

saccade

target

almost

doubled

the

fixation

duration

indicating

that

the

color

information

has

been

retained

when

it

is

currently

task-relevant

In

most

instances

Ss

were

not

aware

of

the

change

but

fixated

a

neighboring

block

of

the

color

of

the

block

they

were

holding

and

placed

it

in

the

appropriate

new

location

The

longer

time

spent

in

the

model

area

in

Figure

reflects

this

additional

fixation

Thus

the

information

retained

from

prior

saccades

is

determined

by

what

is

currently

relevant

for

the

task

Implications

It

appears

that

human

performance

in

tasks

like

these

can

be

quite

nicely

characterized

as

a

sequence

of

deictic

instructions

based

on

a

small

number

of

primitive

operations

This

obviates

the

need

for

complex

memory

representations

These

results

suggest

a

computational

interpretation

of

the

limitations

of

human

working

memory

Rather

than

being

thought

of

as

a

limitation

on

processing

capacity

it

can

be

seen

as

an

integral

part

of

a

system

that

makes

dynamic

use

of

deictic

variables

The

limited

number

of

variables

need

only

be

a

handicap

if

the

entire

task

is

to

be

completed

from

memory

in

that

case

the

short-term

memory

system

is

overburdened

In

the

more

natural

case

of

performing

the

task

with

ongoing

access

to

the

visual

world

the

task

is

completed

perfectly

This

suggests

that

a

natural

metric

for

evaluating

behavioral

programs

can

be

based

on

their

spatio-temporal

information

requirements

These

results

also

support

the

interpretation

of

the

role

of

foveating

eye

movements

outlined

in

Section

Since

Yarbus

's

classic

observations

-LSB-

Yarbus

-RSB-

saccadic

eye

movements

have

often

been

thought

to

reflect

cognitive

events

in

addition

to

being

driven

by

the

poor

resolution

of

peripheral

vision

However

making

this

link

has

proved

sufficiently

difficult

so

as

to

raise

questions

about

how

much

can

be

learned

about

cognitive

operations

by

inspecting

the

fixation

patterns

-LSB-

Viviani

-RSB-

As

discussed

above

one

of

the

difficulties

of

relating

fixations

to

cognitive

processes

is

that

fixation

itself

does

n't

indicate

what

properties

are

being

acquired

In

the

blockcopying

paradigm

however

fixation

appears

to

be

quite

tightly

linked

to

the

underlying

processes

by

marking

the

location

at

which

information

-LRB-

e.g.

color

relative

location

-RRB-

is

to

be

acquired

or

the

location

that

specifies

the

target

of

the

hand

movement

-LRB-

picking

up

putting

down

-RRB-

Thus

fixation

can

be

seen

as

binding

the

value

of

the

variable

currently

relevant

for

the

task

Our

ability

to

relate

fixations

to

cognitive

processes

in

this

instance

is

a

consequence

of

our

ability

to

provide

an

explicit

description

of

the

task

In

previous

attempts

to

glean

insight

from

eye

movements

-LRB-

e.g.

viewing

a

scene

or

identifiying

a

pattern

-RRB-

the

task

demands

are

not

well

specified

or

observable

As

described

in

Section

-LSB-

Kowler

and

Anton

-RSB-

made

a

similar

suggestion

for

the

role

of

fixation

in

binding

variables

They

found

that

saccade

size

in

reading

was

determined

by

the

task

saccades

are

approximately

word-by-word

for

normal

text

but

letter-by-letter

for

scrambled

text

with

the

letters

reversed

That

is

fixation

patterns

reflected

the

kind

of

information

called

for

by

the

task

-LRB-

words

or

letters

-RRB-

and

not

the

demands

of

spatial

resolution

Other

evidence

for

the

importance

of

fixations

in

visual

computations

comes

from

the

findings

of

-LSB-

Schlingensiepen

et

al.

Just

and

Carpenter

-RSB-

who

showed

that

eye

movements

appear

to

be

required

for

making

same/different

judgments

of

complex

patterns

In

addition

to

this

in

investigations

of

chess

playing

-LSB-

Chase

and

Simon

-RSB-

showed

that

eye

fixations

are

intimately

related

to

spatial

working

memory

although

they

did

not

suggest

a

definitive

computational

role

for

fixations

Finally

note

that

our

suggestion

for

the

role

of

fixations

is

rather

different

from

that

of

-LSB-

Noton

and

Stark

-RSB-

who

hypothesized

that

the

eye

movements

themselves

indexed

a

memory

representation

We

can

now

return

to

reexamine

the

computational

hypothesis

illustrated

in

Table

in

the

light

of

our

observations

of

human

performance

in

this

task

We

can

think

of

task

performance

as

being

quite

well

explained

by

the

successive

application

of

three

operations

of

the

kind

illustrated

there

Thus

a

model

fixation

will

acquire

visual

properties

-LRB-

color

relative

location

-RRB-

at

the

marked

-LRB-

fixated

-RRB-

location

-LRB-

cf.

the

identification

or

where

box

in

Table

-RRB-

This

will

be

followed

by

a

visual

search

operation

to

find

the

target

color

in

the

resource

or

the

putdown

location

in

the

workspace

-LRB-

the

location

or

where

box

-RRB-

saccade

programming

to

that

location

and

visual

guidance

of

the

hand

to

the

marked

-LRB-

fixated

-RRB-

location

-LRB-

the

manipulation

box

-RRB-

To

complete

the

task

we

need

in

addition

the

operation

of

holding

a

very

small

number

of

properties

of

the

model

pattern

in

working

memory

and

programming

the

ballistic

phase

of

the

hand

movement

It

is

of

interest

to

consider

how

these

operations

might

correspond

to

the

classic

division

of

visual

pathways

into

dorsal

and

ventral

streams

-LSB-

Ungerleider

and

Mishkin

-RSB-

These

have

been

postulated

to

underlie

what

and

where

or

what

and

how

visual

operations

-LRB-

alternatively

perception

and

action

e.g.

-LSB-

Ungerleider

and

Mishkin

Goodale

and

Milner

-RSB-

-RRB-

One

possibility

might

be

that

the

dorsal

stream

to

parietal

cortex

is

involved

in

positioning

and

keeping

track

of

the

markers

-LRB-

e.g.

programming

the

saccade

to

the

location

identified

by

visual

search

-RRB-

and

that

the

ventral

stream

to

inferotemporal

cortex

does

the

computations

for

the

acquisition

of

information

-LRB-

e.g.

color

relative

location

or

local

configuration

in

the

pattern

-RRB-

at

the

marked

location

-LRB-

The

visual

search

operation

can

be

seen

as

involving

both

streams

-RRB-

Manipulation

or

hand

movements

might

involve

other

parts

of

cortex

acting

relative

to

the

location

marker

given

by

the

parietal

cortex

using

properties

-LRB-

such

as

size

or

orientation

-RRB-

given

by

the

temporal

cortex

This

conceptualization

provides

a

computational

underpinning

for

the

classic

what/where

distinction

and

is

somewhat

different

from

the

what/how

distinction

One

issue

is

how

it

could

explain

the

behavior

of

Goodale

's

patient

DF

who

displayed

normal

hand

shaping

behavior

in

the

context

of

reaching

movements

together

with

an

inability

to

report

the

visual

properties

that

appeared

to

guide

the

hand

shaping

-LSB-

Goodale

and

Milner

-RSB-

On

the

view

we

have

presented

we

would

need

to

postulate

some

basis

for

the

impaired

ability

to

report

visual

properties

other

than

a

loss

of

this

function

in

inferotemporal

cortex

Since

the

damage

appears

to

be

largely

in

the

white

matter

it

is

possible

that

some

aspect

of

the

circuitry

involved

in

reporting

the

perceptual

decision

is

damaged

rather

than

the

computation

itself

Deictic

Strategies

and

Cerebral

Organization

Deictic

actions

necessitate

the

use

of

functional

models

This

is

illustrated

by

the

blocks

task

very

dramatically

particularly

by

the

experiments

that

switch

colors

during

saccades

These

experiments

are

the

principal

evidence

that

the

brain

seems

to

postpone

representing

information

until

just

before

it

is

required

An

obvious

reason

for

timely

task-dependent

computation

is

that

its

products

are

so

varied

that

the

brain

can

not

pre-compute

them

Consider

all

the

information

that

one

might

have

to

know

about

an

image

An

economical

way

of

computing

this

perhaps

the

only

way

is

by

tailoring

the

computation

to

just

that

required

by

the

task

demands

as

they

become

known

In

the

blocks

task

at

one

point

in

time

subjects

need

a

block

of

an

appropriate

color

and

at

another

point

they

need

to

know

where

to

put

that

block

At

both

of

these

times

they

are

fixating

the

model

area

in

the

same

place

The

key

difference

is

that

they

need

to

apply

a

different

computation

to

the

same

image

data

During

the

first

fixation

subjects

need

to

extract

the

color

of

the

next

block

during

the

second

fixation

they

need

the

relative

offset

of

the

next

block

with

respect

to

the

model

pattern

The

hypothesis

that

vision

must

be

functional

depends

on

some

mechanism

of

spanning

the

space

of

task-dependent

representations

One

way

of

doing

this

is

to

compose

primitive

routines

Thus

at

the

level

of

abstraction

below

the

embodiment

level

one

can

think

of

a

set

of

more

primitive

instructions

that

implement

the

binding

required

by

the

embodiment

level

In

this

section

we

describe

how

these

primitives

might

work

and

how

their

functionality

might

map

onto

brain

anatomy

Functional

Routines

Although

the

functional

primitives

could

exist

for

any

modality

let

us

concentrate

on

vision

Visual

routines

were

first

suggested

by

-LSB-

Kosslyn

Just

and

Carpenter

-RSB-

but

-LSB-

Ullman

-RSB-

developed

the

essential

arguments

for

them

Ullman

's

visual

routines

had

a

graphics

flavor

in

contrast

our

main

motivation

is

to

show

how

visual

routines

can

support

deictic

computation

The

two

main

features

of

deictic

visual

routines

are

-LRB-

-RRB-

the

ability

to

mark

aspects

of

the

physical

environment

and

-LRB-

-RRB-

the

ability

to

process

marked

locations

These

are

simply

the

what

and

where

subtasks

described

in

Section

We

will

concentrate

on

just

the

core

of

both

of

these

problems

which

is

some

way

of

marking

locations

in

the

world

The

way

we

suggest

is

by

using

the

output

of

low-level

filters

such

as

those

known

to

exist

in

striate

cortex

For

example

one

way

of

marking

locations

is

by

color

this

is

particularly

useful

in

the

blocks

experiments

since

individual

blocks

have

distinct

colors

-LRB-

although

there

are

several

blocks

with

the

same

color

-RRB-

Reconsider

the

task

of

identifying

the

next

block

to

use

when

the

subject

is

fixating

in

the

model

area

One

way

to

define

the

location

of

a

block

would

be

to

be

able

to

ask

Where

are

the

locations

of

that

block

's

color

on

the

retinotopic

array

at

this

moment

This

constraint

alone

is

insufficient

but

additional

constraints

can

be

added

to

denote

the

resource

area

in

terms

of

its

relative

location

to

the

fixation

point

Note

that

defining

this

area

is

difficult

as

it

is

a

location

in

space

that

will

occupy

a

different

part

of

the

retinotopic

array

depending

on

the

location

of

the

eyes

Thus

one

solution

could

use

routines

that

ffl

compare

the

block

's

color

to

each

of

the

retinal

locations

that

describe

color

ffl

add

the

constraint

that

the

resource

area

is

to

the

right

of

the

display

to

determine

the

candidates

of

the

appropriate

color

in

the

resource

area

and

ffl

choose

one

of

these

candidates

for

the

next

saccadic

eye

movement

The

first

routine

uses

just

one

color

but

Swain

has

shown

in

a

number

of

experiments

that

its

strategy

works

even

better

with

natural

objects

of

many

different

colors

-LSB-

Swain

and

Ballard

-RSB-

This

is

because

the

histogram

of

an

object

's

colors

in

tri-chromatic

space

is

an

almost

unique

identifier

In

fact

both

computer

simulations

-LSB-

Mel

Rao

and

Ballard

-RSB-

as

well

as

formal

analyses

suggest

that

the

more

histogrammed

features

that

are

used

to

define

locations

the

better

One

way

of

augmenting

the

description

is

to

add

local

photometric

intensity

variations

in

the

form

of

responses

to

achromatic

basis

templates

These

codes

express

the

principal

components

of

photometric

values

near

a

point

in

terms

of

a

sum

of

a

fixed

set

of

about

forty-five

filters

-LSB-

Hancock

et

al.

-RSB-

These

filters

vary

in

scale

and

orientation

and

resemble

the

kind

of

filtering

performed

by

striate

cortical

cells

For

practical

purposes

these

responses

provide

a

unique

iconic

zip-code

that

identifies

that

particular

point

as

shown

in

Figure

By

fixating

at

a

point

the

responses

of

the

basis

templates

can

be

recorded

A

where

problem

is

to

return

the

gaze

to

the

marked

point

when

the

gaze

is

not

at

that

point

but

the

features

of

the

marked

point

are

accessible

via

working

memory

In

this

case

the

location

of

the

point

relative

to

the

current

fixation

can

be

determined

by

matching

the

remembered

zip-code

with

the

zip-codes

at

all

the

current

locations

The

best

match

defines

the

target

point

as

shown

in

Figure

When

the

eyes

are

fixating

a

given

point

the

zip-code

is

stored

in

working

memory

Later

at

the

moment

the

eyes

are

required

to

return

to

that

point

the

remembered

responses

are

compared

to

the

current

responses

on

the

retina

The

best

match

defines

a

motor

target

that

can

be

fixated

Conversely

for

the

what

problem

it

is

assumed

that

the

object

to

be

identified

is

at

the

fixation

point

In

this

case

the

feature

vector

at

the

fovea

is

compared

to

a

stored

library

of

filter

templates

For

other

problems

such

as

computing

the

offset

additional

routines

are

needed

but

they

are

of

the

same

character

Brain

Subsystems

At

this

time

an

overall

picture

of

brain

functionality

is

a

distant

goal

Yet

so

much

new

information

about

the

various

brain

subsystems

has

recently

been

discovered

that

one

can

at

least

attempt

to

piece

together

a

picture

of

how

the

functionality

required

by

markers

might

be

implemented

Enough

is

known

about

the

various

brain

subsystems

to

suggest

that

the

operations

that

compose

a

deictic

action

would

use

different

brain

subsystems

for

different

functions

these

brain

subsystems

should

be

temporally

coordinated

in

very

specific

ways

The

main

argument

is

that

at

time

scales

of

a

third

of

a

second

the

circuitry

is

used

functionally

to

subserve

the

demands

of

the

ongoing

tasks

that

the

brain

is

in

the

process

of

carrying

out

This

functionality

forms

representations

but

only

very

transiently

For

example

in

the

process

of

matching

can

be

vulnerable

to

lighting

changes

it

is

vulnerable

to

transformations

such

as

scale

and

rotation

and

its

storage

requirements

can

be

prohibitive

However

recent

developments

-LSB-

Buhmann

et

al.

Jones

and

Malik

-RSB-

have

ameliorated

these

disadvantages

If

the

template

is

created

dynamically

and

has

a

limited

lifetime

then

the

first

objection

is

less

important

as

lighting

conditions

are

likely

to

be

constant

over

the

duration

of

the

task

As

for

the

second

and

third

objections

special

compact

codes

for

templates

can

overcome

these

difficulties

-LRB-

d

-RRB-

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

Figure

Demonstration

that

photometric

basis

templates

uniquely

define

points

in

the

image

even

under

some

view

variation

Shown

for

three

points

are

the

responses

to

basis

filters

as

bars

from

a

zero

baseline

Positive

responses

are

above

the

basline

negative

responses

are

below

In

our

model

marking

these

responses

corresponds

to

placing

them

in

working

memory

The

responses

themselves

are

the

contents

of

the

marker

-LRB-

a

-RRB-

Original

gaze

point

-LRB-

b

-RRB-

Gaze

point

after

a

degree

rotation

-LRB-

c

-RRB-

Nearby

gaze

point

-LRB-

d

-RRB-

Vector

filter

responses

for

all

three

points

Note

how

the

first

two

are

almost

identical

but

are

very

different

from

the

third

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

d

-RRB-

-LRB-

c

-RRB-

Figure

Using

intensity

information

to

define

locations

-LRB-

a

-RRB-

Original

gaze

point

whose

response

vector

is

placed

in

working

memory

-LRB-

b

-RRB-

New

gaze

point

-LRB-

c

-RRB-

Correlation

or

saliency

image

for

the

marked

point

obtained

by

comparing

the

remebered

response

vector

to

retinotopic

response

vectors

from

the

current

image

Lightness

defines

the

most

salient

points

in

retinotopic

coordinates

which

in

turn

are

directly

related

to

motor

error

-LRB-

d

-RRB-

Using

the

maximum

saliency

point

to

saccade

back

to

the

original

point

changing

fixation

the

target

is

selected

then

located

then

related

to

motor

coordinates

Each

one

of

these

steps

represents

the

world

differently

and

does

so

every

time

an

eye

movement

is

made

Section

developed

the

idea

of

minimal

representations

with

respect

to

the

minimal

number

of

markers

to

describe

how

markers

factor

into

computation

at

the

embodiment

level

but

one

should

not

think

that

the

information

represented

by

a

particular

marker

is

small

This

concerns

the

contents

of

a

marker

or

the

information

that

is

the

marker

referent

The

content

of

a

marker

may

be

iconic

and

of

considerable

size

Now

to

a

crude

first

approximation

let

us

think

of

the

cortex

as

a

kind

of

content-addressible

memory

As

such

the

cortex

can

be

modeled

as

holding

the

contents

of

a

marker

The

main

point

is

that

if

the

cortex

holds

the

contents

of

markers

additional

extracortical

neural

circuitry

is

required

to

realize

the

different

functionality

such

as

visual

routines

that

manipulate

these

contents

For

example

all

of

the

following

constraints

need

to

be

met

the

sequence

in

the

task

determines

what

sensory

processing

should

be

done

the

processing

for

deictic

pointing

is

most

efficaciously

carried

out

in

a

representation

that

uses

retinotopic

data

the

results

of

this

process

must

be

linked

to

working

memory

The

logical

place

for

the

calculations

of

visual

routines

is

in

the

retinotopically-indexed

areas

of

cortex

but

the

different

characteristics

of

many

different

areas

may

be

needed

to

implement

the

complete

functionality

of

a

marker-manipulation

task

In

order

to

develop

this

idea

let

us

briefly

review

some

important

ideas

about

two

key

subsystems

ffl

The

basal

ganglia

is

an

extensive

sub-cortical

nucleus

implicated

in

the

learning

of

motor

program

sequences

Independently

-LSB-

Strick

et

al.

-RSB-

and

-LSB-

Miyachi

et

al.

-RSB-

have

shown

that

neurons

in

the

basal

ganglia

that

respond

to

task-specific

sub-sequences

emerge

in

the

course

of

training

Schultz

has

shown

that

basal

ganglia

neurons

learn

to

predict

reward

When

a

monkey

initially

reaches

into

an

enclosed

box

for

an

apple

these

neurons

respond

when

the

fingers

touch

the

apple

If

a

light

is

paired

with

the

apple

reward

in

advance

of

reaching

into

the

box

the

same

neurons

develop

responses

to

the

light

and

not

to

the

actual

touching

These

neurons

are

dopamanergic

that

is

they

are

responsible

for

one

of

the

main

chemical

reward

systems

used

by

the

brain

-LSB-

Schultz

et

al.

-RSB-

Thus

the

implication

is

that

the

monkey

is

learning

to

predict

delayed

reward

and

coding

it

via

an

internal

dopamine

messenger

The

basal

ganglia

has

extensive

connections

to

cortex

that

emphasize

frontal

and

mortor

cortex

-LSB-

Graybiel

and

Kimura

-RSB-

ffl

The

hippocampus

is

an

extensive

sub-cortical

nucleus

implicated

in

the

enabling

of

working

memory

Lesions

of

the

hippocampus

allow

the

carrying

out

of

tasks

that

require

working

memory

but

impair

the

ability

to

retain

these

memories

The

hippocampus

has

extensive

connections

to

cortex

that

heavily

emphasize

abstract

areas

of

cortex

-LSB-

Stefanacci

et

al.

-RSB-

-RSB-

The

point

is

that

the

functionality

that

supports

different

aspects

of

a

marker-related

task

requires

the

coordinated

activity

of

different

places

in

the

brain

Broadly

speaking

information

about

program

sequence

is

represented

in

the

basal

ganglia

and

information

as

to

the

retention

of

items

in

working

memory

depends

on

the

hippocampus

Recent

evidence

suggests

that

these

different

operations

are

in

fact

closely

linked

in

momentary

cognitive

processing

In

recent

studies

of

Parkinson

's

patients

a

disease

associated

with

damage

to

the

basal

ganglia

such

patients

performing

a

task

very

like

the

blocks

task

have

revealed

deficits

in

working

memory

-LSB-

Gabrieli

-RSB-

Consider

the

interaction

of

vision

and

action

If

the

is

the

gateway

that

enables

the

retention

of

temporary

visual

information

and

if

a

basal

ganglia

deficit

produces

a

working

memory

deficit

the

simplest

implication

is

that

this

circuitry

functions

in

a

closed

loop

This

is

understandable

as

the

basal

ganglia

represents

motor

program

sequencing

information

the

visual

cortex

represents

potential

locations

and

the

hippocampus

enables

extracted

locations

to

be

saved

in

working

memory

Thus

the

disparate

purposes

of

saccadic

fixations

to

the

same

part

of

visual

space

in

the

blocks

task

can

be

resolved

by

the

basal

ganglia

which

represents

essential

programmatic

temporal

context

on

why

those

fixations

are

taking

place

and

when

this

information

should

be

used

Another

specific

example

makes

the

same

point

Experiments

in

cortical

area

work

implicates

the

entorhinal

cortex

as

the

source

of

some

of

the

effects

previously

associated

with

hippocampus

Also

hippocampus

may

have

a

role

in

the

transfer

of

short

to

long-term

memory

rather

than

being

the

source

of

the

representation

of

short-term

memory

For

the

purposes

of

discussion

however

these

very

important

points

are

details

it

is

only

necessary

that

this

general

area

be

a

necessary

part

of

the

total

functioning

of

short-term

memory

have

shown

that

there

are

memory

representations

for

the

next

eye

movement

in

motor

coordinates

-LSB-

Goldman-Rakic

-RSB-

However

this

representation

does

not

necessarily

contain

the

information

as

to

when

this

information

is

to

be

used

that

kind

of

information

is

part

of

a

motor

program

such

as

might

be

found

in

the

basal

ganglia

For

this

reason

the

anatomical

connections

of

the

basal

ganglia

should

be

very

important

as

they

may

have

to

influence

the

earliest

visual

processing

The

essential

point

of

the

above

discussion

is

that

the

behavioral

primitives

at

the

embodiment

level

necessarily

involve

most

of

the

cortical

circuitry

and

that

at

the

one-third

second

time

scale

one

can

not

think

of

parts

of

the

brain

in

isolation

This

point

has

also

been

extensively

argued

by

Fuster

-LSB-

Fuster

Fuster

-RSB-

Thus

our

view

is

similar

to

-LSB-

Van

Essen

et

al.

-RSB-

in

that

they

also

see

the

need

for

attentional

dynamic

routing

of

information

but

different

in

that

we

see

the

essential

need

for

the

basal

ganglia

to

indicate

program

temporal

context

Van

Essen

et

al.

's

circuitry

is

restricted

to

the

pulvinar

-LSB-

Kosslyn

-RSB-

has

implicated

the

early

visual

areas

of

striate

cortex

in

functional

processing

in

a

number

of

PET

studies

but

he

too

has

not

addressed

the

mechanisms

of

temporal

context

Marker

Routines

and

Brain

Subsystems

Let

us

now

speculate

on

how

the

visual

routines

that

support

marker

computation

could

be

implemented

in

terms

of

the

overall

cerebral

architecture

In

particular

consider

the

use

of

markers

to

solve

both

what

and

where

problems

First

consider

the

implementation

of

the

use

of

markers

to

remember

locations

particularly

from

the

perspective

of

a

control

program

running

in

the

basal

ganglia

That

program

keeps

track

of

when

it

is

appropriate

to

load

working

memory

Think

of

this

command

as

remember

what

I

am

fixating

In

the

task

there

are

many

different

fixations

with

different

uses

so

there

has

to

be

some

mechanism

that

uses

the

task

context

to

pick

the

appropriate

one

for

a

given

segment

Note

also

that

the

basal

ganglia

via

its

connections

to

retinotopic

cortex

via

thalamus

or

prefrontal

cortex

has

the

necessary

connections

to

send

this

command

-LSB-

Miyachi

et

al.

Fuster

-RSB-

This

putative

function

is

depicted

schematically

in

Figure

This

highly

schematized

figure

suggests

the

overall

control

pathways

The

retinotopically-indexed

neurons

of

thalamus

and

cortex

are

summarized

in

terms

of

a

single

element

that

depicts

the

enhanced

foveal

representation

In

the

blocks

experiment

a

subject

fixates

identical

spatial

locations

under

different

task

contexts

i.e.

in

one

case

seeking

color

features

and

in

another

case

location

features

For

this

reason

we

posit

that

the

basal

ganglia

motor

program

plays

the

crucial

role

of

providing

context

to

the

visual

pathway

so

that

appropriate

kinds

of

visual

processing

can

be

tailored

to

each

situation

Now

consider

the

converse

problem

the

task

context

requires

that

the

gaze

be

returned

to

a

remembered

location

This

could

be

done

in

principle

by

matching

the

features

of

the

desired

location

with

features

currently

on

the

retina

To

do

so

requires

two

main

functional

changes

depicted

in

Figure

First

the

circuitry

of

visual

cortex

is

to

be

used

in

a

different

way

Now

the

features

are

communicated

via

feedback

connections

to

a

point

of

comparison

with

their

counterparts

on

Basal

Ganglia

Hippocampus

InferoTemporal

Cortex

Retinotopic

feature

representations

Thalamus

Targetrepresentations

Parietal

cortex

SC

Thalamus

fovea

fovea

VisualInput

Figure

When

the

task

context

requires

that

the

fixated

location

's

features

be

remembered

the

basal

ganglia

can

cue

the

thalamic

input

pathway

through

explicit

connections

-LRB-

solid

arrows

-RRB-

Basal

Ganglia

Hippocampus

InferoTemporal

Cortex

Retinotopic

feature

representations

Thalamus

Targetrepresentations

Parietal

cortex

SC

Thalamus

fovea

fovea

MotorOutput

VisualInput

Figure

When

the

task

context

requires

that

the

location

of

remembered

features

be

recovered

the

basal

ganglia

can

cue

the

motor

output

pathway

through

explicit

connections

-LRB-

solid

arrows

-RRB-

the

retinotopically-indexed

cortex

Second

the

task

context

specifies

how

the

visual

cortical

circuitry

should

be

used

Once

again

the

basal

ganglia

has

the

natural

role

of

making

this

happen

In

this

case

we

posit

that

the

comparison

may

take

place

in

a

motor

output

area

such

as

parietal

cortex

or

the

superior

colliculus

Herein

previously

remembered

features

would

be

compared

with

retinotopically-indexed

features

from

all

current

locations

The

best

match

would

be

the

candidate

current

location

of

that

feature

The

two

main

points

of

both

of

these

examples

are

-LRB-

-RRB-

although

the

cortex

is

in

some

sense

the

site

of

visual

representations

the

functional

manipulation

of

these

representations

should

come

from

extra-cortical

circuitry

and

-LRB-

-RRB-

the

logical

site

of

these

computations

is

in

retinotopically-indexed

visual

cortex

Discussion

and

Conclusions

The

focus

of

this

paper

has

been

an

abstract

model

of

computation

that

describes

the

interfacing

of

the

body

's

apparatus

to

the

brain

's

behavioral

programs

If

we

view

the

brain

as

hierarchically

organized

we

can

differentiate

between

processes

that

occur

at

different

spatial

and

temporal

scales

It

is

important

to

do

this

because

the

nature

of

the

computations

at

each

level

are

different

Examination

of

the

embodiment

level

provides

a

crucial

link

between

elemental

perceptual

events

that

occur

on

a

shorter

time

scale

of

msec

such

as

recognition

and

events

at

the

level

of

cognitive

tasks

-LRB-

symbolic

programs

-RRB-

that

occur

on

a

longer

time

scale

of

seconds

The

importance

of

examining

this

level

is

that

body

movements

have

a

natural

computational

interpretation

in

terms

of

deictic

pointers

because

of

the

ability

of

the

different

sensory

modalities

to

quickly

direct

their

foci

to

localized

parts

of

space

As

such

deictic

computation

provides

a

mechanism

for

representing

the

essential

features

that

link

external

sensory

data

with

internal

cognitive

programs

and

motor

actions

In

Section

we

explored

the

computational

advantages

of

sequential

deictic

programs

for

behavior

In

Section

we

presented

evidence

that

humans

indeed

use

fixation

in

a

way

that

is

consistent

with

this

computational

strategy

When

performing

natural

tasks

subjects

make

moment-by-moment

tradeoffs

between

the

visual

information

maintained

in

working

memory

and

that

acquired

by

eye

fixations

This

serialization

of

the

task

with

frequent

eye

movements

is

consistent

with

the

interpretation

of

working

memory

whereby

fixation

is

used

as

a

deictic

pointing

device

to

dynamically

bind

items

in

working

memory

In

Section

we

examined

how

component

lowlevel

routines

-LRB-

that

is

at

the

level

of

perceptual

acts

-RRB-

might

be

computed

and

effect

the

movement

of

markers

and

how

the

composition

of

behaviors

at

the

task

level

might

be

implemented

in

brain

circuitry

using

such

routines

Deictic

codes

provide

compact

descriptions

of

the

sensory

space

that

have

many

advantages

for

cognitive

programs

These

are

summarized

as

follows

The

facilitation

of

spatio-temporal

reference

Sequential

computations

in

cognitive

tasks

make

extensive

use

of

the

body

's

ability

to

orient

The

chief

example

of

this

is

in

the

ability

of

the

eyes

to

fixate

on

a

target

in

three-dimensional

space

which

in

turn

leads

to

simplified

manipulation

strategies

The

use

of

just-in-time

representation

Deictic

representations

allow

the

brain

to

leave

important

information

out

in

the

world

and

acquire

it

just

before

it

is

needed

in

the

cognitive

program

This

avoids

the

carrying

cost

of

the

information

The

simplification

of

cognitive

programs

A

way

of

understanding

programs

is

in

terms

of

the

number

of

variables

needed

to

describe

the

computation

at

any

instant

Deictic

variables

or

markers

provide

a

way

of

understanding

this

cost

accounting

Identifying

working

memory

items

with

markers

shows

why

temporary

memory

should

be

minimized

It

simplifies

the

credit

assignment

problem

in

cognitive

programs

as

described

in

Section

-LSB-

Whitehead

and

Ballard

McCallum

Pook

and

Ballard

-RSB-

The

simplification

of

sensory-motor

routines

The

use

of

deictic

codes

leads

to

functional

models

of

vision

wherein

the

representational

products

are

only

computed

if

they

are

vital

to

the

current

cognitive

program

It

is

always

a

good

idea

to

give

the

brain

less

to

do

and

functional

models

show

that

much

of

the

products

of

the

sensorium

that

we

might

have

thought

were

necessary

can

be

done

without

Deictic

codes

lead

us

to

a

different

way

of

thinking

about

working

memory

and

attention

Traditionally

cognitive

operations

have

been

thought

of

as

being

fundamentally

constrained

by

some

kind

of

capacity

limits

on

processing

-LRB-

see

e.g.

-LSB-

Just

and

Carpenter

Norman

and

Bobrow

-RSB-

-RRB-

Similarly

attention

has

been

viewed

as

some

kind

of

limited

mental

resource

that

constrains

cognitive

processing

However

as

-LSB-

Allport

-RSB-

has

pointed

out

viewing

attention

as

a

limited

resource

may

be

little

more

than

a

redescription

of

the

phenomena

and

does

not

explain

why

the

limitations

exist

Instead

viewing

attention

as

a

marker

gives

its

selective

nature

a

computational

rationale

In

considering

attentional

limitations

or

selectivity

Allport

argues

that

some

kind

of

selectivity

is

essential

for

coordinated

perceptuo-motor

action

-LRB-

Thus

an

eye

movement

requires

some

kind

of

visual

search

operation

to

select

a

saccade

target

-RRB-

This

is

very

compatible

with

the

ideas

developed

here

in

which

we

view

attention

as

a

marker

that

is

manipulated

by

current

task

goals

This

view

explains

the

paradox

that

pre-attentive

visual

search

apparently

operates

on

information

which

has

undergone

some

kind

of

segmentation

thought

to

require

attention

This

makes

sense

if

we

think

of

selective

attention

as

being

the

extraction

of

the

information

relevant

for

the

current

task

and

this

may

be

a

low-level

feature

or

a

high-level

semantically-defined

target

Evidence

by

-LSB-

Johnston

and

Dark

-RSB-

that

selective

attention

can

be

guided

by

active

schemata

-LRB-

of

any

kind

-RRB-

is

consistent

with

this

Similarly

capacity

limits

in

working

memory

can

now

be

seen

not

as

a

constraint

on

processing

but

as

an

inevitable

consequence

of

an

efficient

system

that

uses

deictic

variables

to

preserve

only

the

products

of

the

brain

's

computations

that

are

necessary

for

the

ongoing

task

On

this

view

interference

in

dual

task

experiments

will

depend

on

the

extent

to

which

the

different

tasks

compete

for

the

particular

brain

circuitry

required

for

task

completion

This

conception

of

working

memory

as

the

currently

active

marked

variables

also

leads

to

a

very

simple

interpretation

of

the

tradeoffs

between

working

memory

load

and

eye

movements

in

which

fixation

can

be

seen

as

a

choice

of

an

external

rather

than

an

internal

marker

An

interpretation

of

brain

computations

in

terms

of

binding

variables

in

behavioral

programs

blurs

the

distinction

between

perception

and

cognition

which

have

traditionally

been

thought

of

as

different

domains

It

challenges

the

idea

that

the

visual

system

constructs

a

three-dimensional

model

of

the

scene

containing

its

parts

as

components

with

detailed

location

and

shape

information

for

each

of

the

parts

and

that

the

products

of

the

perceptual

computation

are

then

delivered

up

to

cognitive

mechanisms

for

further

processing

A

description

and

critique

of

this

view

has

been

presented

by

-LSB-

Churchland

et

al.

-RSB-

The

idea

of

an

elaborate

scene

model

is

perhaps

clearest

in

the

computer

vision

literature

where

until

recently

the

goal

of

the

models

has

been

primarily

one

of

reconstruction

-LSB-

Marr

Brady

-RSB-

The

emergence

of

an

alternative

approach

-LRB-

called

active

or

animate

vision

-LSB-

Brooks

Brooks

Agre

and

Chapman

Ballard

-RSB-

-RRB-

that

takes

advantage

of

observer

actions

to

minimize

representations

forms

the

foundation

for

the

ideas

presented

here

There

is

more

ambiguity

about

the

way

perceptual

representations

in

humans

are

conceived

On

the

one

hand

the

difference

between

processing

of

attended

and

unattended

information

is

clearly

acknowledged

and

the

limitations

set

by

working

memory

are

recognized

as

fundamental

On

the

other

hand

it

is

often

implicitly

assumed

that

the

function

of

perception

is

to

construct

a

representation

of

the

arrangement

and

identities

of

objects

in

a

scene

We

are

inclined

to

think

of

perception

as

fundamentally

parallel

and

cognition

as

fundamentally

serial

However

the

intimate

relation

between

fixations

and

the

serialized

acquisition

of

information

required

for

task

completion

presents

a

challenge

for

our

understanding

of

the

nature

of

perceptual

experience

In

the

block

copying

task

described

in

Section

manipulations

on

a

given

block

are

largely

independent

of

the

information

acquired

in

previous

views

This

suggests

that

it

is

unnecessary

to

construct

an

elaborate

scene

description

to

perform

the

task

and

that

there

is

only

minimal

processing

of

unattended

information

In

addition

since

color

and

location

information

appear

to

be

acquired

separately

it

appears

that

even

in

the

attended

regions

the

perceptual

representation

may

be

quite

minimal

Thus

human

vision

may

indeed

reflect

the

computational

economies

allowed

by

deictic

representations

and

may

only

create

perceptual

descriptions

that

are

relevant

to

the

current

task

A

similar

suggestion

to

this

has

also

been

made

by

-LSB-

Nakayama

-RSB-

and

by

O'Regan

-LSB-

O'Regan

and

L?evy

Schoen

O'Regan

-RSB-

O'Regan

suggested

that

only

minimal

information

about

a

scene

is

represented

at

any

given

time

and

that

the

scene

can

be

used

as

a

kind

of

external

memory

and

this

is

indeed

what

our

observers

appeared

to

do

The

issue

of

the

complexity

of

the

visual

representation

is

often

confused

with

the

issue

of

whether

visual

information

is

integrated

across

different

eye

positions

An

internal

scene

representation

is

usually

assumed

to

reflect

information

acquired

from

several

fixations

so

evidence

that

visual

information

can

be

integrated

across

eye

movements

has

been

seen

as

important

evidence

for

this

kind

of

view

However

the

issues

are

really

separate

Humans

can

clearly

integrate

visual

information

across

eye

movements

when

they

are

required

to

do

so

-LSB-

Hayhoe

et

al.

Hayhoe

et

al.

-RSB-

and

some

ability

to

relate

information

across

time

and

space

seems

necessary

for

coordinated

action

It

seems

most

natural

to

think

of

visual

computations

as

extracting

information

in

any

of

a

range

of

reference

frames

as

required

by

the

task

independently

of

the

particular

eye

position

At

the

same

time

a

number

of

studies

reveal

that

the

complexity

of

the

representations

that

can

be

maintained

across

several

eye

positions

is

limited

-LSB-

Irwin

Irwin

et

al.

Lachter

and

Hayhoe

Grimes

and

McConkie

-RSB-

Thus

we

would

expect

that

the

recent

findings

of

-LSB-

Duhamel

et

al.

-RSB-

that

neurons

in

macaque

parietal

cortex

will

fire

in

anticipation

of

a

stimulus

reaching

their

receptive

fields

at

the

end

of

a

saccade

would

be

dependent

on

the

nature

of

the

task

and

the

monkey

's

behavioral

goals

One

way

of

reconciling

the

fragmentary

nature

of

the

representations

with

the

richness

of

our

perceptual

experience

would

be

to

suppose

that

our

everyday

experience

of

the

world

reflects

events

over

a

longer

time

scale

than

those

revealed

by

fixations

and

also

to

acknowledge

that

operations

such

as

the

identification

operation

in

Table

can

take

place

on

patterns

across

the

entire

visual

field

The

ability

to

reach

for

objects

that

are

out

of

view

is

often

cited

as

evidence

that

people

do

use

A

B

Figure

The

fact

that

subjects

can

reach

or

look

to

targets

outside

their

immediate

field

of

view

does

not

necessarily

imply

complete

spatial

representations

complex

three-dimensional

models

but

the

embodiment

structure

of

spatial

working

memory

allows

an

alternate

explanation

The

key

is

to

separate

the

ability

to

use

three-dimensional

information

which

people

obviously

can

do

from

the

need

to

build

elaborate

temporary

models

which

is

extremely

difficult

if

not

impossible

Figure

shows

the

deictic

explanation

of

the

reaching

example

A

subject

places

an

object

into

spatial

working

memory

at

a

certain

point

-LRB-

A

-RRB-

perhaps

by

marking

it

with

fixation

Then

at

a

later

time

while

fixating

a

new

object

the

original

object

can

be

grasped

using

any

of

a

variety

of

coordinate

systems

-LRB-

cf.

-LSB-

Hayhoe

et

al.

-RSB-

-RRB-

However

the

crucial

constraint

from

our

point

of

view

is

that

the

object

must

have

been

inventoried

into

spatial

working

memory

To

be

represented

the

object

has

to

use

a

marker

from

the

spatial

working

memory

budget

It

is

not

so

much

that

the

brain

can

not

compute

in

body-centered

frames

but

rather

that

the

main

focus

is

in

deictic

reference

frames

For

example

in

the

use

of

pointing

in

teleassistance

-LRB-

Section

-RRB-

the

robot

arm

must

support

the

movement

of

the

hand

but

this

role

is

secondary

to

the

control

function

which

is

described

in

hand-centered

coordinates

Thus

we

arrive

at

a

conception

of

the

computations

performed

by

the

brain

in

which

disparate

areas

of

the

brain

are

involved

in

the

computation

of

currently

relevant

sensory-motor

primitives

The

functional

model

that

has

been

proposed

must

be

reconciled

with

the

extensive

neural

circuitry

of

visual

cortex

which

has

traditionally

been

investigated

as

a

bottom-up

stimulus-driven

system

The

suggestion

here

is

that

even

the

activity

of

areas

like

may

be

context-dependent

Instead

both

striate

and

extra-striate

visual

areas

may

in

fact

be

computing

highly

specific

task-relevant

representations

Dramatic

evidence

for

this

kind

of

a

view

of

cortex

is

provided

by

experiments

by

-LSB-

Maunsell

-RSB-

In

recordings

from

parietal

cortex

of

the

monkey

they

found

that

cells

sensitive

to

motion

would

respond

differently

to

identical

visual

stimuli

depending

on

the

experimental

contingencies

and

whether

or

not

the

monkey

anticipated

that

the

stimulus

would

move

The

case

that

the

brain

uses

external

referents

as

implied

by

a

deictic

system

is

bolstered

by

observations

that

the

neurons

in

visual

cortex

have

logical

zero

measures

such

as

zero

disparity

and

zero

optic

flow

These

relative

measures

are

properties

of

the

exocentric

fixation

point

Andersen

and

Snyder

-LSB-

Stricanne

et

al.

Snyder

and

Andersen

-RSB-

have

also

shown

that

parietal

cortex

contains

neurons

sensitive

to

exocentric

location

in

space

Additional

evidence

comes

from

motor

cortex

recordings

which

show

the

use

of

exocentric

frames

-LSB-

Tagaris

et

al.

Pellizzer

et

al.

-RSB-

and

from

reaching

experiments

which

show

task-relevant

frames

-LSB-

Tillary

et

al.

-RSB-

Compelling

evidence

that

disparate

areas

of

the

brain

are

involved

in

perceptual

events

is

given

by

experiments

by

-LSB-

Tanenhaus

et

al.

-RSB-

who

showed

that

spoken

language

comprehension

is

controlled

momentby-moment

by

the

visual

stimuli

to

which

the

utterance

refers

We

can

reconcile

the

evidence

that

the

cortex

responds

in

a

bottom-up

manner

with

the

functional

view

described

here

by

considering

the

time

scale

on

which

the

relevant

computations

are

performed

The

same

brain

circuitry

may

be

seen

as

doing

very

different

things

at

different

times

At

the

time

scale

of

most

perceptual

events

that

is

approximately

ms

physical

constraints

of

real-world

invariants

help

organize

the

incoming

stimulus

However

at

longer

time

scales

of

ms

very

different

mechanisms

come

into

play

For

example

we

argue

that

the

same

cortical

areas

participate

in

the

computation

of

the

target

of

an

eye

movement

Infero-temporal

cortex

supplies

the

referent

in

terms

of

features

and

retinotopic

cortex

is

the

source

of

the

localized

features

of

potential

matches

The

coordination

of

the

matching

process

must

be

done

in

a

timely

manner

and

we

posit

that

this

is

mediated

via

extra-cortical

signals

from

the

basal

ganglia

In

summary

we

have

presented

a

model

at

a

level

of

abstraction

that

accounts

for

the

body

's

pointing

mechanisms

This

is

because

traditional

models

that

have

been

described

at

very

short

or

very

long

time

scales

have

proved

insufficient

to

capture

important

features

of

behavior

The

embodiment

model

utilizes

large

parts

of

the

brain

in

a

functional

manner

that

at

first

might

seem

at

odds

with

the

seamlessness

of

cognition

but

can

be

resolved

by

understanding

cognitive

awareness

and

embodiment

models

to

operate

at

different

levels

of

abstraction

and

therefore

different

time

scales

One

way

to

understand

this

is

to

use

conventional

computers

as

a

metaphor

Consider

the

way

virtual

memory

works

on

a

conventional

computer

workstation

Virtual

memory

allows

the

applications

programmer

to

write

programs

that

are

larger

than

the

physical

memory

of

the

machine

What

happens

is

that

prior

to

the

running

of

the

program

it

is

broken

up

into

smaller

pages

and

then

at

run

time

the

requisite

pages

are

brought

into

the

memory

from

peripheral

storage

as

required

This

strategy

works

largely

because

conventional

sequential

programs

are

designed

to

be

executed

sequentially

and

the

information

required

to

interpret

an

instruction

is

usually

very

localized

Consider

now

two

very

different

viewpoints

From

the

application

programmer

's

viewpoint

it

appears

that

a

program

of

unlimited

length

can

be

written

But

the

system

programmer

's

viewpoint

is

very

different

predominantly

because

it

accounts

for

functionality

at

much

shorter

time

scales

The

environment

is

very

dynamic

as

individual

pages

are

moved

in

and

out

of

physical

memory

It

is

just

this

difference

that

captures

the

difference

between

the

level

of

cognitive

awareness

and

the

level

we

are

terming

embodiment

Thus

our

perceptual

and

cognitive

awareness

which

neglects

details

of

the

brain

's

machinery

will

not

necessarily

reflect

events

on

shorter

time

scales

that

occur

at

the

embodiment

level

where

the

human

's

real-time

sensory-motor

system

has

many

creative

ways

to

interact

with

the

world

in

a

timely

manner

Acknowledgments

This

research

was

generously

supported

by

the

National

Institutes

of

Health

under

Grants

and

and

by

the

National

Science

Foundation

under

Grants

and

References

-LSB-

Agre

and

Chapman

-RSB-

Agre

P.

and

Chapman

D.

-LRB-

-RRB-

Pengi

An

implementation

of

a

theory

of

activity

In

Proc.

pages

-LCB-

Seattle

WA

-LSB-

Allport

-RSB-

Allport

A.

-LRB-

-RRB-

Visual

attention

In

Posner

M.

editor

Foundations

of

Cognitive

Science

pages

-LCB-

MIT

Press

Cambridge

MA

-LSB-

Arbib

-RSB-

Arbib

M.

-LRB-

-RRB-

Perceptual

structures

and

distributed

motor

control

In

Handbook

of

Physiology

The

Nervous

System

II

Motor

Control

pages

-LCB-

American

Physiological

Society

Bethesda

MD.

-LSB-

Arbib

et

al.

-RSB-

Arbib

M.

Iberall

T.

and

Lyons

D.

-LRB-

-RRB-

Coordinated

control

programs

for

movements

of

the

hand

In

Hand

Function

and

the

Neocortex

pages

-LCB-

Springer-Verlag

Berlin

-LSB-

Baddeley

-RSB-

Baddeley

A.

-LRB-

-RRB-

Working

Memory

Oxford

Clarendon

Press

-LSB-

Baddeley

-RSB-

Baddeley

A.

-LRB-

-RRB-

Working

memory

The

interface

between

memory

and

cognition

Cognitive

Neuroscience

-LRB-

-RRB-

-LCB-

-LSB-

Baker

and

Gollub

-RSB-

Baker

G.

and

Gollub

J.

-LRB-

-RRB-

Chaotic

Dynamics

An

Introduction

Cambridge

University

Press

New

York

NY

-LSB-

Ballard

-RSB-

Ballard

D.

-LRB-

-RRB-

Cortical

connections

and

parallel

processing

Structure

and

function

Behavioral

and

Brain

Sciences

-LRB-

-RRB-

-LCB-

-LSB-

Ballard

-RSB-

Ballard

D.

-LRB-

-RRB-

Animate

vision

Artificial

Intelligence

Journal

-LCB-

-LSB-

Barlow

-RSB-

Barlow

H.

-LRB-

-RRB-

Single

units

and

cognition

A

neurone

doctrine

for

perceptual

psychology

Perception

-LCB-

-LSB-

Barto

et

al.

-RSB-

Barto

A.

Sutton

R.

and

Watkins

C.

-LRB-

-RRB-

Sequential

decision

problems

and

neural

networks

In

Touretzky

D.

editor

Advances

in

Neural

Information

Processing

Systems

Morgan

Kaufmann

San

Mateo

CA

-LSB-

Bernstein

-RSB-

Bernstein

N.

-LRB-

-RRB-

The

Coordination

and

Regulation

of

Movement

London

Pergamon

Press

-LSB-

Brady

-RSB-

Brady

J.

-LRB-

-RRB-

Preface

the

changing

shape

of

computer

vision

Artificial

Intelligence

-LRB-

-LCB-

-RRB-

-LCB-

-LSB-

Broadbent

-RSB-

Broadbent

D.

-LRB-

-RRB-

Perception

and

Communication

Oxford

University

Press

Oxford

-LSB-

Brooks

-RSB-

Brooks

R.

-LRB-

-RRB-

A

robust

layered

control

system

for

a

mobile

robot

IEEE

J.

Robotics

and

Automation

-LCB-

-LSB-

Brooks

-RSB-

Brooks

R.

-LRB-

-RRB-

Intelligence

without

reason

Technical

Report

AI

Memo

Massachusetts

Inst

of

Technology

-LSB-

Buhmann

et

al.

-RSB-

Buhmann

J.

M.

Lades

M.

and

von

der

Malsburg

C.

-LRB-

-RRB-

Size

and

distortion

invariant

object

recognition

by

hierarchical

graph

matching

In

Proc.

IEEE

IJCNN

-LRB-

Vol

II

-RRB-

pages

-LCB-

San

Diego

CA

-LSB-

Chase

and

Simon

-RSB-

Chase

W.

and

Simon

H.

-LRB-

-RRB-

Perception

in

chess

Cognitive

Psychology

-LCB-

-LSB-

Churchland

et

al.

-RSB-

Churchland

P.

Ramachandran

V.

and

Sejnowski

T.

-LRB-

-RRB-

A

critique

of

pure

vision

In

Koch

C.

and

Davis

J.

editors

Large-Scale

Neuronal

Theories

of

the

Brain

pages

-LCB-

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Crisman

and

Cleary

-RSB-

Crisman

J.

and

Cleary

M.

-LRB-

-RRB-

Deictic

primitives

for

general

purpose

navigation

Proc.

AIAA

Conf

on

Intelligent

Robots

in

Factory

Field

Space

and

Service

-LRB-

CIRFFSS

-RRB-

-LSB-

Duhamel

et

al.

-RSB-

Duhamel

J.

Colby

C.

and

Goldberg

M.

-LRB-

-RRB-

The

updating

of

the

representation

of

visual

space

in

parietal

cortex

by

intended

eye

movements

Science

-LCB-

-LSB-

Fuster

-RSB-

Fuster

J.

-LRB-

-RRB-

The

Prefrontal

Cortex

Anatomy

Physiology

and

Neuropsychology

of

the

Frontal

Lobe

Raven

Press

New

York

second

edition

-LSB-

Fuster

-RSB-

Fuster

J.

-LRB-

-RRB-

Memory

in

the

Cerebral

Cortex

An

Empirical

Approach

to

Neural

Networks

in

the

Human

and

Nonhuman

Primate

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Gabrieli

-RSB-

Gabrieli

J.

-LRB-

-RRB-

Contribution

of

the

basal

ganglia

to

skill

learning

and

working

memory

in

humans

In

Houk

J.

Davis

J.

and

Beiser

D.

editors

Models

in

Information

Processing

in

the

Basal

Ganglia

pages

-LCB-

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Goldberg

-RSB-

Goldberg

D.

E.

-LRB-

-RRB-

Genetic

Algorithms

in

Search

Optimization

and

Machine

Learning

Addison-Wesley

-LSB-

Goldman-Rakic

-RSB-

Goldman-Rakic

P.

-LRB-

-RRB-

Toward

a

circuit

model

of

working

memory

and

the

guidance

of

voluntary

motor

action

In

Houk

J.

Davis

J.

and

Beiser

D.

editors

Models

in

Information

Processing

in

the

Basal

Ganglia

pages

-LCB-

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Goodale

and

Milner

-RSB-

Goodale

M.

and

Milner

A.

-LRB-

-RRB-

Separate

visual

pathways

for

perception

and

action

Trends

in

Neurosciences

-LCB-

-LSB-

Graybiel

and

Kimura

-RSB-

Graybiel

A.

and

Kimura

M.

-LRB-

-RRB-

Adaptive

neural

networks

in

the

basal

ganglia

In

Houk

J.

Davis

J.

and

Beiser

D.

editors

Models

in

Information

Processing

in

the

Basal

Ganglia

pages

-LCB-

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Grimes

and

McConkie

-RSB-

Grimes

J.

and

McConkie

G.

-LRB-

to

appear

-RRB-

On

the

insensitivity

of

the

human

visual

system

to

image

changes

made

during

saccades

In

Akins

K.

editor

Problems

in

Perception

Oxford

University

Press

Oxford

UK

-LSB-

Hancock

et

al.

-RSB-

Hancock

P.

Baddeley

R.

and

Smith

L.

-LRB-

-RRB-

The

principal

components

of

natural

images

Network

-LCB-

-LSB-

Hayhoe

et

al.

-RSB-

Hayhoe

M.

Lachter

J.

and

Feldman

J.

-LRB-

-RRB-

Integration

of

form

across

saccadic

eye

movements

Perception

-LCB-

-LSB-

Hayhoe

et

al.

-RSB-

Hayhoe

M.

Lachter

J.

and

Moller

P.

-LRB-

-RRB-

Spatial

memory

and

integration

across

saccadic

eye

movements

In

Rayner

K.

editor

Eye

Movements

and

Visual

Cognition

Scene

Perception

and

Reading

pages

-LCB-

Springer-Verlag

New

York

-LSB-

Hertz

et

al.

-RSB-

Hertz

J.

Krogh

A.

and

Palmer

R.

-LRB-

-RRB-

Introduction

to

the

Theory

of

Neural

Computation

volume

of

Santa

Fe

Institute

Studies

in

the

Sciences

of

Complexity

Lecture

Notes

Addison-Wesley

-LSB-

Irwin

-RSB-

Irwin

D.

-LRB-

-RRB-

Information

integration

across

saccadic

eye

movements

Cognitive

Psychology

-LCB-

-LSB-

Irwin

et

al.

-RSB-

Irwin

D.

Zacks

J.

and

Brown

J.

-LRB-

-RRB-

Visual

memory

and

the

perception

of

a

stable

visual

environment

Perception

and

Psychophysics

-LCB-

-LSB-

Johnston

and

Dark

-RSB-

Johnston

W.

and

Dark

V.

-LRB-

-RRB-

Selective

attention

Annual

Review

of

Psychology

-LCB-

-LSB-

Jones

and

Malik

-RSB-

Jones

D.

G.

and

Malik

J.

-LRB-

-RRB-

A

computational

framework

for

determining

stereo

correspondence

from

a

set

of

linear

spatial

filters

In

Proc.

European

Conf

on

Computer

Vision

-LSB-

Just

and

Carpenter

-RSB-

Just

M.

and

Carpenter

P.

-LRB-

-RRB-

Eye

fixations

and

cognitive

processes

Cognitive

Psychology

-LCB-

-LSB-

Just

and

Carpenter

-RSB-

Just

M.

and

Carpenter

P.

-LRB-

-RRB-

A

capacity

theory

of

comprehension

Individual

differences

in

working

memory

Psychological

Review

-LRB-

-RRB-

-LCB-

-LSB-

Koch

and

Crick

-RSB-

Koch

C.

and

Crick

F.

-LRB-

-RRB-

Some

further

ideas

regarding

the

neuronal

basis

of

awareness

In

Koch

C.

and

Davis

J.

editors

Large-Scale

Neuronal

Theories

of

the

Brain

pages

-LCB-

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Kosslyn

-RSB-

Kosslyn

S.

-LRB-

-RRB-

Image

and

Brain

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Kowler

and

Anton

-RSB-

Kowler

E.

and

Anton

S.

-LRB-

-RRB-

Reading

twisted

text

Implications

for

the

role

of

saccades

Vision

Research

-LCB-

-LSB-

Koza

-RSB-

Koza

J.

R.

-LRB-

-RRB-

Genetic

Programming

On

the

Programming

of

Computers

by

Means

of

Natural

Selection

MIT

Press

Cambridge

MA

-LSB-

Lachter

and

Hayhoe

-RSB-

Lachter

J.

and

Hayhoe

M.

-LRB-

submitted

-RRB-

Capacity

limits

in

integrating

information

across

saccades

Perception

and

Psychophysics

-LSB-

Luria

-RSB-

Luria

A.

-LRB-

-RRB-

The

Mind

of

a

Mnemonist

A

Little

Book

about

a

Vast

Memory

Harvard

University

Press

Cambridge

MA

-LSB-

Marr

-RSB-

Marr

D.

-LRB-

-RRB-

Vision

W.H.

Freeman

and

Co.

Oxford

-LSB-

Maunsell

-RSB-

Maunsell

J.

-LRB-

-RRB-

presentation

Woods

Hole

Workshop

on

Computational

Neuroscience

-LSB-

McCallum

-RSB-

McCallum

R.

-LRB-

-RRB-

Reduced

training

time

for

reinforcement

learning

with

hidden

state

In

Proc.

Int

l.

Machine

Learning

Workshop

-LRB-

Robot

Learning

-RRB-

New

Brunswick

NJ

-LSB-

Mel

-RSB-

Mel

B.

W.

-LRB-

-RRB-

Object

classification

with

high-dimensional

vectors

Presentation

Telluride

Workshop

on

Neuromorphic

Engineering

-LSB-

Miller

-RSB-

Miller

G.

-LRB-

-RRB-

The

magic

number

seven

plus

or

minus

two

Some

limits

on

your

capacity

for

processing

information

Psychological

Review

-LCB-

-LSB-

Miyachi

et

al.

-RSB-

Miyachi

S.

Miyashita

K.

Karadi

Z.

and

Hikosaka

O.

-LRB-

-RRB-

Effects

of

the

blockade

of

monkey

basal

ganglia

on

the

procedural

learning

and

memory

In

Abstracts

Annual

Meeting

Society

for

Neuroscience

page

-LRB-

-RRB-

Miami

Beach

FL

-LSB-

Nakayama

-RSB-

Nakayama

K.

-LRB-

-RRB-

The

iconic

bottleneck

and

the

tenuous

link

between

early

visual

processing

and

perception

In

Blakemore

C.

editor

Vision

Coding

and

Efficiency

pages

-LCB-

Cambridge

University

Press

-LSB-

Newell

-RSB-

Newell

A.

-LRB-

-RRB-

Unified

Theories

of

Cognition

Harvard

University

Press

Cambridge

MA

-LSB-

Norman

and

Bobrow

-RSB-

Norman

D.

and

Bobrow

D.

-LRB-

-RRB-

On

data-limited

and

resourcelimited

processes

Cognitive

Psychology

-LCB-

-LSB-

Noton

and

Stark

-RSB-

Noton

D.

and

Stark

L.

-LRB-

-RRB-

Scanpaths

in

saccadic

eye

movements

while

viewing

and

recognizing

patterns

Vision

Research

-LSB-

O'Regan

-RSB-

O'Regan

J.

-LRB-

-RRB-

Solving

the

real

mysteries

of

visual

perception

The

world

as

an

outside

memory

Canadian

J.

Psychology

-LCB-

-LSB-

O'Regan

and

L?evy

Schoen

-RSB-

O'Regan

J.

and

L?evy

Schoen

A.

-LRB-

-RRB-

Integrating

visual

information

from

successive

fixations

Does

trans-saccadic

fusion

exist

Vision

Research

-LCB-

-LSB-

Pellizzer

et

al.

-RSB-

Pellizzer

G.

Sargent

P.

and

Georgopoulos

A.

-LRB-

-RRB-

Motor

cortex

and

visuomotor

memory

scanning

In

Abstracts

Annual

Meeting

Society

for

Neuroscience

page

-LRB-

-RRB-

Miami

Beach

FL

-LSB-

Pelz

et

al.

-RSB-

Pelz

J.

Hayhoe

M.

Ballard

D.

and

Forsberg

A.

-LRB-

-RRB-

Separate

motor

commands

for

eye

and

head

Investigative

Ophthalmology

and

Visual

Science

Supplement

-LSB-

Pollatsek

and

Rayner

-RSB-

Pollatsek

A.

and

Rayner

K.

-LRB-

-RRB-

Eye

movements

and

lexical

access

in

reading

In

Balota

D.

d'Arcais

G.

F.

and

Rayner

K.

editors

Comprehension

Processes

in

Reading

pages

-LCB-

Lawrence

Erlbaum

Associates

Publisher

Hillsdale

NJ

-LSB-

Pook

and

Ballard

-RSB-

Pook

P.

and

Ballard

D.

-LRB-

-RRB-

Deictic

teleassistance

In

Proc.

IEEE/RSJ/GI

Int

l.

Conf

on

Intelligent

Robots

and

Systems

Munich

Germany

-LSB-

Pook

and

Ballard

-RSB-

Pook

P.

and

Ballard

D.

-LRB-

-RRB-

Teleassistance

Contextual

guidance

for

autonomous

manipulation

In

Proc.

Nat'l

Conf

on

Artificial

Intelligence

-LRB-

AAAI

-RRB-

Seattle

WA

-LSB-

Rao

and

Ballard

-RSB-

Rao

R.

and

Ballard

D.

-LRB-

-RRB-

Iconic

representations

for

active

vision

Technical

Report

Department

of

Computer

Science

U.

Rochester

Submitted

for

journal

publication

-LSB-

Schlingensiepen

et

al.

-RSB-

Schlingensiepen

K.-H.

Campbell

F.

Legge

G.

and

Walker

T.

-LRB-

-RRB-

The

importance

of

eye

movements

in

the

analysis

of

simple

patterns

Vision

Research

-LCB-

-LSB-

Schultz

et

al.

-RSB-

Schultz

W.

Apicella

P.

Romo

R.

and

Scarnati

E.

-LRB-

-RRB-

Contextdependent

activity

in

primate

striatum

reflecting

past

and

future

behavioral

events

In

Houk

J.

Davis

J.

and

Beiser

D.

editors

Models

of

Information

Processing

in

the

Basal

Ganglia

pages

-LCB-

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Shastri

-RSB-

Shastri

L.

-LRB-

-RRB-

From

simple

associations

to

systematic

reasoning

Behavioral

and

Brain

Sciences

-LRB-

-RRB-

-LCB-

-LSB-

Simon

-RSB-

Simon

H.

-LRB-

-RRB-

The

architecture

of

complexity

In

Proc.

American

Philosophical

Society

Vol

pages

-LCB-

-LSB-

Snyder

and

Andersen

-RSB-

Snyder

L.

and

Andersen

R.

-LRB-

-RRB-

Effects

of

vestibular

and

neck

proprioceptive

signals

on

visual

responses

in

posterior

parietal

cortex

In

Abstracts

Annual

Meeting

Society

for

Neuroscience

page

-LRB-

-RRB-

Miami

Beach

FL

-LSB-

Stefanacci

et

al.

-RSB-

Stefanacci

L.

Suzuki

W.

and

Amaral

D.

-LRB-

-RRB-

Organization

of

reciprocal

projections

between

the

monkey

amygdala

and

the

perirhinal

-LRB-

areas

and

-RRB-

and

parahippocampal

-LRB-

areas

TF

and

TH

-RRB-

cortices

In

Abstracts

Annual

Meeting

Society

for

Neuroscience

page

-LRB-

-RRB-

Miami

Beach

FL

-LSB-

Stricanne

et

al.

-RSB-

Stricanne

B.

Xing

J.

Mazzoni

P.

and

Andersen

R.

-LRB-

-RRB-

Response

of

lip

neurons

to

auditory

targets

for

saccadic

eye

movements

A

distributed

coding

for

sensorimotor

transformation

In

Abstracts

Annual

Meeting

Society

for

Neuroscience

page

-LRB-

-RRB-

Miami

Beach

FL

-LSB-

Strick

et

al.

-RSB-

Strick

P.

Dum

R.

and

Picard

N.

-LRB-

-RRB-

Macro-organization

of

the

circuits

connecting

the

basal

ganglia

with

the

cortical

motor

areas

In

Houk

J.

Davis

J.

and

Beiser

D.

editors

Models

in

Information

Processing

in

the

Basal

Ganglia

pages

-LCB-

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Swain

and

Ballard

-RSB-

Swain

M.

and

Ballard

D.

-LRB-

-RRB-

Color

indexing

Int

l.

J.

Computer

Vision

-LRB-

-RRB-

-LCB-

-LSB-

Tagaris

et

al.

-RSB-

Tagaris

G.

Kim

S.-G.

Menon

R.

Strupp

J.

Andersen

P.

Ugurbil

K.

and

Georgopoulos

A.

-LRB-

-RRB-

High

field

-LRB-

Telsa

-RRB-

functional

MRI

of

mental

rotation

In

Abstracts

Annual

Meeting

Society

for

Neuroscience

page

-LRB-

-RRB-

Miami

Beach

FL

-LSB-

Tanenhaus

et

al.

-RSB-

Tanenhaus

M.

Spivey-Knowlton

M.

Eberhard

K.

and

Sedivy

J.

-LRB-

-LRB-

in

press

-RRB-

-RRB-

Using

eye

movements

to

study

spoken

language

comprehension

Evidence

for

visually-mediated

incremental

interpretation

In

Inoui

T.

and

McClelland

J.

editors

Attention

and

Performance

XVI

Integration

in

Perception

and

Communication

MIT

Press

Cambridge

MA

-LSB-

Tillary

et

al.

-RSB-

Tillary

S.

H.

Flanders

M.

and

Soechting

J.

-LRB-

-RRB-

A

coordinate

system

for

the

synthesis

of

visual

and

kinesthetic

information

J.

Neuroscience

-LRB-

-RRB-

-LCB-

-LSB-

Ullman

-RSB-

Ullman

S.

-LRB-

-RRB-

Visual

routines

Cognition

-LCB-

Also

in

S.

Pinker

-LRB-

Ed

-RRB-

Visual

Cognition

Cambridge

MA

Bradford

Books

-LSB-

Ungerleider

and

Mishkin

-RSB-

Ungerleider

L.

and

Mishkin

M.

-LRB-

-RRB-

Two

cortical

visual

systems

In

Ingle

D.

Goodale

M.

and

Mansfield

R.

editors

Analysis

of

Visual

Behavior

pages

-LCB-

MIT

Press

Cambridge

MA

-LSB-

Van

Essen

et

al.

-RSB-

Van

Essen

D.

Anderson

C.

and

Olshausen

B.

-LRB-

-RRB-

Dynamic

routing

strategies

in

sensory

motor

and

cognitive

processing

In

Koch

C.

and

Davis

J.

editors

LargeScale

Neuronal

Theories

of

the

Brain

pages

-LCB-

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Viviani

-RSB-

Viviani

P.

-LRB-

-RRB-

Eye

movements

in

visual

search

Cognitive

perceptual

and

motor

control

aspects

In

Kowler

E.

editor

Eye

Movements

and

their

Role

in

Visual

and

Cognitive

Processes

Reviews

of

Oculomotor

Research

pages

-LCB-

Elsevier

-LSB-

Whitehead

and

Ballard

-RSB-

Whitehead

S.

and

Ballard

D.

-LRB-

-RRB-

Active

perception

and

reinforcement

learning

Neural

Computation

-LRB-

-RRB-

-LCB-

-LSB-

Whitehead

and

Ballard

-RSB-

Whitehead

S.

and

Ballard

D.

-LRB-

-RRB-

Learning

to

perceive

and

act

by

trial

and

error

Machine

Learning

-LRB-

-RRB-

-LCB-

-LSB-

Woodward

et

al.

-RSB-

Woodward

D.

Kirillov

A.

Myre

C.

and

Sawyer

S.

-LRB-

-RRB-

Neostriatal

circuitry

as

a

scalar

memory

Modeling

and

ensemble

neuron

recording

In

Houk

J.

Davis

J.

and

Beiser

D.

editors

Models

of

Information

Processing

in

the

Basal

Ganglia

MIT

Press

-LRB-

A

Bradford

Book

-RRB-

Cambridge

MA

-LSB-

Yarbus

-RSB-

Yarbus

A.

-LRB-

-RRB-

Eye

Movements

and

Vision

Plenum

New

York

binding
embodiment
eye movements
markers
natural tasks
working memory

brain computation
deictic computations
sensory-motor tasks