Agent Robotics:
Learning-from-observation
Katsu Ikeuchi
Applied Robotics ResearchMicrosoftLearning-from-observation (LfO)
Task model(Minsky’s frame)

What-to-do
Where-to-do

How-to-do

Not direct mimicking
Teaching mode

Execution mode

LfO (indirect mimicking)
Task models
What-to-do
where-to-do

GPT-based
encoder(cerebrum)
Human demonstration

RL-trained
skill agents(Cerebellum)
Robot execution• GPT-based encoder observes human demonstration and encode them as task models• Skill agents corresponding to task models mimics the demonstration using where-to-doPre-requisite for GPT-encoding• GPT knows the collection of skill agents available
Library of manipulation-skill agents
Library of grasp-skill agents
Passive Formskill agent

Passive Forceskill agent

Active Forceskill agent

Pick skill
agent

Drawer-open
skill agent

Door-open
skill agent

Bring skill
agent

Drawer-adjust
skill agent

Door-adjust
skill agent

Place skill
agent

Drawer-close
skill agent

Door-close
skill agent

Closure theory

Kuhn-Tucker theory…

GPT-Encoder (Ver23Sep): verbal + visual
Verbalinstruction“Grasp ared cup”

GPT

What-to-do

Affordance
Analyzer

Where-to-do

Visualdemonstrations

Wake2023chatgpt-IEEE-ACCESSGPT-Encoder (Ver23Sep): verbal + visual
Verbalinstruction“Grasp ared cup”

Short sentence

GPT

What-to-do

Affordance
Analyzer

Where-to-do

Visualdemonstrations

ChatGPT to get what-to-do
ChatGPT can generate a sequence of what to do (step-by-stepaction primitives) from natural language input

What-to-doGPT-Encoder (Ver23Sep): verbal + visual
Verbalinstruction“Grasp ared cup”

GPT

What-to-do

AffordanceAnalyzer

Where-to-do

Visualdemonstrations

Stop & go teachingAffordance analyzer
GPT-output

Retrieved Task models

Move_hand()

Where to move hand

Grasp_object()

From which direction to approach

Move_object()

To where to move the object

what-to-do

where-to-do

PassiveForce
PassiveForce
PassiveForce
Which-handPassiveForce
Which-handWhich-handWhich-object
Which-handWhich-object
Approachdirection
Which-object
Approachdirection
Which-object
Labanotation
Approach direction
Labanotation
Approach direction
Labanotation
Labanotation

Abstract task models(Minsky’s frames)• Where-to-do• Necessary parameters for execution• Defined to each what-to-do

Segmentation
Grounding• Minsky’s frame type design• Deamon to extract from input video

GPT-Encoder (Ver23Nov): VLM + LLM (visual input only)

Visual & (verbal)
demonstrations

GPT

What-to-doAA

Where-to-do

Wake2023gpt-arXiv

Task models without verbal input using GPT-4
Affordance analyzer

GPT4-based task analyzer

Object
names

GPT4V
Video
analyzerVideo
demonstration

Sceneanalyzer

textualinstruction

Environmental
description

Open-vocab
object detector
Grasp/release
video clips

Taskplanner
GPT4

Symbolic
task model

Handloc. FoA

Hand-object
dist. FoA
grasp/releasetime and location
Affordance
extractor

Embodiedtask model

GPT4V

Blue components/lines are text-based information, and the red components are vision-related information• Symbolic task planner: generated what-to-do• Affordance analyzer: instantiate the tasks with affordance (i.e., skill parameters)

Skill-agents retrieved from the library
Verbalinstruction“Grasp ared cup”

GPT

What-to-do

Skill agents

Visualdemonstrations

Affordance Analyzer

Where-to-doSkill agent Library

how-to-doSkill agent to execute primitive actions
Drawer-open (PTG31)
Task model
Axis direction

Where-to-paramfrom demoForce
feedback

Visualfeedback

IK solver
Exec world

Demo world
Hint & Goal

StateStateskill-agent

ActionHand motion

Task models

RL-trained
Skill agents

Nextage

GPTbased
encoder
DemoRL-trained
Skill agents

Fetch

Designing agent libraryLibrary of skill agents• A collection of reusable agents to execute primitive
actions on different robot hardware• Roughly corresponds to “verb” such as pickup or grasp• Necessary and sufficient set to cover the action domain• Grasp library:

given by Closure theory• Manipulation library: given by Kuhn-Tucker Theory

Library ofGrasp-skill
agentsClosure theory

Library of
Manipulation-skill
agentsKuhn-Tucker
theoryManipulation-skill agent

How to represent: face contact relation
Manipulation primitive action causes contact-state transition

Face contact

Verb “Pick”“Pick” breaks the face contact

Ikeuchi1994Assembly-IEEE-TRA

State of face contact
State of contact = movable directions
One directional contact

Multi directional contacts

𝑿 ∙𝑵 ≥𝟎
𝑿: possible movable direction
𝑵: Constraint Normal direction

𝑋𝑁𝑁1
Movable directions =Polygonal area on
Gaussian sphere

Movable directions =Northern hemisphere of Gaussian sphere

𝑿 ∙ 𝑵𝟏 ≥ 𝟎
𝑿 ∙ 𝑵𝟐 ≥ 𝟎⋮
𝑿 ∙ 𝑵𝒏 ≥ 𝟎

Solution of Simultaneous linear
inequality equations

Kuhn-Tucker theory

𝑋 ∙ 𝑁1 ≥ 0

𝑋 ∙ 𝑁2 ≥ 0⋮
𝑋 ∙ 𝑁𝑛 ≥ 0Simultaneous linear inequality
equationsKuhn-Tucker theory
Solution areas of the equations
can be characterized into thefollowing the classes

Possible transitions (possible primitive actions)

7 X 7 =49
49 transitions???
49 primitive actions???

Physically Possible transitions

Physically possible:
20 transitions
20 primitive actions

Some examples
From NC

Bring

NC:

NC:
Often appear in YouTube

Place

insert

NC:

PC:

NC:

TR:

NC:

OT:

NC:

PR:

NC:

OP:

NC:

FT:

Often appear in YouTube

In industrial applications,
Yes, but in service robot
Not often

Some examples
Pick
PC:
from PC

NC:

Wipe PC:

PC:

PC:

TR:

PC:

OT:

PC:

PR:

PC:

OP:

PC:

FT:

Often appear in YouTube

Discuss laterFrequently appeared transitions in YouTube
Bring
Place

Pick

Drawer-adjust
Drawer-openDrawer-close

YouTube appear:
6 transitions
6 tasks

Physical manipulation agents
Translation tasks

Rotation tasks

BringPlacePick

Drawer-adjust
Drawer-open

Drawer-close

6 translation tasks

Door-adjust
Door-open

Door-close

3 rotation tasks

Tool-env common sense:
Common-sense: while wiping, do not detach from thetable surface

Physical & Semantic constraints• Physical constraint: movable
only upper directions due tothe table surface
Physical surface• Semantic constraint: for wiping,
not to detach from the tablesurface
Semantic surface(common-senserepresentation)

Ikeuchi2023semantic-IJRR

Semantic constraints extracted from YouTube cocking video
Semantic Ping

SemanticSphere

Semantic Walls

Semantic Hinge

Semantic Tube

Semantic manipulation agents
Translation tasks

Rotation tasks

Bring-carefully
Pick-carefully

Place-carefully
Bowl

Wipe
Wipe-end

Wipe-start

Bowl-start

Peel

Pour
Peel-end

Peel-start

Bowl-end

Pour-start

Pour-end

Hold

Hold

10 Translation tasks

6 rotation tasks

Agents in the current library• PTG1• Picking• Placing• Bringing• PTG3• DrawerOpening• DrawerClosing• DrawerAdjusting• STG1• Bring-carefully• STG2• Wiping• PTG5• DoorOpening• DoorClosing• Door Adjusting
Takamatsu2023Designing-arXiv

RL training of agents

Maintain/Detach/Constraint dimension

Constraint direction

Pure detachment
direction

Maintenance directions

𝑋 ∙𝑁 ≥0

Maintenance = 2 DoFs
Detachment = 1 DoFs
Constraint = 0 DoFs

(2,1,0)

Maintenancedirections

𝑋 ∙𝑁 ≥0𝑋 ∙ −𝑁 ≥ 0

Maintenance = 2 DoFs
Detachment = 0 DoFs
Constraint = 1 DoFs

(2, 0, 1)

State nameNC

DOFs Admissible translation directions on the Gaussian sphere & Dimensions3

NC (M=3, D=0, C=0)

2.5

PC1(M=2,D=1, C=0)2

TR(M=2, D=0, C=1)

1.5

OT1(M=1, D=1, C=1)

1

PR(M=1, D=0, C=2)

0.5

OP(M=0, D=1, C=2)0

FT(M=0, D=0, C=3)

Non-contact
translation

PCPC2(M=1, D=2, C=0)

Partial contact
translation

TR
Translation contact
translation

OT
One-way translation
contact translation

PR
Prismatic contact
translationOP
One-way prismatic
contact translation

FT
Fully contact
translation

OT2(M=0, D=2, C=1)

PCN (M=0,D=3, C=0)

Dimension transition provides control laws
UBringTS(M, D, C) = (3, 0, 0)

S: motion direction
T: perpendicular to motion
U:Perpendicular to motion(M, D, C) = (3, 0, 0)

Motion direction (S):
Maintenance to Maintenance → Position control
Perpendicular direction (T): Maintenance to Maintenance → Position control
Perpendicular direction (U): Maintenance to Maintenance → Position control

If S = goal-s AND T = goal-t AND U = goal-u,then reward

PlaceTSU(3, 0, 0)(2, 1, 0)Motion direction (S):
Maintenance to Detachment → force control
Perpendicular direction (T): Maintenance to Maintenance → position control
Perpendicular direction (U): Maintenance to Maintenance → position control
If F+s > delta-zero AND T = goal-t AND U = goal-u,then reward

Tinsert

SU(M, D, C) = (3, 0, 0)(M, D, C) = (1, 0, 2)

Motion direction (S):Maintenance to Maintenance → position control
Perpendicular direction (T): Maintenance to Constraint → visual control then force control
Perpendicular direction (U): Maintenance to Constraint → visual control then force control

If Before Transition AND |T – feature-along-t-direction | > delta-gap, then penalty
If Before Transition AND |U – feature-along-u-direction | > delta-gap, then penalty
If AfterTransition AND F-t > delta-collision, then penalty
If AfterTrasnsition AND F-u > delta-collision, then penalty
If S = goal-s, then reward

U
S: motion direction
T: perpendicular to motion
U:Perpendicular to motion

TS

StartEnd

Exampleposition

Force

VisionControl
If S = goal-s AND T = goal-t AND U = goal-u,then reward

Bring NC: 3,0,0 NC: 3,0,0S: (M-M)
T: (M-M)
U: (M-M)

Place NC: 3,0,0 PC: 2,1,0

T: (M-M)
U: (M-M)

NC: 3,0,0 TR: 2,0,1

S: (M-M)
T: (M-M)

U: (M-C)

If BeforTransition AND |U – feature U| > delta-gap, then penalty
If AfterTransition AND F-u > delta-collision, then penalty
If S = goa-s AND T = goal-t, then reward

NC: 3,0,0 OT: 1,1,1

S: (M-M)

T: (M-D)
U: (M-C)

If BeforeTransition AND | T – feature t | > delta-gap, then penalty
If BeforeTransition AND | U – feature u | > delta-gap, then penaly
If AfterTransition AND F-t > delta-collision, then penalty
If AfterTransition AND F-t < delta-zero, then penalty
If After Transition AND F-u > delta-collision, then penalty
If S = goal-s, then reward

insert NC: 3,0,0 PR: 1,0,2

S: (M-M)

T: (M-C)
U: (M-C)

If BeforeTransition AND |T – feature-t | > delta-gap, then penalty
If Before Transition AND |U – feature-u | > delta-gap, then penalty
If AfterTransition AND F-t > delta-collision, then penaltyIf AfterTrasnsition AND F-u > delta-collision, then penalty
If S = goal-s, then reward

If F+s > delta-zero AND T = goal-t AND U = goal-u,then reward

S: (M-D)

RL-trained PTG33 agent (Drawer-close)
zyPROPx

If Force-y > delta-collision, penalty
If Force-z > delta-collision, penalty
If Force-x > delta-zero, reward-250

Force-x

Force-y-200]N[-150-100-50[Time step]
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 230Time
RL learning curve

50

Force along x-axis

TimeRL-trained PTG51 agent (Door opening)
If Force-y > delta-collision, penalty
If Force-z > delta-collision, penalty
If X = Goal-x, reward

Nextage @Shinagawa

OR×3

RVFetch @Redmond×3

RL-trained STG2 agent (wipe)

If Force-y > delta-collision, penalty
If Force-y > delta-zero, penalty
If X = Goal-x, reward

Insert (NC-PR)
Maintenance = 3
Detachment = 0
Constraint = 0

If |T – feature-t | > delta-gap, then penalty
If |U – feature-u | > delta-gap, then penalty
If S = goal-S, then reward

Maintenance = 1
Detachment = 0
Constraint = 2

Grasp-skill agent (Current version)
Current

Grasp types• Grasping depends on the goal of the task sequence

Need power to push

Need control to point

Need power and control
to writeGrasp taxonomy in Robotics community
Felix et alPurpose of task: from taxonomy to closure
Large diameter

Small diameter

Passive force closure

Passive force
contact-web

Passive form closure

Passive form
contact-web

Medium wrap
Power disk
Power sphere

Adducted thumb

Medium wrap

Extension type

Fixed Hook
Prismatic 2-finger

Prismatic 3-finger

Active-force
contact-web

Prismatic 4-finger
Precision sphere

Grasp taxonomy

Active force Closure
Closure theory (Yoshikawa)Contact-web（SeedNoid）

Three grasp agents prepared
RGBD

Pre-trained CNN
for passive-force
contact-web

Pre-trained agent
for passive-force
contact-web

RGBD

Pre-trained CNN
for passive-form
contact-web

Pre-trained agent
for passive-form
contact-web

RGBD

Pre-trained CNN
for active-force
contact-web

Pre-trained agent
for active-force
contact-web

Saito&Sasabuchi2023Constraint-MDPIEach contact-web based agent (end-2-end system)

RGBD

Pre-trained CNN
for active-force-closure
contact-web(Observation sub-agent)
Active-force-closure
Contact-web
Pre-trained sub-agent
for active-force-closure
contact-web(Execution sub-agent)Super quadricquadric

super quadric𝒙 𝟐𝒚 𝟐𝒛 𝟐++=𝟏𝒂𝒃𝒄𝒙𝒂𝜶+𝒚 𝜷𝒃𝒛+
𝒅Size parameter: a, b, c𝜸

=𝟏Shape parameter: 𝜶, 𝜷, 𝜸Shape variations due to parameters: 𝜶, 𝜷, 𝜸Training CNN
Size parameter: a, b, c Shape parameter: 𝜶, 𝜷, 𝜸Super quadricTraining data
Depth image

TrainedCNN

Contact web position

Reinforcement learningHint information- Contact web- Approach direction

State- current finger positions- current contact force direction

Train various objects of the same grasp

learn against different size and shapes

The same agent can grasp various shaped objects

Passive-force agent

Active-force agent

The same object with two different agents

Errant robot projectErrand humanoid
Human order:“Go to Kitchen
and clean up thetable”

ChatGPT
interface inTSS

Skill library inTSS• LfO generated program• Symbolic map

Humanoid at Kitchen

Key components
Teaching• Learning from observation for manipulation• HoloLens based map-making & navigation
Execution• ChatGPT-based program generation• TSS execution platform with skill libraries

Diachronic discussion
Learning-from-observation

Learning-from-observation
Relation-1

Relation-2
ActionHardware independent
essence of the task

Observation

Direct mimicking
does not workPerformance

How to obtain the essence?
Top down approach

Bottom up approach• Design the task models
based on Robotics theories• Learn everything from scratch

Kanade’s principle:
Do not apply learning approaches to those you can solve without themTop-down LfO: starting point

We started this effort 30 years ago at CMU!

Ikeuchi & Reddy CMU-RI 89

Object recognition & Task recognition

Marr 82

Ikeuchi, Suehiro TRO94

Key Idea: Essence = State transition
ABA

Put B side of A

B

BA

Put B on top of A

B
A

State transition

Necessary skills

Domains explore the possible sets of states
Two blocks

1988

Machine parts

1990

Polyhedron

HouseholdDance

2000

2020

2010

RopeGesturePolyhedral world: state = face-contact

Face-contact states

Extracted state transitions &required skillsTakamatsu et.al. “assembly tasks” IJRR2007

Machine parts: States = Parts mating

Polyhedral rep+

Screw-alignGear-alignParts mating

Miura & Ikeuchi Task PAMI98

Knot-tying: Status = P-data in the knot theory
P-data representation

Reidemister move (action primitives)

Execution mode

States = P data in the Knot theory

Takamatsu et. al., “Knot-tying,” TRA 2006

Human dance: State = Key pose & foot contact

Key pose (Labanotation)

Foot contact
Nakaoka et. al. ICRA2003

Synchronic discussion
Learning-from-observation

Imitation learning vs LfO
Imitation learning = mimic all trajectories• It works only when the object and theenvironment are exactly same because
the system mimics all the trajectories
Human demonstration• Error by demonstrator will be mimicked > fatigue of the operator• Pick up a dish<bring trajectory>• Pick up a sponge<bring trajectory>• Wipe<bring trajectory>• Place the dish<bring trajectory>• Place the sponge

Learning from observation• Only mimic where it importantLfO = symbolic teleoperation• Task-encoding based on GPT: what-to-do & where-to-do• Task-decoding based agents: how-to-do

Symbolic representation

Humandemo

Forward pretrained model(GPT-base task encoding)Pretrained skill agents(RL-based task decoding)

Cerebrum (?)

Cerebellum (?)?

Robotexec

Cerebrum vs Cerebellum
Task models

Current design

GPT-based

Agent-based

Cerebrum (?)
What-to-do
Where-to-do

Cerebellum (?)
How-to-do

Action domain
Gesture

GPT

Agent

Navigation

GPT

Agent

Manipulation

GPT

Agent

Locomotion

GPT

Agent
Amount of reaction/disturbancefrom the environment

Defense of only vision (not force)• In children’s imitation, the connection with the mother is limited
to the visual world only
• Visual observation is done through Piaget’s schema; not entire
actions• in LfO, Affordance analyzer using Minsky’s frame (Task model)• Force information is not shared; force feedback is learned
separately through circular reactions(Reinforcement learning (?))

LfO and Piaget’s theory• Sensormotor stage• Circular reactions• Physical sensations• Coordinating their body2• Preoperational stage• Symbolic thought• Ego-centric view7• Logical thought• Decentering view

• Scientific reasoning• Imitation behavior• Hand actions & face expressions→ Learning-from-observation (?)• Concrete operational stage• Formal operational stage• Repeat same actions→ Hand-eye calibration (?)→ Reinforcement learning (?)

11

Summary• Learning from observation• GPT-based encoding• TSS-skill library• Manipulation agent library• Grasp agent library• Diachronic discussion• Synchronic discussion

Recent publication & Team
GPT-encoder• Wake et. al. : arXiv:2311.12015 (2023)• GPT-4V(ision) for robotics

Katsu Ikeuchi
Jun TakamatsuTask/skill model design• Ikeuchi et.al.: IJRR (2024)• Semantic constraints to represent common sense

Kazu Sasabuchi• Skill model training• Takamatsu et. al.: arXiv:2403.02316(2024)• Designing library of skill-agents for hardware-level reusability• TSS-Platform

Naoki Wake

• Sasabuchi el.al.: IEEE RAL (2020)• Task-oriented motion mapping on robots

Atsu Kanehira