Generalizable Autonomy in Robot Manipulation

Animesh Garg

Generalizable Autonomy in Robot Manipulation

Vacuuming

Sweeping/Mopping

Cooking

Laundry

Generalizable Autonomy in Robot Manipulation

Vacuuming

Diversity:
New Scenes,
Tools,…

Complexity:
Long-term
Settings

Sweeping/Mopping

Cooking

Laundry

Generalizable Autonomy in Robot Manipulation
Vision: Build Intelligent Robotic Companions
towards Human Enrichment and Augmentation

Generalizable Autonomy in Robot Manipulation

1956

Generalizable Autonomy in Robot Manipulation
Aspirational Robotic AssistantDartmouth AI Meeting
U N I M AT E
1st Industrial robot

1956 ‘61 1968

Generalizable Autonomy in Robot Manipulation
Boston Dynamics

Dartmouth AI Meeting
U N I M AT E
1st Industrial robot
Personal Robot
Assistant1956 ‘61 1968

2013

Generalizable Autonomy in Robot Manipulation
Sony AI x Cooking

Dartmouth AI Meeting
U N I M AT E
1st Industrial robot
Personal Robot
Assistant1956 ‘61 1968

Atlas Robot
Humanoid

2013 2018

Generalizable Autonomy in Robot Manipulation
Unstructured/Unknown
New Environment

Dartmouth AI Meeting
U N I M AT E
1st Industrial robot
Personal Robot
Assistant1956 ‘61 1968

Staged Cooking
Atlas Robot
Humanoid

2013 2018 2019

Generalizable Autonomy in Robot Manipulation
How to Generalize to
Unstructured Scenarios?
Then

Now

Dartmouth AI Meeting
U N I M AT E
1st Industrial robot

Personal Robot
Assistant1956 ‘61 1968

Structured
Environments
Staged Repeated
Tasks
Atlas Robot
Humanoid

2013 2018 2019

Generalizable Autonomy in Robot Manipulation
How to Generalize to
Unstructured Scenarios?
Then

Manufacturing/Retail

Now

Personal/Service

Healthcare/Medicine

Generalizable Autonomy in Robot Manipulation
Vision: Build Intelligent Robotic Companions
Approach: Learning with Structured Inductive Bias and Priors
Demonstration

Task Imitation

Generalization

Learned
Policies

Instructional Input(Teleoperation, Video, Language)

Learn to do the task inSame Environment

New Task Variationsin Novel Environments

Layers of Imitation

Movement
SkillsSkill
Sequencing

Semantic
PurposeControl

Planning

Perception

Task Specification

Cheng et al. Sci.Rob. 2019

Generalizable Autonomy in Robot Manipulation

Perception
Control
Visuo-Motor
SkillsPlanning
Compositional
Planning

Task
Structure

Data for
Robotics

Generalizable Autonomy in Robot Manipulation
Data for
Robotics

Visuo-Motor
SkillsCompositional
Planning

Task
Structure

Visuo-Motor Skills
Challenge: Algorithmic frameworks to learn a diversity of skills
Approach: Close the Visuo-Motor Loop with Learning based Control

Vacuuming

Sweeping/Mopping

Cooking

Cleaning

Visuo-Motor Skills: Generalization

Cleaning

Hard Stains – Push Harder?

Different Surfaces – Be Gentle?

Skills: Surface Wiping

Generalization

Visuo-Motor Skills: Current Paradigm
Model Based Task (Operational) Space Control
Reference Desired
Generator State

Actual State: Image, Force, Joint Enc.
Desired State: x!

ActionRobot Model Parameters: 𝑀, 𝐽
Action: 𝜏

Robot Model

Environment
Model

Robot Model

ActualState

Environment

Environment Model + Reference Generator𝑥̈ !"# = 𝐾$ (𝑥% − 𝑥) + 𝐾& 𝑥̇ % − 𝑥̇ + 𝑥̈ %𝑀 𝑞, 𝑞̇ + 𝐶 𝑞, 𝑞̇ + 𝐺 𝑞 + 𝜀 𝑞, 𝑞̇ = 𝜏̇ ̇ + 𝐽𝑀() 𝐹)
𝜏 = 𝐽' 𝐽𝑀() 𝐽' () (𝑥̈ !"# −𝐽𝑞+ Leverages Robot Model+ Compliant Control- Needs Environment (Task) Model- Task Dependent State- Explicit State Estimation[Khatib.1987], [Bruyninckx et al. , ’96], [Schaal ‘03], [Rodriguez et al. 2012], [Vijaykumar et al. ‘11],[Ijspreet et al. ‘13] ][Li, Billard, et al. ‘14], [Lee, Abbeel et al. ‘15]

Visuo-Motor Skills: Current Paradigm
Agent

Deep Reinforcement Learning
State:
Image

Policy𝜋Action: Reward
TorqueτState

ActionTraining
Environment
Testing
Environment+ Model Free: No Environment Model+ State is Image- Sample Inefficient- Learn robot model (implicitly)- If Training ≉Testing:
Policy Fails!

[Agrawal et al., ’16], [Levine et al., ’16], [Peng et al., ’17], [Gu et al., ’16], [Chebotar et al., ’17], [Yahya et al., ’16], [James et al., ’17], [Popov et al., ’17]

Visuo-Motor Skills: Our Approach
RL with Variable Impedance Task-Space
Reference Desired
Generator State

Agent
RewardState

Training
Environment

Robot Model
ActionActionEnvironment
Model

ActualState

Environment

IROS 2019

Visuo-Motor Skills: Our Approach
RL with Variable Impedance Task-Space
Action 𝑎(20 Hz)Model-Free
RL Agent𝜋Reference Generator(learned)

Controller 𝑓
ActionActualState𝜏 = 𝑓(𝜋 𝑜! )Robot Model(Deterministic)

Environment

IROS 2019

Visuo-Motor Skills: Our Approach
RL with Variable Impedance Task-Space
Model-Free
RL Agent

Action 𝑎(20 Hz)Interpolation𝜋Controller 𝑓

SensoryObs.
𝑜!

Environment

Torque 𝜏(500 Hz)𝜏 = 𝑓(𝜋 𝑜! )𝜋 𝑜' = 𝑎: [𝑥( , 𝑥̇ ( , 𝐾) , 𝐾* ]Pose and
Velocity

ImpedanceGains𝜏 = 𝑓(𝑥( , 𝑥̇ ( , 𝐾) , 𝐾* )Deterministic Position-Velocity
Control Jacobian 𝐽 and Inertia 𝑀+ Model Free: No Environment Model+ State is Image+ Leverages Robot Model+ Compliant Control+ Sample Efficient+ Transferable
IROS 2019

Visuo-Motor Skills: Action Representation
Minimize the number
of Dirty Tiles

Surface Wiping
Input: Image (48x48)

Reward: 𝜆) ∑ 𝑑𝑖𝑟𝑡_𝑜𝑛_𝑡𝑎𝑏𝑙𝑒

Don’t push with morethan Robot Payload

Maintain Contact
with the Table+ 𝜆* 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒_𝑡𝑜_𝑡𝑎𝑏𝑙𝑒 − 𝜆+ 𝕀 (𝐹 ≥ 40𝑁)
Policy Output

6000

5ewardEE Var Imp (Ours)

4000

EE Fixed Imp. (Med.)EE Fixed Imp. (High)
Joint Var. Imp.

Performance Gap
after 1M steps

2000

Joint Velocity
Joint Imp.
EE Imp (low)Joint Torque

0
0.0

0.5

1.0

1.5
6

Trained Policy Rollout (Ours)

1umber Rf 6teSs (10 )Evaluation During Training (PPO)IROS 2019

Visuo-Motor Skills: Action Representation
Evaluation on Real Robot without Fine-Tuning

Training on Panda

𝜏 = 𝑓,-. (𝜋 𝑜/ )

𝜏 = 𝑓0"12 (𝜋 𝑜/ )Success 80% (10 Trials)

Generalizable Autonomy in Robot Manipulation
Data for
Robotics

Visuo-Motor
SkillsCompositional
Planning

Task
Structure

Skills: Imitation from Heuristics

Promise of Deep RL
closed loop-control with images…albeit, with a lot of training[Kalashnikov et al (2018). Levine et al. (2016), Pinto et al. (2016), Kalashnikov et al. (2018),
Yu et al. (2016), Haarnoja et al. (2018), Lee et al. (2019), Vecerik et al. (2017)]

Skills: Heuristics often beat RL
RL struggles with structured, multi-step skills
Even simple heuristics beat RL

IROS 2019, Mahmood et al. 2018

Skills: Exploration without Guidance

Random Exploration is slow

…even when first steps are obvious

Can Human Intuition Guide Exploration?
Gu et al. 2017

Skills: Imitation from Heuristics
Teachers
RL Task
HumanExpertIntuition
Implement Useful Skills...but not full solution
Teachers
Black-box controllers
solving parts of the task

AC-Teach: CoRL 2019

Skills: Imitation from Heuristics
Teachers
Human
ExpertAgent Exploration
Guided by Teachers

Goals: A) faster agent training B) optimal test-time agent performance

Skills: Imitation from Heuristics
Naive action choice might not work well!

AC-Teach: CoRL 2019

Off-Policy RL: DDPG Review

Lillicrap et al. 2015

AC-Teach: Actor-Critic with Teachers
Agent Policy
Actor
Teacher 1

Bayesian
CriticTeacher 2...

Estimate quality of
the advice

Teacher N

Environment

Off-Policy Data
Added to Buffer 𝜋3ActionSelection 𝜋+
AC-Teach: CoRL 2019

Experiments
Task:

Teachers:

grab hookposition hook

pull

push
AC-Teach: CoRL 2019

Results

AC-Teach is able to leverage a single teacher well
AC-Teach: CoRL 2019

Results

AC-Teach speeds up training given multiple teachers
AC-Teach: CoRL 2019

Results

AC-Teach has agent learn behaviors not in teacher set
AC-Teach: CoRL 2019

Generalizable
in Robot Manipulation
Visuo-Motor Autonomy
Skills
Grasping

Pushing

Picking
WipingOpen door

IROS 2019

CoRL 2019

Data for
Robotics

Task
Structure
Action Representations
provideCompositional and Weak-Supervision
Visuo-Motor
Planning
Skillsstructure
to enable learning efficiency and generalization

Generalizable Autonomy in Robot Manipulation
Stack
Pick

Grasp

Place

Move

ReleaseObject Arrangement

Visuo-Motor
SkillsData for
Robotics

RSS 2018, IJRR 2019

Compositional
Planning

Task
StructureSequential Skills
Primitive Skills
Grasping

Pushing
Picking

WipingOpen door

Skills: Surface Wiping
Sequential Skills
Hammering (with unknown objects)
Cutting (with new knife)

Skills: Tool Use

Sweeping (with new broom)Sequential Skills: Manipulation with Tools

Task-Oriented Grasping

Tool-Use
Initial State

Unknown Object

Task-Agnostic
Grasping1

Optimizes for Grasp
Success Only
Suboptimal for Task!

1 Pinto et al. ’16, Levine et al. ‘16, Mahler et al. ‘18, Kalashnikov et al. ‘18

Sweeping

HammeringVisuo-Motor Skills: Task-Oriented Grasping
Output =Input =

Hammering

Depth Image 𝑜Task T

Task
Success

Grasp Parameters 𝑔 = (𝑔" , 𝑔# , 𝑔$ , 𝑔% )𝑔∗, 𝜋 ∗ = argmax 𝑄'4 (𝑜, 𝑔)
7,4

Policy a = 𝜋(𝑜, 𝑔)

Score Function𝑔)1,
𝑄'4 𝑜, 𝑔 = 𝑃4 𝑆' = 1
1, 𝑆𝑜,
1 𝑜,𝑜,𝑔)𝑃𝑔) 𝑆5 = 1 𝑜, 𝑔)
5 =4
𝑄'4 𝑜, 𝑔 = 𝑄Task
𝑜, 𝑔 𝑄5 𝑜, 𝑔'|5 Success

Grasp
Success

Task Conditioned
Grasp Success

Grasp Success
RSS 2018, IJRR 2019

Visuo-Motor Skills: Task-Oriented Grasping
CNN Model

OutputLoss Function

Input: Depth Image (64x64)
grasping
model

𝑄5 𝑜, 𝑔ℒ(𝑆5 , 𝑄5 𝑜, 𝑔; 𝜃) )task-oriented
grasping
model

4
𝑄'|5
𝑜, 𝑔

4
𝕀(𝑆5 = 1 )ℒ(𝑆' , 𝑄'|5𝑜, 𝑔; 𝜃* )Manipulation

𝜋 𝑎 | 𝑜, 𝑔; 𝜃+∇3" log 𝜋 𝑎 | 𝑜, 𝑔; 𝜃+

grasping stage

Training

manipulation stage

4𝑄'4 𝑜, 𝑔 = 𝑄'|5𝑜, 𝑔 𝑄5 𝑜, 𝑔

18K diverse training objects

Task Conditioned
Grasp Success

Grasp Successlarge-scale data collection (1.5m Episodes)

RSS 2018, IJRR 2019

Visuo-Motor Skills: Task-Oriented Grasping
hammeringtask

Testing
Sample grasp
candidates

1

manipulation
policyrobusttask-oriented
grasp

2

task execution

grasp ranking

task-oriented
grasping
model

3

4

RSS 2018, IJRR 2019

Sequential Skills: Task-Oriented Grasping
80
70

Success Rate

60

29%

50
40

29%

30
20
10
0

Hammering
Trained Policy Rollout (Ours)
Unseen Test Objects
RSS 2018, IJRR 2019

Antipodal Baseline

Sweeping
Task-Agnostic

Our Model

Two-Stage
Joint
Optimization Optimization

Sequential Skills: Task-Oriented Grasping
80
70

Success Rate

60

36%

50
40
30
20
10
0

Hammering
Trained Policy Rollout (Ours)
Unseen Test Objects
RSS 2018, IJRR 2019

Antipodal Baseline

Task-Agnostic

Our Model

Two-Stage
Joint
Optimization Optimization

Sequential Skills: Task-Oriented Grasping
80
70

Success Rate

60

36%

50

47%

40
30
20
10
0

Hammering
Trained Policy Rollout (Ours)
Unseen Test Objects
RSS 2018, IJRR 2019

Antipodal Baseline

Sweeping
Task-Agnostic

Our Model

Two-Stage
Joint
Optimization Optimization

Sequential Skills
Primitive Skills
Grasping

Pushing
Picking

WipingOpen door

Skills: Surface Wiping
Sequential Skills
Hammering (with unknown objects)
Cutting (with new knife)

Skills: Tool Use

Sweeping (with new broom)Sequential Skills: Multi-Step Reasoning

Skills: Multi-Step Reasoning

Generalization

Can we learn multi-step reasoning in robotics
under physical and semantic constraints
target object
goal position

Model-based learningchoose action sequenceat , ……, at+H
dynamicss’ ~ f (· | s, a)s0[Deisenroth et al, RSS’07], [Guo et al, NeurIPS’14], [Watter et al, NeurIPS’15], [Finn et al, ICRA’17], …...

Model-based learning?

[Deisenroth et al. RSS’07] [Agrawal et al. ICRA’16]

data ↑
learning ↑[Ebert et al. CoRL’17][Janer et al. ICRA’19]

CAVIN: Hierarchical planning in learned latent spaces
CAVIN Planner

effect code cs0motion code z

Leverage Hierarchical Abstraction in Action Space
Without Hierarchical SupervisionCAVIN: Hierarchical planning in learned latent spaces
CAVIN Plannermotion code z

effect code cs0subgoalsCAVIN: Hierarchical planning in learned latent spaces
CAVIN Planner

effect code cs0actions

motion code z

CAVIN: Hierarchical planning in learned latent spaces
choosec ~ p (c)meta-dynamics

s0

CAVIN: Hierarchical planning in learned latent spaces
choosec ~ p (c)meta-dynamics
st+3T
st+2T

s0
st+T

Hierarchical planning in learned latent spaces
choosec ~ p (c)choosez ~ p (z)meta-dynamics

action generators0

CAVIN: Hierarchical planning in learned latent spaces
choosec ~ p (c)choosez ~ p (z)meta-dynamics

action generator

at+1
at+2
at+3

s0

dynamics

Learning with cascaded variational inferencetask-agnostic interaction

meta-dynamics

action generatorh (s’’ | s, c)g (a | s, c, z)

c

z

μc

Σc

qh (c | s, s’’)μzΣz

qg (z | s, c, a)visual observation
preprocess
kinect2 sensor

st

CAVIN Planner

action[ x, y, Δx, Δy ]

Tasksclearing

crossing

insertion

goal position

targetobjectgoal position

targetobjectClear all objects within theMove the target to the goal

Move the target to the goal

area of blue tiles.

without traversing red tiles.

across grey tiles

SimulatedRealQuantitative Evaluation
100

Dense

12%

90
80

Sparse15%

Hierarchical Latent space dyn.
↓
Better performance with sparse
reward signal

Success Rate (%)

70
60
50
40
30
20
10
0
MPC

CVAE-MPC

SeCTARCAVIN

CAVIN-Real

Averaged over 3 Tasks
with 1000 test instances each

MPC (Guo et al. ‘14, Agrawal et al. ‘16, Finn et al. 17); CVAE-MPC (Ichter et al. 18), SeCTAR (Co-Reyes et al ‘18)

5xMove 2 obstacles

5x

Get around5x

Squeeze through5xOpen pathGeneralizable
Autonomy
Compositional
Planningin Robot Manipulation
Stack
Pick

Grasp

Place

Move

ReleaseObject Arrangement

RSS 2018,
IJRR 2019

CoRL 2019 (oral)

Data for
Robotics

Task
StructureSelf-Supervision
and Structured
Latent Variable Models
Compositional
Visuo-Motor
Planning
Skillsto good representations that generalize
leadGeneralizable Autonomy in Robot Manipulation
Stack
Pick

Grasp

Place

Move

ReleaseObject Arrangement

Visuo-Motor
SkillsMove_to (Red)ICRA 2018

Data for
Robotics

CVPR 2019 (oral)

Compositional
Planning

Task
Structure

Complex Task Structure
Visuo-Motor Skills
Grasping

HammeringPushing

Cutting

Picking
Open door

WipingSweeping

Visuo-Motor Skills
Complex Task Structure
Laundry
Put inWasher

Pick

Push

Grasp

Put
Soap

Complex Task Structure

Open

Pick

Stack

Do Laundry

Place

Move

ReleaseObjectArrangement

Compositional Planning: Current Paradigm
Desired
Schulman et al 2013

Train ≠ Test

Reinforcement Learning

Imitation Learning- Sample Inefficient- Task Segmentation is non-trivial- Multi-step Structured Tasks- Multi-modality of Search Space- Needs non-trivial Reward Shaping - Fixed Permutation of Primitives

Meta Imitation Learning- New Task Structures- Few-Shot performance- Input State as Video

RL: [Schaal 1997], [Chebotar et al., ’17], [Yahya et al., ’16], [James et al., ’17], [Popov et al., ’17], [Zhu et al. 18], [Hausman et al. 18]
Imitation: [Calinon et al 2008], [Argall et al 2009], [Kober, Peters, et al. 09], [Pastor et al, 09], [Schulman et al. 2013], [Kroemer et al, 15], [Garg et al 2017]

Compositional Planning: Challenge
Task Domain

I. Learn Multiple Tasks
II. Generalize to New
in the Same Domain Tasks with a Single Demo
Training Tasks

Instructional Demos

Test Task

Compositional Planning[Duan et al. 17; Finn et al. 2017;
Wang et al. 2017; Yu et al. 2018]

Our Method[ICRA’18], [CVPR’19], [IROS’19]

Models input demonstration
as a flat sequence

Models input demonstration
as a Compositional Hierarchy

One Shot Imitation Learning from Videos

Compositional Planning: Task Programming
Block Stacking (...):
while (done):
pick_and_place (RED, BLUE):
pick (RED):
move_to (RED)Grasp (RED)<end> Popplace(BLUE):
move_to (BLUE)
Release (RED)<end> Pop<end> Pop

Task 1
Sub-task 1
Move Red-block on top of Blue[ICRA’18, CVPR ‘19]

Compositional Planning: Task Programming
Program 1

Training Task Structures

Program 2
Task 1

Task 2[ICRA’18, CVPR ‘19]

Neural Task Programming (NTP)
Training supervisionmeta-learning
model

{(

Demo

Demo

,

)}

Hierarchical Program Trace
Demo Conditional Policy

Kinematics or Video

Next Program
pick (blue)

Current State
End-to-End LSTM Model

Current Program
pick_place(blue, green)

ℒ(

,

)

Loss: Compare with
Ground Truth Program

Hierarchical Policy Learning as Program Induction[ICRA’18, CVPR ‘19]

…

…
,QSXW7DVN6SHF(23)DOVH
Block Stacking(23)DOVH
Block Stacking

3LQEORFNBVWDFNLQJ

3LQEORFNBVWDFNLQJ

2XWSXW7DVN6SHF

(QY2EVHUYDWLRQ,QSXW7DVN6SHF

Pick and(23)DOVH
Place

3RXWPRYHBWR(QY2EVHUYDWLRQ,QSXW7DVN6SHF

Pick(23)DOVH

3LQSLFN
3RXWJULS$UJVEORFNB(PRYHBWR EORFNB(

Move_to (Blue)

2XWSXW7DVN6SHF

3RXWGURSUHWXUQ,QSXW7DVN6SHF

Pick(237UXH

UHWXUQ

3LQSLFN

3LQSLFNBDQGBSODFH

2XWSXW7DVN6SHF

(QY2EVHUYDWLRQ

2XWSXW7DVN6SHF

3RXWSLFNBDQGBSODFH

Pick and (237UXH
Place

3LQSLFNBDQGBSODFH
3RXWSLFN,QSXW7DVN6SHF

(QY2EVHUYDWLRQUHWXUQ

3RXWSLFNBDQGBSODFH,QSXW7DVN6SHF

(QY2EVHUYDWLRQ$UJVEORFNB(JULS EORFNB(

Grasp (Blue)(QY2EVHUYDWLRQ
3LQSODFH(QY2EVHUYDWLRQ(23)DOVH
Place

3RXWPRYHBWRUHWXUQ,QSXW7DVN6SHF

3LQSODFH
3RXWUHOHDVH$UJVEORFNB%PRYHBWR EORFNB%

Move_to (Red)UHWXUQ,QSXW7DVN6SHF(237UXH
Place

return(QY2EVHUYDWLRQ$UJV1$UHOHDVH

Release( )UHWXUQNeural Task Programming

Autonomous Execution

Demo[ICRA ’18]

8x

Recovery from Intermediate Failures
Output is not anOpen Loop State
MachineClosed Loop
Feedback PolicyNeural Task Programming
Results
Test Task Structures with State Input
1.00

Flat Policy
NTP (Ours)

84%

0.75

NTP (Detector) Pose Est. + Plan
NTP (Ours)
E2E Plan

0.75

Ours
62%

0.50

0.50

35%
0.25

1%

0.00
50

100

400

# of Training Tasks

1000

0.00

N/A

0.25

N/A

Performance
Accuracy

1.00

Test Task Structures with Vision Input

50

100

400

1000

# of Training Tasks

Better Generalization than Flat Policy + Works with Vision[ICRA’18, CVPR ‘19]

Compositional Planning: Task Programming
meta-learning
model
Demo Conditional Policy

DemoKinematics or Video

Compositional
Model Prior
Next Program
pick (blue)

Current State
End-to-End LSTM Model

Current Program
pick_place(blue, green)

Black-Box
Model

Program Induction
Inductive Bias on
Input-Output[ICRA’18, CVPR ‘19]

Compositional Planning: Task Graphs
7DVN'HPRQVWUDWLRQ
7DVN

meta-learning
model

7DVN&RQ
2XWSXW3

CCC

Demo Conditional Policy

Demo

173

7DVN

Kinematics or Video

Task Graph
Generator

6WDWHREV

CCC

Task Graph
Executor
Neural Task Graph

Hierarchical Policy Learning as Graph Induction

Current
Observation[ICRA’18, CVPR ‘19]

Neural Task Graphs (NTG)
7DVN'HPRQVWUDWLRQ
7DVN

meta-learning
model

7DVN&RQ
2XWSXW3

CCC

Demo Conditional Policy

Demo

173

7DVN

Kinematics or Video

Task Graph
Generator

6WDWHREV

CCC

Task Graph
Executor
Neural Task Graph

Parsed Sequence

Graph Completion

Hierarchical Policy Learning as Graph Induction

Current
Observation[ICRA’18, CVPR ‘19]

Neural Task Graphs (NTG)
7DVN'HPRQVWUDWLRQ
7DVN

meta-learning
model

7DVN&RQ
2XWSXW3

CCC

Demo Conditional Policy

Demo

173

7DVN

Kinematics or Video

Task Graph
Generator

6WDWHREV

CCC

Task Graph
Executor
Neural Task Graph

Hierarchical Policy Learning as Graph Induction

Current
Observation[ICRA’18, CVPR ‘19]

Neural Task Graphs (NTG): Representation
Conjugate Task Graph

Task Graph
Pick (Orange)
Pick (Red)Pick (Green)

Place (Red)

Place (Green)

Place (Green)

Valid States

{.

…

}

Pick (Green)

Pick (Orange)

Pick (Red)

Place (Red)

Nodes: States Combinatorial

Nodes: Actions

Edges: Action

Edges: States (Preconditions)Finite[ICRA’18, CVPR ‘19]

Neural Task Graphs (NTG): Execution
Node
Localizer

Selected
Node
Place (Green)

Pick (Green)

Selected
Pick (Orange)
Edge

Observation

Edge
Classifier

Pick (Red)

Place (Red)Pick (Red)

Selected
Next Action[ICRA’18, CVPR ‘19]

Neural Task Graphs (NTG)
Training supervision

{(
Demo

Demo

7DVN

7DVN'HPRQVWUDWLRQ

,

CCC

)}

7DVN&RQ
2XWSXW3

Action Sequence

173

7DVN

Kinematics or Video

Task Graph
Generator

CCC

6WDWHREV

Demo Conditional
PolicyTask Graph
Executor
Neural Task Graph

Hierarchical Policy Learning as Graph Induction

Current
Observation[ICRA’18, CVPR ‘19]

Neural Task Graph
Results
Ours
Test Task Structures with Vision Input

Test Task Structures with State Input
1.00

91%
84%

Flat Policy

0.75

NTG (Ours)

NTP (Detector)90%

NTP (Ours)
0.75

NTG (Ours)

62%
0.50

0.50

35%

1%
0.00

0.00
50

100

400

# of Training Tasks

1000

N/A

0.25

0.25

N/A

Performance
Accuracy

NTP (Ours)

1.00

50

100

400

1000

# of Training Tasks

Weaker Supervision and Better Generalization[ICRA’18, CVPR ‘19]

Compositional Planning: NTP and NTG

Object Sorting(NTP)

Table Clean Up(NTP)

Sequential Search and Prediction
AI2 Thor with NTG[ICRA’18, CVPR ‘19]

Generalizable
Autonomy
Task Structure
Learningin Robot Manipulation
Stack
Pick

Grasp

Place

Move

ReleaseObject Arrangement

Move_to (Red)ICRA 2018

CVPR 2019 (oral)

Data for
Robotics

Task
Structure
Compositional
priors with modular
structure enable
Compositional
Visuo-Motor
Planning
Skillsgeneralizable
learning in hierarchical domains

Generalizable Autonomy in Robot Manipulation+
Data for
Robotics

CoRL 2018, IROS 2019

Visuo-Motor
SkillsCompositional
Planning

Task
Structure

Data for Robotics
Visuo-Motor
Skills
Grasping

Complex Task
Structure

Procedural Execution
from Videos

?

Stack

Pushing

Picking
WipingSequential
Tasks

Open door

Pick

Grasp

Place

Move

ReleaseObject Arrangement

Dataset
Size

106

105

103

Reinforcement
Self-Supervision

Structured
Supervision

WeaklySupervised?
How to get Larger Sets of Structured Supervision?

Data for Robotics: Imitation + RL

Rajeswaran et al. (2018)
25 demonstrations~ 10 Minutes

Finn et al. (2017)
30 demonstrations~ 10 Minutes

Vecerik et al. (2017)
100 demonstrations~ 30 Minutes

Data for Robotics
16
13.7

Dataset Size (Hours)

14

Vision12
10

Samples

Large-scale supervision in robotics is difficult

8

Robotics~103
Samples

6

Expert needs to demonstrate, not label
4.08

4
2

NLP

>106

1.66

2.35

0

JIGSAWS

Deep Imitation

DAML

MIME

Gao et al.
2014

Zhang et al.
2018

Yu, Finn et al.
2018

Sharma et al.
2018>106-8SamplesData for Robotics: RoboTurk

User Interface+
Web Browser View

+ Scales easily with commodity hardware+ Natural 6-DoF Free Space Control[CoRL ’18, IROS 2019]

RoboTurk: Scaling Imitation with CloudROBOTURK

RoboTurk: Imitation for everyone, everywhereRoboTurk Pilot Datasets
Simulated Data

Real Robot Data

137.5 hours of demonstrations

111 hours of robot demos

22 hours of total platform usage

1 week of data collection

3 dexterous manipulation tasks

3 dexterous manipulation tasks3224 total attempted demos

2144 total demonstrations

15 novice, remote users

54 non-expert users[CoRL ’18, IROS 2019]Data for Robotics: RoboTurk
16
13.7

Dataset Size (Hours)

14
12
10
8
6
4.08

4
2

1.66

2.35

0

JIGSAWS

Deep Imitation

DAML

MIME

Gao et al.
2014

Zhang et al.
2018

Yu, Finn et al.
2018

Sharma et al.
2018[CoRL ’18, IROS 2019]Data for Robotics: RoboTurk
160
137.5

Dataset Size (Hours)

140
120

111.25

100
80

10x

60
40
20
0

13.7
1.66

2.35

4.08

JIGSAWS

Deep Imitation

DAML

Gao et al.
2014

Zhang et al.
2018

Yu, Finn et al.
2018

MIME
Sharma et al.
2018

RoboTurk-Sim RoboTurk-RealOurs
CoRL 2018

Ours
IROS 2019[CoRL ’18, IROS 2019]Data for Robotics: RoboTurk

Trained Policy Rollout

Imitation + RL

Task Performance

Task Performance vs. Number of Demonstrations
1000
800

Pure RL(Sparse Reward)

600

Nut Assembly

400
200
0
0

1

10

100

Number of Demonstrations
Distributed PPO with curriculum: 32 workers, 24/48 hours

1000
Bin Picking[CoRL ’18, IROS 2019]

Generalizable
Autonomyin Robot Manipulation
Data for Robotics:
RoboTurk+
CoRL 2018, IROS 2019

Data for
Robotics

Task
Structure
Structured supervision
for Robotics
through scalable
Compositional
Visuo-Motor
Planning
Skills
crowdsourcing
can empower robot learning in complex tasks.

Generalizable Autonomy in Robot Manipulation

Perception
Control
Visuo-Motor
SkillsPlanning
Compositional
Planning

Task
Structure

Data for
Robotics

Opportunity: Personal Robotics”

Instructional Youtube Video
How to make Meatball Pasta?

Where / How should Rosie start?
What is the recipe?
How to execute the plan?
How to plan?

Reasoning for Physical Interaction
Understanding Purpose

Ideal Tool
During Training

Task-Based Tool Adaptation
During Execution

Grounding: So many ways to “make eggs”

Generalizable Autonomy in Robot Manipulation
FailurePersonFrame
Holds
On TopWood Top

Drill
ScrewsHigher-Order Semantics

What makes anobject a hammer?

Success
State Change: Breaking Eggs•

Perception for Physical Interaction•

Transfer Learning with Formal Guarantees•

Reasoning through Learned Dynamics•

Continual Skill Adaptation & Accumulation

Generalizable Autonomy in Robot Manipulation
Learning &Control

Learning with Structured Inductive Bias and Priors- Efficiency andPlanning
Generalization
Data for
Robotics-Perception
Combine: Domain Expertise + Data-Driven Methods

Visuo-Motor
SkillsCompositional
Planning

Task
Structure

Generalizable Autonomy in Robot Manipulation

garg@cs.toronto.edu@Animesh_Garg

Animesh Garg