Robot Learning: Algorithms
and Applications

Abdeslam Boularias

Outline
1. Overview!
2. Optimal control!
3. Inverse optimal control!
4. Grasping !
5. Manipulation!
6. Navigation

Outline
1. Overview!
2. Optimal control!
3. Inverse optimal control!
4. Grasping !
5. Manipulation!
6. Navigation

Acting in unstructured environments

DLR

How can a robot learn
to perform complex
tasks from experience?

DLR

How can a robot learn
to perform complex
tasks from experience?
Robotics

MachineLearning

Artificial
Intelligence

DLR

Controlling a Dynamical System

ActionDLRControlling a Dynamical System

Observation

ActionDLROutline
1. Overview!
2. Optimal control!
3. Inverse optimal control!
4. Grasping !
5. Manipulation!
6. Navigation

Example

Path planning: a simple sequential decision-making problem

Example

Path planning: a simple sequential decision-making problem

States and Actions

Markov Decision Process (MDP)Notations❖

S: set of states (e.g. position and velocity of the robot)!

❖

A: set of actions (e.g. force)!

❖

T: stochastic transition function

next state❖

R: reward (or cost) function,

current state

current action

Policies
A policy is a function " that maps each state to an action,

Value function
The value (or utility) of a policy " is the sum of
rewards that one expects to gain by following it.

Goal: finding an optimal policy.

Partially Observable Markov Decision Process❖

Observations are partial and noisy.!

❖States cannot be precisely known. !

❖Belief state: a probability distribution on states

unobserved actual states

Belief states

Partially Observable Markov Decision Process❖

Observations are partial and noisy.!

❖States cannot be precisely known. !

❖Belief state: a probability distribution on states

unobserved actual states

Belief states

Partially Observable Markov Decision Process❖

Observations are partial and noisy.!

❖States cannot be precisely known. !

❖Belief state: a probability distribution on states

unobserved actual states

Belief states

Partially Observable Markov Decision Process (POMDP)Example: the robot can sense an obstacle only after bumping into it.

Where am I?

Belief statePartially Observable Markov Decision Process (POMDP)Example: the robot can sense an obstacle only after bumping into it.

Where am I?

Belief statePartially Observable Markov Decision Process (POMDP)Example: the robot can sense an obstacle only after bumping into it.

Where am I?

Belief stateThe target of the ball should be predicted in advance

Once the opponent strikes
the ball, it becomes too late
for the robot to start reacting.

Probabilistic graphical model of intention-driven dynamics
Intended
TargetHidden variables

Observations
Positions of the ball,the racket and joints of
the opponent, tracked
using a Kinect camera

xt-1

xt

xt+1

ot-1

ot

ot+1

.

.

.

Observations ot are generated according to a Gaussian Process.
From observations ot , one can calculate a probability
distribution on the intended target.

approx. 320ms
0.5

posterior

0.4

approx. 160ms

approx. 80ms (before hit)backhandmiddleforehand

0.3
0.2
0.1
0

−0.2

0.4

X

−0.2

0.4

X

−0.2

0.4

X

approx. 320ms
0.5

posterior

0.4

approx. 160ms

approx. 80ms (before hit)

Optimal Stopping Problem

backhandmiddleforehand

0.3
0.2
0.1
0

−0.2

0.4

X

−0.2

0.4

X

−0.2

0.4

X

A Monte-Carlo planning algorithm
Sample current state and intention st = [xt , target]

A Monte-Carlo planning algorithm
Sample current state and intention st = [xt , target]
Sample subsequent state xt+1 ∼ P( .|st)A Monte-Carlo planning algorithm
Sample current state and intention st = [xt , target]
Sample subsequent state xt+1 ∼ P( .|st)Sample subsequent observation ot+1 ∼ P( .| xt+1).

A Monte-Carlo planning algorithm
Sample current state and intention st = [xt , target]
Sample subsequent state xt+1 ∼ P( .|st)Sample subsequent observation ot+1 ∼ P( .| xt+1)
Update belief bt+1 provided observation ot+1.

A Monte-Carlo planning algorithm
Sample current state and intention st = [xt , target]
Sample subsequent state xt+1 ∼ P( .|st)Sample subsequent observation ot+1 ∼ P( .| xt+1)
Update belief bt+1 provided observation ot+1
Predict the expected performance V(bt+1).

A Monte-Carlo planning algorithm
Sample current state and intention st = [xt , target]
Sample subsequent state xt+1 ∼ P( .|st)Sample subsequent observation ot+1 ∼ P( .| xt+1)
Update belief bt+1 provided observation ot+1
Predict the expected performance V(bt+1)
No

enough samples?

.

A Monte-Carlo planning algorithm
Sample current state and intention st = [xt , target]
Sample subsequent state xt+1 ∼ P( .|st)Sample subsequent observation ot+1 ∼ P( .| xt+1).

Update belief bt+1 provided observation ot+1
Predict the expected performance V(bt+1)

Wait and seeNoNoenough samples?

Yes

V(bt) > V(bt+1)
Yes

strike back!

Average number of successful returns
170

average number of successful returns

LSPI

MCP

160
150.2 ± 1.2
147.4 ± 1.3

150144.2 ± 1.8

145.5 ± 1.2

152.0 ± 0.7
147.3 ± 3.0

139.6 ± 2.2
140131.3 ± 3.1
130

120

110

100

180

360

720

1440

number of training episodes

Z. Wang, A. Boularias et al. (2015) in Artificial Intelligence Journal.

8: Performance of the LSPI and MCP methods evaluated in ten repetitions on the test data with 207 valid trials, baseOptimal Control

RewardTransition Function (Dynamics)

Designing a useful reward function forcomplex behaviors is a tedious task.

PolicyOptimal Control

RewardTransition Function (Dynamics)

PolicyInverse Optimal Control

ExamplesDesigning a useful reward function forcomplex behaviors is a tedious task.

Outline
1. Overview!
2. Optimal control!
3. Inverse optimal control!
4. Grasping !
5. Manipulation!
6. Navigation

Inverse Optimal Control
Assumption: The reward is a linear function of state-action features

def

R = w

T

rewardfeatures (e.g. velocity, energy,
distance from goal, ..)unknown
weights

Inverse Optimal Control
Assumption: The reward is a linear function of state-action features

def

R = w
Value of policy "V (⇡) =

H
X
t=0

T

Est [R(st , ⇡(st ))]Inverse Optimal Control
Assumption: The reward is a linear function of state-action features

def

R = w
Value of policy "V (⇡) =H
X

T

Est [R(st , ⇡(st ))]

t=0

=

n
X

k=1

expected features under ! !
(e.g. expected energy, distance, etc.)

wk

H
X
t=0

|

Est [ k (st , at )] = wT (⇡)
{z

(⇡)

}

Inverse Optimal Control: Problem Statement
Given an expert’s policy "∗, find reward weights w such that:

w

T

⇤

(⇡ )

Value of the expert’s policy

max w
⇡

T

!

(⇡)

Value of an arbitrary policy

In other terms, expert’s policy "∗ has the highest possible value.

!

Inverse problems are generally ill-posed
Given an expert’s policy "∗, find reward weights w such that:

w

T

⇤

(⇡ )

Value of the expert’s policy

max w
⇡

T

(⇡)

Value of an arbitrary policy!

Relative Entropy Inverse Optimal Control
Find P, a probability distribution on state-action trajectories ⌧Z

P (⌧ )d⌧ = 1

8⌧ : P (⌧ )kE⌧ ⇠P [ (⌧ )]⇤⌧Expected features under P

0(⇡ )k  ✏Expected features in the expert’s
demonstration

Relative Entropy Inverse Optimal Control
Find P, a probability distribution on state-action trajectories ⌧Solve
Subject to:convexmin DKL (P kQ)

ZPReference distribution

P (⌧ )d⌧ = 1

8⌧ : P (⌧ )kE⌧ ⇠P [ (⌧ )]⇤⌧Expected features under P

0(⇡ )k  ✏Expected features in the expert’s
demonstration

Relative Entropy Inverse Optimal Control
Solution

Reference distribution

P (⌧ |w) / Q(⌧ ) exp w

T

(⌧ )

Expected return

Reward weights w are obtained by gradient descentIn the remainder of this paper, we will proceed as follows.
function of a policy π evalu
In Sect. 2, we present the theoretical background for mod!∞eling decision processes, including MDPs and the used IRL
"
π
t
(s)
=
E
γ
R(st , atValgorithms. We present the experimental setup and evaluaRobot
Table
Tennis:LearningtoImitatean
ExpertPlayert=0
tions in Sect. 3. In Sect. 4, we summarize our approach andthe results.
and corresponds to the exp
Goal: Learn a reward function from demonstrations ofstartinga professionalfrom stateplayers. The oby V ∗ (s) = maxπ V π (s) ∀a MDP is to find the optim
maximizes the value for eve
Trajectories
We assumeof
thatthethe rewa
m feature fu
ballcombinationand theofbodies
reward function is thereforeof the players werem"captured
wi f i (s, a) =R(s, a) = using
i=1
infrared markers.

Fig. 1 Considered Scenario. Two people playing a competitive match

where w ∈ Rm and f(s, a)known, bounded basis func
R. For a given trajectory τ
counts are given by f iτ =

Robot Table Tennis: Learning to Imitate an Expert Player
Biol Cybern

State st : position of the ball + positions of the players at time tBiol Cybern

Learned reward function

Red indicates good location for bouncing the ball.

The preferences of the locations on the ta
presented directly after the current one. A horizon of threedent from the state information of the opmeans that we consider up to two state–action pairs following
do reflect parts of the strategy that will als
the current one.
other features. The resulting reward functio
The results of the average reward differences of the nonopent algorithms showed a preference for the
timal policies and the expert for the whole game and the states
opponent would have to return the ball usin
directly before the terminal are displayed in Table 2. In genwhile the areas that are suited for returni
eral, the average reward difference was reduced slightly with
K. Muelling, A. Boularias et al. (2014) in Biological Cybernetics
108(5): 603-619.
the forehand and the areas directly after t
increasing horizon, while the average reward difference forrather avoided (see Fig. 7a). The difference
the last H − 1 states before the terminal state increases withreward for the goal preferences on the tab
growing planning horizon, reaching its maximum with a hori-

Robot Table Tennis: Learning to Imitate an Expert Player

Average reward of each player is predicted from just theway she/he plays (without looking at the scores)

Table 2 Summary of the results for the different horizons with Algorithm 3
horizon

Naive 1

Naive 2

Naive 3

Naive 4

Naive 5SkilledAverage reward difference

1

1.30

0.04

1.17

0.91

0.74

0.30

with respect to the expert

2

1.20

0.07

1.22

0.87

0.72

0.33

3

1.16

0.07

1.24

0.86

0.71

0.33

Average reward differences

2

0.91

0.92

0.57

0.38

directly before terminal state

3

1.12

−0.21

1.23

0.89

0.76

−0.12

0.04

0.24

The differences in the average reward with respect to the expert trained with the different horizons. The differences in the averbefore the terminal define the differences of the reward of the expert and the spared test subject for the state before the termTheskilledplayeris themostsimilar to
reward of the two states
beforethe terminal
for the horizons
2 and3, respectively

the expert in terms of predicted rewards

123

Example: Ball-in-a-Cup game (Kendama)Example: Ball-in-a-Cup game (Kendama)A human expert (Jens Kober) providing a demonstration

Example: Ball-in-a-Cup game (Kendama)success rate

1

Relative Entropy

0.5

Maximum Margin Planning
0

100

500

1000
samples
RE IRL2000
MMP

A. Boularias et al. (2011) in Artificial Intelligence and Statistics (AISTATS).

Outline
1. Overview!
2. Optimal control!
3. Inverse optimal control!
4. Grasping !
5. Manipulation!
6. Navigation

Purposeful Grasping
Parameters of a grasping action (position and rotation of the hand) should
be chosen depending on the intended goal.
Example: Use the handle if you plan to pour water.

Purposeful Grasping

Barrett® hand
3D image of an unknown object

Purposeful Grasping

Barrett® handsegmented object

Vision: Segment the object into partsPurposeful Grasping

irection
d
g
n
i
h
c
a
appro

Barrett® handsegmented object

Vision: Segment the object into parts

Planning: Simulate grasping actions for each part

Purposeful Grasping

irection
d
g
n
i
h
c
a
appro

Barrett® handsegmented object

Vision: Segment the object into parts

Planning: Simulate grasping actions for each part

Planning-driven Vision:
Segment the object according
to the intended goalPurposeful Grasping

irection
d
g
n
i
h
c
a
appro

Barrett® handsegmented object

Vision: Segment the object into parts

Planning: Simulate grasping actions for each part

Planning-driven Vision:
Segment the object according
to the intended goal

Learning Grasping Points with Associative Markov Networks!

An object is represented as a
k-nearest neighbor graph (V, E)

Each node in the graph can be labeled
as a success or failure with yi 2 {1, 1}

Learning Grasping Points with Associative Markov Networks!

An object is represented as a
k-nearest neighbor graph (V, E)

Each node in the graph can be labeled
as a success or failure with yi 2 {1, 1}

Joint distribution of the labels of
all points Y = {y1 , y2 , . . . , yn }
⇣X
⌘
X
T
T
P (Y ) / exp
yi wnode+wiedge ij
i2V(i,j)2E
yi =yjLearning Grasping Points with Associative Markov Networks!

An object is represented as a
k-nearest neighbor graph (V, E)

Each node in the graph can be labeled
as a success or failure with yi 2 {1, 1}

Joint distribution of the labels of
all points Y = {y1 , y2 , . . . , yn }
⇣X
⌘
X
T
T
P (Y ) / exp
yi wnode+wiedge ij
i2V(i,j)2E
yi =yjweight vectors

Learning Grasping Points with Associative Markov Networks
An object is represented as a
k-nearest neighbor graph (V, E)!

Each node in the graph can be labeled
as a success or failure with yi 2 {1, 1}

-1

-1

-1
-1

-1

Failure examples

+1
+1
+1
+1 +1

Success examples

Joint distribution of the labels of
all points Y = {y1 , y2 , . . . , yn }
⇣X
⌘
X
T
T
P (Y ) / exp
yi wnode+wiedge ij
i2V(i,j)2E
yi =yjweight vectors

Associative Markov
Networks

Logistic Regression

A. Boularias et al. (2011) in IEEE International Conference on Intelligent Robots and Systems (IROS)

Results
Percentage of successful grasps
100

Associative Markov Network
Logistic Regression

80

60

40

20

Setup

0
watering can plastic basket woven basket

A. Boularias et al. (2011) in IEEE International Conference on Intelligent Robots and Systems (IROS)

An Autonomous Robot for Rubble Removal

Source: Amedeo Troiani/Getty Images Europe

Source: AFP

Rubble removal is a major
challenge in search-and-rescue
missions
Tele-operation is
tedious and requires ahuman expert

An Autonomous Robot for Rubble Removal

Two Barrett® arms and hands with a Kinect® camera

An Autonomous Robot for Rubble Removal
Most autonomous
grasping techniques use
models of the objects

Two Barrett® arms and hands with a Kinect® camera

Objects found in rubble,
such as rocks, areirregular and unknown
to the robot. Therefore, wecannot rely on models!

Grasping Regular Objects: A Simple Heuristic

Take a 3D imageof the scene

Segment the 3D point
cloud into facets byusing the mean-shiftalgorithm

Simulate grasping
actions for each facet bychecking for collisions

Grasping Regular Objects: A Simple Heuristic
Calculate the angles
between the fingertipsof the robotic hand andthe extreme points of
the object
Execute the grasp thathas the maximum
contact angles

Grasping Regular Objects: A Simple Heuristic
Calculate the angles
between the fingertipsof the robotic hand andthe extreme points of
the object
Execute the grasp thathas the maximum
contact angles
92% success rate withunknown objects!

A. Boularias et al. (2014) in Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

Grasping Irregular Objects

The number of grasping actions that need to be simulated andevaluated is very high (thousands) when the objects are irregular
Simulation is too slow (0.1 second per action) for real-time requirements

Grasping Irregular Objects
Idea: Learn to predict the outcome of the simulation

Real-time prediction

Pile of rocks

Predicted success probabilities
using k-Nearest Neighbors

Features of Grasping Actions

Extract all the points in the 3D cloudthat may collide with the robot’s hand(the blue strip in the figures)

Feature matrix: elevations of thepoints of collisionPredicting Success Probabilities of Grasping Action

Learned probabilities,
obtained in 2 seconds
with k-NN

Predicting Success Probabilities of Grasping Action

Learned probabilities,
obtained in 2 seconds
with k-NN

Ground truth, obtained in
200 seconds fromsimulations

Predicting Success Probabilities of Grasping Action

Learned probabilities,
obtained in 2 seconds
with k-NN

Ground truth, obtained in
200 seconds fromsimulations

Predicting Success Probabilities of Grasping Action

Learned probabilities,
obtained in 2 seconds
with k-NN

Ground truth, obtained in
200 seconds fromsimulations

Predicting Success Probabilities of Grasping Action

Learned probabilities,
obtained in 2 seconds
with k-NN

Ground truth, obtained in
200 seconds fromsimulations

Predicting Success Probabilities of Grasping Action

Learned probabilities,
obtained in 2 seconds
with k-NN

Ground truth, obtained in
200 seconds fromsimulations

Best grasping pointaccording to the learned
probabilities≠

Best grasping pointaccording to thesimulator

Best grasping pointaccording to the learned
probabilities≠

Best grasping pointaccording to thesimulator

Fine-tuning in Simulation
Goal: best
grasping point
in simulation

Search, in simulation, for the best action by starting fromthe best action according to the learned probabilities

Fine-tuning in Simulation
Best grasping
point
according to
the simulator?

Simulation is computationally expensive, which
actions should be simulated to find the best one?

Fine-tuning in Simulation
Best grasping
point
according to
the simulator?

Simulation is computationally expensive, which
actions should be simulated to find the best one?

Fine-tuning in Simulation
Best grasping
point
according to
the simulator?

Simulation is computationally expensive, which
actions should be simulated to find the best one?

Fine-tuning in Simulation
Best grasping
point
according to
the simulator?

Simulation is computationally expensive, which
actions should be simulated to find the best one?

Black-box Optimization
Input: position and rotation of the robotic hand
Unknown function f

Output: stability of the simulated grasping action

Bayesian Black-box Optimization
Gaussian Process

courtesy of advancedoptimizationatharvard.wordpress.com !

Compute a posterior probability distribution p on all
possible objective functions f, given all the grasping
actions that have been simulated and evaluated so far.

Bayesian Black-box Optimization
Gaussian Process

courtesy of advancedoptimizationatharvard.wordpress.com !

Compute a posterior probability distribution p on all
possible objective functions f, given all the grasping
actions that have been simulated and evaluated so far.

Bayesian Black-box Optimization
Gaussian Process

courtesy of advancedoptimizationatharvard.wordpress.com !

Compute a posterior probability distribution p on all
possible objective functions f, given all the grasping
actions that have been simulated and evaluated so far.

Bayesian Black-box Optimization
Gaussian Process

How should we choose the next point
to evaluate?

Greedy Entropy Search
Compute a distribution Pmax on the optimal action x:
Pmax (x) = ZP x = arg maxn f (x̃)
x̃2R=f :Rn !Rp(f )⇧x̃2Rn {x} ⇥ f (x)f (x̃) df

Heaviside step function

Greedy Entropy Search
Compute a distribution Pmax on the optimal action x:
Pmax (x) = ZP x = arg maxn f (x̃)
x̃2R=f :Rn !Rp(f )⇧x̃2Rn {x} ⇥ f (x)f (x̃) df

Heaviside step function

The next action x to evaluate is the one that contributes the mostto the entropy of Pmax, i.e. the one with that maximizes

Pmax (x) log Pmax (x)Anytime Optimization
Error = (value of the actual best action)(value of the best action found)

1.2

Grid Search
Entropy Search
Greedy Entropy Search

1

error

0.8

0.6

0.4

0.2

0

5

10

15

20

25

30

35

40

time in seconds

45

50

55

60

Anytime Optimization
Error = (value of the actual best action)(value of the best action found)

1.2

Grid Search
Entropy Search
Greedy Entropy Search

1

error

0.8

0.6

0.4

0.2

0

5

10

15

20

25

30

35

40

time in seconds

45

50

55

60

Anytime Optimization
Error = (value of the actual best action)(value of the best action found)

1.2

Grid Search
Entropy Search
Greedy Entropy Search

1

error

0.8

0.6

0.4

0.2

0

5

10

15

20

25

30

35

40

time in seconds

45

50

55

60

Anytime Optimization
Error = (value of the actual best action)(value of the best action found)

1.2

Grid Search
Entropy Search
Greedy Entropy Search

1

error

0.8

0.6

0.4

0.2

0

5

10

15

20

25

30

35

40

time in seconds

45

50

55

60

Anytime Optimization
Error = (value of the actual best action)(value of the best action found)

1.2

Grid Search
Entropy Search
Greedy Entropy Search

1

error

0.8

0.6

0.4

0.2

0

5

10

15

20

25

30

35

40

time in seconds

45

50

55

60

Anytime Optimization
Error = (value of the actual best action)(value of the best action found)

1.2

Grid Search
Entropy Search
Greedy Entropy Search

1

error

0.8

0.6

0.4

0.2

0

5

10

15

20

25

30

35

40

time in seconds

45

50

55

60

Anytime Optimization
Error = (value of the actual best action)(value of the best action found)

1.2

Grid Search
Entropy Search
Greedy Entropy Search

1

error

0.8

0.6

0.4

0.2

0

5

10

15

20

25

30

35

40

time in seconds

45

50

55

60

Results of Experiments on Grasping Rocks
34% success rate without learning, using only thecenters of the rocks, and without a time budget

A. Boularias et al. (2014) in Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

Results of Experiments on Grasping Rocks
34% success rate without learning, using only thecenters of the rocks, and without a time budget
56% success rate with learning, Bayesian
optimization, and a time budget of 1 second

A. Boularias et al. (2014) in Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

Results of Experiments on Grasping Rocks
34% success rate without learning, using only thecenters of the rocks, and without a time budget
56% success rate with learning, Bayesian
optimization, and a time budget of 1 second
74% success rate with learning, Bayesian
optimization, and a time budget of 5 seconds
A. Boularias et al. (2014) in Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

Outline
1. Overview!
2. Optimal control!
3. Inverse optimal control!
4. Grasping !
5. Manipulation!
6. Navigation

Pushing objects
Pushing and moving objects is needed forgrasping objects in confined environments.

Fixedobstacles

Overview of the integrated system
Get an image ofthe scene froman RGB-D sensor

Overview of the integrated system
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Overview of the integrated system
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Sample a number ofgrasping and pushingactions for each object

Overview of the integrated system
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Overview of the integrated system
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Predict the value of eachsampled action using thevalues of the actions
executed in previous states

Overview of the integrated system
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Execute the action
with the highest Upper

Predict the value of each

Confidence Bound (UCB),

sampled action using theand obtain a binary

values of the actions

reward based on the joint

executed in previous statesangles of the fingers

Overview of the integrated system
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Re-compute the valueof every previous
state (scene) based on

Execute the action
with the highest Upper

Predict the value of each

Confidence Bound (UCB),

sampled action using theand obtain a binary

values of the actions

Re-evaluate the actions

reward based on the joint

executed in previous statessampled in every state

angles of the fingers

the value of the best
action in the next state

Policy Iteration

Overview of the integrated system
Get an image of

Segment the scene

the scene fromimage into objects

an RGB-D sensor

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Re-compute the valueof every previous
state (scene) based onwith the highest Upper

Predict the value of each

Confidence Bound (UCB),

sampled action using theand obtain a binary

values of the actions

Re-evaluate the actions

reward based on the joint

executed in previous statessampled in every state

angles of the fingers

Tune the hyper-parameters

the value of the best

(kernel bandwidths)

action in the next stateby cross-validation

example 4

example 5. example 1 .
test point .

.

example 4

example 2.

.

example 3

Policy Iteration.

Execute the action

Overview of the integrated system
Get an image of

Segment the scene

the scene fromimage into objects

an RGB-D sensor

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Re-compute the value

Repeatof every previous
state (scene) based onwith the highest Upper

Predict the value of each

Confidence Bound (UCB),

sampled action using theand obtain a binary

values of the actions

Re-evaluate the actions

reward based on the joint

executed in previous statessampled in every state

angles of the fingers

Tune the hyper-parameters

the value of the best

(kernel bandwidths)

action in the next stateby cross-validation

example 4

example 5. example 1 .
test point .

.

example 4

example 2.

.

example 3

Policy Iteration.

Execute the action

Segmentation
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

The objects are unknown. We make one assumption:
the shape of an object is overall convex.

For real-time segmentation, we use a cascade ofalgorithms.

Segmentation
1. Detect and remove
the support surface
by using the RANSAC
algorithm (Fischler
and Bolles 1981).	

Segmentation
1. Detect and remove
the support surface
by using the RANSAC
algorithm (Fischler
and Bolles 1981).	

2. Cluster the voxels
into supervoxels witha fast, local, k-means
based on depth andcolor properties(Papon et al. 2013).	

!

Segmentation
1. Detect and remove
the support surface
by using the RANSAC
algorithm (Fischler
and Bolles 1981).	

2. Cluster the voxels
into supervoxels witha fast, local, k-means
based on depth andcolor properties(Papon et al. 2013).	

!

3. Cluster thesupervoxels into
facets (flatcontiguous regions),
using the mean-shiftalgorithm.
	Segmentation
1. Detect and remove
the support surface
by using the RANSAC
algorithm (Fischler
and Bolles 1981).	

4. Cluster the facets
into objects, using
the spectralclustering algorithm.

2. Cluster the voxels
into supervoxels witha fast, local, k-means
based on depth andcolor properties(Papon et al. 2013).	

3. Cluster thesupervoxels into
facets (flatcontiguous regions),
using the mean-shiftalgorithm.

!

	

	

Segmentation❖

The proposed approach works also with natural objects, such as rocks.

Pile of rocks

Segmented image

Extracting Features
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled actionExtracting Features
Grasping features of thepushedobject’s neighbors!
!

+(a) Grasp action (top view)(b) Push action (top view)(c) Grasp action (side view)(d) Push action (side view)(e) Grasp features(f) Push features

Patch of the depth image
in the pushing direction

Action Evaluation
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Predict the value of eachsampled action using thevalues of the actions
executed in previous states

Action Evaluation
The clutter clearing task is formalized as aMarkov Decision Process
State = 3D image of the scene
Action = Parameters of a grasp or a push
Reward = 1 for each successful grasp,!
0 for anything else.

Action Evaluation
The value (expected sum of rewards) of an action a
in a state s is predicted asQ̂⇡ (s, a) =Pt 1

Empirical
value

K
(s
,
a
),
(s,
a)
V̂
(s
)
i
i
⇡
i
i=0
.
Pt 1
K
(s
,
a
),
(s,
a)
i
i
i=0

Current state and action

Similarity measure (Kernel) Data: state (image) and action at time i

Action Evaluation
Grasping or Pushing8><1 if type(a) = type(aj ) ^K (si , ai ), (s, a) =k (si , ai )(s, a)k2  ✏type(a) ,>:0 else .
Similarity measure (kernel)

FeaturesThresholdExploration versus Exploitation
Each action should be executed sufficiently manytimes until a certain confidence on its value is attained
In state st at time t, execute action a that maximizes:Q̂⇡⇤ (st , a) + ↵Predicted Value(for exploitation)

s

Pt 1

2 ln ti=0 K(si , ai ), (st , a)Novelty(for exploration).

Policy Iteration
Get an image ofthe scene froman RGB-D sensor

Segment the scene
image into objects

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Re-compute the valueof every previous
state (scene) based on

Execute the action
with the highest Upper

Predict the value of each

Confidence Bound (UCB),

sampled action using theand obtain a binary

values of the actions

Re-evaluate the actions

reward based on the joint

executed in previous statessampled in every state

angles of the fingers

the value of the best
action in the next state

Policy Iteration

Policy Iteration
Kernel density estimation for learning the transition function
between the states contained in the training data sequence
The learned transition and reward functions are used forevaluating and improving policies.

Policy Iteration
Kernel density estimation for learning the transition function
between the states contained in the training data sequence
The learned transition and reward functions are used forevaluating and improving policies.
Sequence of Data
Push

Grasp

Push

Push

Grasp

Policy Iteration
Kernel density estimation for learning the transition function
between the states contained in the training data sequence
The learned transition and reward functions are used forevaluating and improving policies.
Sequence of Data
Push

Grasp

Push

Similar states!

Push

Grasp

Policy Iteration
Kernel density estimation for learning the transition function
between the states contained in the training data sequence
The learned transition and reward functions are used forevaluating and improving policies.
Sequence of Data
Push

Grasp

Push

Similar states!

Push

Grasp

Policy Iteration
Kernel density estimation for learning the transition function
between the states contained in the training data sequence
The learned transition and reward functions are used forevaluating and improving policies.
Sequence of Data
Push

Grasp

Push

Similar states!

Grasp
Push

Grasp

Bandwidth Selection
Get an image of

Segment the scene

the scene fromimage into objects

an RGB-D sensor

Sample a number ofgrasping and pushingactions for each object

Extract the features ofeach sampled action

Re-compute the valueof every previous
state (scene) based onwith the highest Upper

Predict the value of each

Confidence Bound (UCB),

sampled action using theand obtain a binary

values of the actions

Re-evaluate the actions

reward based on the joint

executed in previous statessampled in every state

angles of the fingers

Tune the hyper-parameters

the value of the best

(kernel bandwidths)

action in the next stateby cross-validation

example 4

example 5. example 1 .
test point .

.

example 4

example 2.

.

example 3

Policy Iteration.

Execute the action

Bandwidth Selection
The kernel’s threshold (range) plays a major role in the
proposed system. It indicates which data points are similar.

example 4 .
example 1 .

example 5 .

test point .
example 2 .

example 3 .

✏example 4 .

Bandwidth Selection
The kernel’s threshold (range) plays a major role in the
proposed system. It indicates which data points are similar.

example 4 .
example 1 .

example 5 .

test point .
example 2 .

example 3 .

✏example 4 .

Bandwidth Selection
The range is automatically tuned by selecting the threshold thatminimizes the Bellman error in the training data,BE(✏) =

1
t2

tX
2 1⇣

t1 i=tri +✏V̂⇡ˆ (si+1 )1

immediate! Predicted value!
rewardof next state✏Q̂⇡ˆ (si , ai )⌘2

Predicted value!
of current state.

Bandwidth Selection
The range is automatically tuned by selecting the threshold thatminimizes the Bellman error in the training data,BE(✏) =

1
t2

tX
2 1⇣

t1 i=tri +✏V̂⇡ˆ (si+1 )1

immediate! Predicted value!
rewardof next statet1✏Q̂⇡ˆ (si , ai )⌘2

Predicted value!
of current statet2
testing data sequence

Push

Push

Grasp

Push

Grasp

.

Bandwidth Selection
The range is automatically tuned by selecting the threshold thatminimizes the Bellman error in the training data,BE(✏) =

1
t2

tX
2 1⇣

t1 i=tri +✏V̂⇡ˆ (si+1 )✏Q̂⇡ˆ (si , ai )1

immediate! Predicted value!
rewardof next state⌘2

Predicted value!
of current statet1

t2
testing data sequence

Push

Push

Grasp

.

Push

Grasp

Learning Curve: Reinforcement Learning V.S. Regression
Moving Average Reward per Step

1

Reinforcement Learning with UCB
Value Regression with UCB
Maximum Average Reward per Step

0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0

20

40

60

80

100

120

time−step
A. Boularias et al. (2015) in Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

Outline
1. Overview!
2. Optimal control!
3. Inverse optimal control!
4. Grasping !
5. Manipulation!
6. Navigation

Grounding Spatial Relations for Robot Navigation
Stay to the right of the car; screen the back of
the building that is behind the car.

J. Oh et al. (2015) in Conference of the Association for the Advancement of Artificial Intelligence (AAAI)

A. Boularias et al. (2015) in IEEE International Conference on Robotics and Automation (ICRA)Grounding Spatial Relations for Robot Navigation
Stay to the right of the car; screen the back of
the building that is behind the car.

?

?

?
Which building? which car??
Grounding: map each noun in the command to an object in the world

Grounding Spatial Relations for Robot Navigation
Stay to the right of the car; screen the back of
the building that is behind the car.

?

?

?
Which building? which car??
Grounding: map each noun in the command to an object in the world

Spatial concepts (such as behind and near) are learned from examples
Bayesian probabilistic model for dealing with object recognition errors

Grounding Spatial Relations for Robot Navigation
predicted
traffic barrel
path planbuilding

traffic barrelbuildingcar
Robot traffic barrel

Environment as perceived by the robot

Planned path

Results: the robot navigated to the correct goal 88% of the time.

Merci !