Predictive Visual Feedback Control with
Eye-in/to-Hand Configuration via
Stabilizing Receding Horizon Approach
Toshiyuki Murao ∗ Hiroyuki Kawai ∗∗ Masayuki Fujita ∗∗∗∗School of Industrial Technology, Advanced Institute of Industrial
Technology, Tokyo, Japan (e-mail: murao-toshiyuki@aiit.ac.jp).
∗∗
College of Engineering, Kanazawa Institute of Technology, Ishikawa,Japan (e-mail: hiroyuki@neptune.kanazawa-it.ac.jp).
∗∗∗
Graduate School of Science and Engineering, Tokyo Institute ofTechnology, Tokyo, Japan (e-mail: fujita@ctrl.titech.ac.jp).
Abstract: This paper investigates vision based robot control via a receding horizon control
strategy for an eye-in/to-hand system, as a predictive visual feedback control. Firstly, thedynamic visual feedback system with the eye-in/to-hand conﬁguration is reconstructed in orderto improve the performance of the estimation. Next, a stabilizing receding horizon control for the
3D dynamic visual feedback system, a highly nonlinear and relatively fast system, is proposed.
The stability of the receding horizon control scheme is guaranteed by using the terminal costderived from an energy function of the visual feedback system. Furthermore, simulation results
are assessed with respect to the stability and the performance.

5z

1. INTRODUCTION
Robotics and intelligent machines need sensory information to behave autonomously in dynamical environments.
Visual information is particularly suited to recognize unknown surroundings. In this sense, vision is one of thehighest sensing modalities that currently exist. The combination of mechanical control with visual information,so-called visual feedback control is important when we
consider a mechanical system working in dynamical environments (Chaumette and Hutchinson [2006, 2007]).
Classically, there have been two typical conﬁgurations tovisual servo control: eye-in-hand conﬁguration and eye-tohand conﬁguration. The ﬁrst conﬁguration has a camera
mounted directly on a robot’s end-eﬀector, and the second
one has a camera ﬁxed in the workspace. Recently, newcamera conﬁgurations combined each classical one havebeen proposed. Flandin et al. [2000] addressed an eyein-hand and an eye-to-hand cooperation approach thateach camera information is partitioned into the positioning
task and the orientation one, respectively. In Lippielloet al. [2007], the occlusion problem was tackled by using
multi eye-in-hand and eye-to-hand cameras. Althoughgood control approaches for each new visual feedback
system are reported in those papers, stability does notaddressed and the manipulator dynamics is negligible.
The authors discussed passivity based control in the 3D
workspace with an eye-in/to-hand conﬁguration as shownin Fig. 1 (Murao et al. [2005]). This conﬁguration consistsof a robot manipulator (a work manipulator) and a camerathat is attached to the end-eﬀector of another robot manipulator (a camera manipulator). While the objective of
this system is obviously to control the work manipulator,
we also control the camera one in order to enlarge theﬁeld of view. It should be noted that this system includes

CameraManipulator

Work
Manipulator

gwz

5h

gzc g
ch

gwh

5c

Cameragco
Target Object

gho
5o

gwc
gwo

5w
World Frame

Fig. 1. Eye-in/to-hand visual feedback system.
the both dynamic visual feedback systems with the eyein-hand conﬁguration and the eye-to-hand one as the special case. However, while it has to be normally treatedboth the estimation problem and the control one in a
typical position based visual feedback control, the control
task has negatively aﬀected the estimation one, in Murao
et al. [2005]. Moreover, the control law is not based on
optimization, the desired control performance cannot be
guaranteed explicitly.
Receding horizon control, also recognized as model predictive control is a well-known control strategy in which
the current control action is computed by solving, a ﬁnite
horizon optimal control problem on-line (Mayne et al.
[2000]). For the receding horizon control, many researchers
have tackled the problem of stability guarantees. On thecontrary, for nonlinear and relatively fast systems such as
in robotics, few implementations of the receding horizon
control have been reported. Jadbabaie et al. [2001] showedthat closed-loop stability is ensured through the use of aterminal cost consisting of a control Lyapunov function.
Moreover, these results were applied to the Caltech Ducted
Fan to perform aggressive maneuvers (Milam et al. [2005]).

Visual feedback, however, is not considered here. Predictive control could be of signiﬁcant beneﬁt when used inconjunction with visual servoing. With the incorporationof visual information, the system could anticipate thetarget’s future position and be waiting there to intercept
it (Hunt and Sanderson [1982]). In Murao et al. [2006],
the authors proposed stabilizing receding horizon control
for the eye-in-hand visual feedback system. However, thissystem has been restricted to eye-in-hand systems.
In this paper, as a predictive visual feedback control, astabilizing receding horizon control is applied to the 3D
visual feedback system with the eye-in/to-hand conﬁguration, a highly nonlinear and relatively fast system. Firstly,
the dynamic visual feedback system with the eye-in/tohand conﬁguration is reconstructed in order to improve
the performance of the estimation. Next, a stabilizing
receding horizon control for the 3D visual feedback systemusing a control Lyapunov function is proposed. Then, thecontrol performance of the simple passivity based control
scheme and the stabilizing receding horizon control scheme
is evaluated through simulation results. The proposed formulation generalizes the results presented in the previous
papers (Murao et al. [2005]: concerning the eye-in/to-hand
conﬁguration) and (Murao et al. [2006]: concerning thepredictive visual feedback control).
Throughout this paper, we use the notation eξ̂θab ∈ R3×3
to represent the change of the principle axes of a frame Σb
relative to a frame Σa . ξab ∈ R3 speciﬁes the direction ofrotation and θab ∈ R is the angle of rotation. For simplicity
we use ξ̂θab to denote ξˆab θab . The notation ‘∧’ (wedge)ˆ = ξ ×θis the skew-symmetric operator such that ξθ
for the vector cross-product × and any vector θ ∈ R3 .
The notation ‘∨’ (vee) denotes the inverse operator to ‘∧’,i.e., so(3) → R3 . Recall that a skew-symmetric matrix
corresponds to an axis of rotation (via the mapping a →â). We use the 4 × 4 matrix


ξ̂θab
e
p
ab
gab =
(1)
0
1
as the homogeneous representation of gab = (pab , eξ̂θab ) ∈SE(3) describing the conﬁguration of a frame Σb relative
to a frame Σa . The adjoint transformation associated withgab is denoted by Ad(gab ) (Murray et al. [1994]).
2. DYNAMIC VISUAL FEEDBACK SYSTEM WITH
EYE-IN/TO-HAND CONFIGURATIONThis section mainly reviews our previous works (Murao
et al. [2005], Fujita et al. [2007]) via the passivity based
visual feedback control with the eye-in/to-hand conﬁguration. Additionally, a modiﬁed camera control error system
and a modiﬁed hand one are proposed in order to improve
the performance of the estimation from the practical pointof view. An energy function and a stabilizing control law,which play an important role for a predictive visual feedback control, are derived.

robot’s end-eﬀector and as depicted in Fig. 1, where thecoordinate frames Σw , Σh , Σz , Σc , and Σo represent theworld frame, the hand (end-eﬀector of the work manipulator) frame, and the base frame of the camera manipulator, the camera (end-eﬀector of the camera manipulator)
frame, and the object frame, respectively. Then, the relative rigid body motion from Σc to Σo can be represented
by gco . Similarly, the rigid body motions gwc , gwz , gwh and
gwo , and the relative rigid body motions gch , gzc and gho
are represented, respectively, as shown in Fig. 1.
The objective of visual feedback control is to bring theactual relative rigid body motions gco and gho to givenreferences gcd and ghd , respectively. The references gcd andghd are assumed to be constant throughout this paper. We
deﬁne the camera control error gec and the hand control
error geh as follows:
−1
gco ,
(2)
gec = gcd
−1
gho .
geh = ghd

(3)

ξ̂θ

Using the notation eR (e ), the vector of the camera
control error and the hand control error are given by ec :=[pTec eTR (eξ̂θec )]T and eh := [pTeh eTR (eξ̂θeh )]T , respectively.
Note that ei = 0 iﬀ pei = 0 and eξ̂θei = I3 (i ∈ c, h).
Firstly, we consider the relative rigid body motion gco inorder to achieve the control objective. The relative rigid
body motion from Σc to Σo can be led by using thecomposition rule for rigid body transformations (Murray
et al. [1994], Chap. 2, pp. 37, eq. (2.24)) as follows:−1
gco = gwc
gwo .

(4)

The relative rigid body motion involves the velocity of eachrigid body. To this aid, let us consider the velocity of a rigid
body as described in Murray et al. [1994]. We deﬁne thebody velocity of the camera relative to the world frameb
T
T TΣw as Vwc
= [vwcωwc] , where vwc and ωwc represent thevelocity of the origin and the angular velocity from Σw toΣc , respectively (Murray et al. [1994] Chap. 2, eq. (2.55)).
Diﬀerentiating (4) with respect to time, the body velocityof the relative rigid body motion gco can be written as
follows (See Fujita et al. [2007]):
b
b
b
−1 V
Vco
= −Ad(gco) wc + Vwo(5)bis the body velocity of the target object relative
where Vwo
to Σw .

The visual feedback control task requires information of
the relative rigid body motion gco . Since the measurable
information is only the image information f (gco ) in thevisual feedback system, we consider a nonlinear observer
in order to estimate the relative rigid body motion gco
from the image information f (gco ).
Firstly, using the basic representation (5), we choosebestimates ḡco and V̄coof the relative rigid body motion
and velocity, respectively as
b
b
−1 V
V̄co
= −Ad(ḡco) wc + ue .

(6)

2.1 Basic Representation for Visual Feedback System andEstimation Error System

The new input ue is to be determined in order to drive thebestimated values ḡco and V̄coto their actual values.

The visual feedback system considered in this paper has arobot manipulator and a camera mounted on the anotherIn order to establish the estimation error system, we deﬁne
the estimation error between the estimated value ḡco andthe actual relative rigid body motion gco as−1gee = ḡco
gco .

(7)

Using the notation eR (eξ̂θ ), the vector of the estimation
error is deﬁned as ee := [pTee eTR (eξ̂θee )]T . Therefore, if thevector of the estimation error is equal to zero, then theestimated relative rigid body motion ḡco equals the actual
relative rigid body motion gco .
Suppose the attitude estimation error θee is small enoughthat we can let eξ̂θee  I + sk(eξ̂θee ). Therefore, using aﬁrst-order Taylor expansion approximation, the estimation
error vector ee can be obtained from image informationf (gco ) and the estimated value of the relative rigid body
motion ḡco as follows (Fujita et al. [2007]):ee = J † (ḡco )(f − f¯),(8)¯
where f is the estimated value of image information. In thesame way as the basic representation (5), the estimation
error system can be represented bybb−1 ue + V
Vee
= −Ad(gee
wo .
)

(9)

2.2 Camera and Hand Control Error Systems
In this subsection, let us consider the dual of the estimation error system, which we call the control error system, inorder to establish the visual feedback system. In previous
work (Murao et al. [2005]), we deﬁned the camera control−1
error as gec = gcd
ḡco , which represent the error between
the estimated value ḡco and the reference of the relative
rigid body motion gcd . However, the estimation input
ue has aﬀected directly the camera control error system,
because the camera control error was deﬁned using theestimated value ḡco . This has deteriorated the performanceof the estimation. In this paper, we reconstruct the camera
control error system using the new camera control error−1
gec = gcdgco (2) in order to remove the above negativeeﬀect. Thus, we propose the way of deriving gec (2) fromthe estimation error vector ee and the estimated value ḡco ,not using nonmeasurable value gco .
Using gee , the camera control error can be transformed as
−1
−1
−1
−1
gec = gcd
gco = gcd
ḡco ḡcogco = gcd
ḡco gee .
(10)
In Equation (10), gcd and ḡco are known information.
While the estimation error vector ee can be obtained as
Equation (8), the estimation error matrix gee cannot bedirectly obtained, because gee is deﬁned using nonmeasurable value gco as Equation (7). Therefore, we consider theway of deriving gee from ee .

Because of the deﬁnition of the estimation error vector ee ,i.e., ee := [pTee eTR (eξ̂θee )]T , the position estimation error pee
can be derived directly from ee . Concerning to the rotation
estimation error eξ̂θee , if we assume that the region of theattitude estimation error is − π2 ≤ θee ≤ π2 , then ξθee canbe derived as follows:
sin−1 eR (eξ̂θee )ξθee =eR (eξ̂θee ).
(11)eR (eξ̂θee )Hence, gee can be derived from ee through ξθee using
Equation (1). Here, it should be noted that the assumption− π2 ≤ θee ≤ π2 will not be a new constraint, because wehave already set the assumption that the attitude estimation error θee is small enough in deriving the estimation

error vector ee (in Subsec.2.1). Therefore, it is possible
to derive the new camera control error gec using known
information gcd , ḡco and ee .
In the same way as the estimation error system (9), thecamera control error system can be represented asbbb−1 Ad −1 V
Vec
= −Ad(gec(12))(g ) wc + Vwo .
cd

Similar to the camera control error system, we derive the−1hand control error system, using geh = ghd
gho (3), instead−1of geh = ghd ḡho (Murao et al. [2005]). Using gee , the handcontrol error can be transformed as
−1
−1 −1
−1 −1
geh = ghd
gho = ghd
gch gco = ghd
gch ḡco gee .
(13)−1
gwz gzh can be obtained directly, because
Here gch = gwc
the rigid body motions gwc , gzh and gwz are known by theangles of the manipulators and the structure of the system.
According to Equation (13), it is possible to derive the new
hand control error geh . Moreover, the hand control error
system can be represented as
b
b
b
Veh
= −Ad(g−1 ) Ad(g−1 ) Vwh
+ Vwo
,
(14)
eh

hd

b
where Vwh
is the body velocity of the hand relative to Σw .

2.3 Passivity based Dynamic Visual Feedback System with
Eye-in/to-Hand Configuration
The manipulator dynamics of the camera manipulator andthe work one (we call the hand one, too) can be writtenas
Mi (qi )q̈i + Ci (qi , q̇i )q̇i + gi (qi ) = τi + τid , i ∈ c, h(15)where Mi ∈ Rni×ni is the inertia matrix, Ci ∈ Rni×ni is
the Coriolis matrix, gi ∈ Rni is the gravity vector, and qi ,q̇i and q̈i are the joint angle, velocity and acceleration,
respectively. τi is the vector of the input torque, andτid represents a disturbance input. Here, due to space
limitations, the subscripts c and h are used in the caseof the camera manipulator and the hand one, respectively.
Since the manipulator dynamics is considered, the camerabbody velocity and the hand one are given by Vwc=b
Jcb (qc )q̇c and Vwh = Jhb (qh )q̇h , respectively, where Jib (qi )is the body manipulator Jacobian (Murray et al. [1994]).
Next, we propose the control law for the manipulator asτi = Mi (qi )q̈id + Ci (qi , q̇i )q̇id + gi (qi )T(qi )AdT(g−1 ) ei + uξi ,+Jibidi ∈ c, h (16)where q̇id and q̈id represent the desired joint velocity andacceleration, respectively. The new input uξi is to be
determined in order to achieve the control objective.
Let us deﬁne the error vectors with respect to the joint
velocities of the camera manipulator and the hand oneas ξc := q̇c − q̇cd and ξh := q̇h − q̇hd , respectively.
Moreover, we design the references of the joint velocities††
as q̇cd := Jcb(qc )ucd and q̇hd := Jhb(qh )uhd where uid isthe desired body velocity which will be obtained from thebbvisual feedback system. Thus, Vwc
in (12) and Vwh
in (14)should be replaced by ucd and uhd , respectively.
Using (9), (12) and (14)–(16), the eye-in/to-hand visualfeedback system with the manipulator dynamics (we callthe dynamic visual feedback system) can be derived asfollows:⎤⎤ ⎡ −M −1 C ξ + M −1 J T AdT
−1 eh
h h
hb
h
h
ξ˙h
)
(ghd
⎥
T
⎢ ξ˙c ⎥ ⎢−Mc−1 Cc ξc + Mc−1 Jcb
AdT(g−1 ) ec ⎥
⎢
⎥ ⎢
cd
⎥
⎢
⎢Vb ⎥ = ⎢⎥−Ad(g−1 ) Jhb ξh⎢ eh ⎥ ⎢⎥ho⎣Vb ⎦ ⎣⎦ec
−1 Jcb ξc−Ad(gco
)
b
Vee
0
⎤ ⎡
⎡ −1
⎤
0
0
0
Mh 0
Mh−1 0 0⎥⎢ 0 Mc−1−1000 ⎥ ⎢ 0 M 0⎥⎢c⎥
0 ⎥u +⎢
00−Ad(g−1 ) 0+⎢0
0 I ⎥w
⎥ ⎢
⎢
eh
⎣
⎦
⎣ 0
−1
0
0 −Ad(gec
0
0
0 I⎦
)
0
0 I
0
0
0
0 −Ad(g−1 )
⎡

ee

(17)
where u := [uTξh uTξc (Ad(g−1 ) uhd )T (Ad(g−1 ) ucd )T uTe ]T .
hd
cd
We deﬁne the state and the disturbance of the dynamic
visual feedback system as x := [ξhT ξcT eTh eTc eTe ]T and
T
T
b T T
τcd
(Vwo
) ] , respectively.
w := [τhdbRemark 1. If the camera velocity Vwc= 0, the desired
camera velocity ucd = 0, gwz = I, and, the camera control
error ec and the camera manipulator are not considered,then the eye-in/to-hand dynamic visual feedback system(17) represents the eye-to-hand one. On the other hand,b
if the hand velocity Vwh, the desired hand velocity uhd ,the hand control error eh and the work manipulator are
not considered, then the eye-in/to-hand dynamic visualfeedback system (17) represents the eye-in-hand one. Thus,
the dynamic visual feedback systems with the eye-in-hand
conﬁguration and the eye-to-hand one are regarded as the
special cases of the system (17). This is one of main meritsof this conﬁguration.
2.4 Energy Function and Stabilizing Control Law
Before constructing the dynamic visual feedback control
law, we derive an important lemma.
Lemma 1. If w = 0, then the dynamic visual feedbackTsystem (17) satisﬁes 0 uT νdt ≥ −β, ∀T > 0 whereν := N x, N := diag{I, I, −I, −I, −I} and β is a positive
scalar.
Due to space limitations, the proof is only sketched. By
using the following energy function, the proof can becompleted.
11
V (x) = ξhT Mh (qh )ξh + ξcT Mc (qc )ξc2
2+E(geh ) + E(gec ) + E(gee ), (18)where E(gab ) := 12 pab 2 +φ(eξ̂θab ) and φ(eξ̂θab ) := 12 tr(I−eξ̂θab ) is the error function of the rotation matrix.
We now propose the following control input for the interconnected system:u = −Kν := uk .

Nc
eh

KNh
KNc
Kh

Ad(ghd)

uNh

Nh

uNc

Nc

uhd

eh

ue

ee

ucd

ec

uNh

KNh

uNc

KNc
Kh

Ad(ghd)uhd

Ad(e^ )
NGeh

ee
ec

Ke
Kc

Ad(gcd)ueKeAd(e^ )NGec

Kc

Ad(gcd)ucd

Fig. 2. Block diagram of the control law. (Left: proposed
method, Right: previous one (Murao et al. [2005]))
The block diagrams of the control law in the case of
the proposed method and the previous one (Murao et al.
[2005]) are shown in the left side and the right one of Fig. 2,
respectively. In Murao et al. [2005], the camera control
error ec and the hand one eh have interfered with thecontrol input for the estimation ue = Ke (ee −AdT(eξ̂θeh ) eh −AdT(eξ̂θec ) ec ). In the sequel, the presence of ec or eh has
negatively aﬀected the estimation of the relative rigid
body motion ḡco . On the other hand, with the proposed
new visual feedback control law (19), each feedback input
is separated by each error vector, i.e., uξh = −Kξh ξh ,uξc = −Kξc ξc , uhd = Ad(ghd ) Kh eh , ucd = Ad(gcd ) Kc ec andue = Ke ee . Thus, the proposed control law can overcome
the problem for the above undesirable inﬂuence.
3. PREDICTIVE VISUAL FEEDBACK CONTROLThe objective of this section is to propose a predictive
visual feedback control based on optimal control theory.
A camera can provide more information than the current
derivation from a nominal position at the sample instant.
This property can be exploited to predict the target’s
future position and improve the control performance. As apredictive visual feedback control, we propose a stabilizing
receding horizon control based on optimization in thispaper.
3.1 Control Lyapunov Function
In this section, the ﬁnite horizon optimal control problem(FHOCP) for the visual feedback system (17) is considered. The FHOCP for the visual feedback system (17) attime t consists of the minimization with respect to the
input u(τ, x(τ )), τ ∈ [t, t+T ], of the following cost function
t+T

J(x0 , u, T )=l(x(τ ), u(τ ))dτ +F (x(t + T ))

(21)

t

(19)

K := diag{Kξh , Kξc , Kh , Kc , Ke } > 0
Theorem 2. If w = 0, then the equilibrium point x = 0 for
the closed-loop system (17) and (19) is asymptotic stable.
Proof Diﬀerentiating (18) with respect to time yields andusing the control input (19), it can be obtained thatV̇ = xT N T u = −xT N T KN x.
This completes the proof.

Nh

(20)
(Q.E.D)l(x(t), u(t))=qξh (t) ξh (t) 2 +qξc (t) ξc (t) 2 +Eqh (geh (t))+ Eqc (gec (t))+Eqe (gee (t))+uT (t)R(t)u(t)(22)
F (x)=ρV (x)(23)

qξh (t)≥0, qξc (t)≥0, qpi (t)≥0, qRi (t)≥0, ρ > 0,
where R(t) is a positive diagonal matrix, and Eqi (gei (t)) :=qpi (t) pei (t) 2 +qRi (t)φ(eξ̂θei (t) ) (i ∈ h, c, e), with the statex(t) = x0 . The speciality of the cost function (21)–(23) isthat the terminal cost is derived from an energy function

of the visual feedback system. Furthermore, the rotation
error related part of the stage cost is derived from the error
function φ(eξ̂θ ) instead of the commonly used quadratic
form eR (eξ̂θ ) 2 . For a given initial condition x0 , we denote
this solution of the FHOCP as u∗ (τ, x(τ )), τ ∈ [t, t + T ].
In receding horizon control, at each sampling time δ, theresulting feedback control at state x0 is obtained by solving
the FHOCP and settinguRH := u∗ (δ, x0 ).

(24)

The following lemma concerning a control Lyapunov function is important to prove a stabilizing receding horizon
control. The deﬁnition for a control Lyapunov function
S(x) is given by
inf Ṡ(x) + l(x, u) ≤ 0,(25)
uwhere l(x, u) is a positive deﬁnite function (Jadbabaie
et al. [2001]).
Lemma 3. Suppose that w = 0, θeh ≤ π2 , θec ≤ π2 ,θee ≤ π2 and the design parameter ρ satisﬁesρ2 I≥4QR,(26)

where Q := diag{qξh Inh , qξc Inc , qph I3 , qRh I3 , qpc I3 , qRc I3 ,qpe I3 , qRe I3 }. Then, the energy function ρV (x) of thevisual feedback system (17) can be regarded as a control
Lyapunov function.
Proof Using Equation (20), which is the time derivativeof V along the trajectory of the system (17), the positivedeﬁnite function l(x(t), u(t)) (22) and the stabilizing control law uk (19) with K = ρ2 R−1 for the system, Equation(25) can be transformed into
inf [Ṡ(x) + l(x, u)]
u

T 

ρ
ρ
= inf u + R−1 N x R u + R−1 N x + qξh ξh 2u22ρ2 T T −1
2− x N R Nx+qξc ξc +Eqh (geh )+Eqc (gec )+Eqe (gee )4
ρ2≤ − xT R−1 x + qξh ξh 2 + qξc ξc 2 + qph peh 24+qReh eR (eξ̂θeh ) 2 + qpc pec 2 + qRec eR (eξ̂θec ) 2+qpe pee 2 + qRee eR (eξ̂θee ) 2

 2
ρ −1
T
R − Q x,
= −x
4

(27)

where we have used the fact that φ(eξ̂θ )≤ eR (eξ̂θ ) 2 forall θ ≤ π2 . Therefore, the condition inf u [Ṡ(x) + l(x, u)]≤0(Q.E.D)will be satisﬁed, if the assumption ρ2 I≥4QR.
Lemma 3 shows the energy function ρV (x) of the visualfeedback system (17) can be regarded as a control Lyapunov function in the case of ρ2 I≥4QR.
3.2 Stabilizing Receding Horizon Control for the 3D
Eye-in/to-Hand Visual Feedback System
Suppose that the terminal cost is the control Lyapunov
function ρV (x), the following theorem concerning thestability of the receding horizon control holds.
Theorem 4. Consider the cost function (21)–(23) for thevisual feedback system (17). Suppose that w = 0,θeh ≤ π2 , θec ≤ π2 , θee ≤ π2 , and ρ2 I≥4QR, then thereceding horizon control for the visual feedback system
is asymptotically stabilizing.
Proof Our goal is to prove that J(x∗ (t), uRH , T ), whichis the cost-to-go applying the receding optimal control
uRH , will qualify as a Lyapunov function for the closed
loop system. Construct the following suboptimal control
strategy for the time interval [t + δ, t + T + δ] ∗u (τ )τ ∈ [t + δ, t + T ]
ρ −1
(28)
ũ =
uk (τ ) = − R N x τ ∈ [t + T, t + T + δ]2
where uk is the stabilizing control law (19) with K = ρ2 R−1
for the visual feedback system. The associated cost isJ(x∗ (t + δ), ũ, T )= J(x(t), u∗ , T ) + ρ[V (x(t + T + δ)) − V (x∗ (t + T ))]t+T +δ

t+δ
∗

−

∗

l(x∗ (τ + T ), uk )dτ, (29)l(x (τ ), u )dτ +
t
∗

t+T

where x is the optimal state trajectory. This cost, which
is an upper bound for J(x∗ (t + δ), u∗ , T ), satisﬁes
J(x∗ (t + δ), u∗ , T ) − J(x∗ (t), u∗ , T )≤ ρ[V (x(t + T + δ)) − V (x∗ (t + T ))]t+T +δ

t+δ
∗

−

∗

l(x∗ (τ + T ), uk )dτ.(30)l(x (τ ), u )dτ +tt+TUsing the positive deﬁnite function l(x(t), u(t)) (22) andthe stabilizing control law uk (19) for the system, anddividing both sides by δ and taking the limit as δ → 0,
Equation (30) can be transformed intoJ(x∗ (t + δ), u∗ , T ) − J(x∗ (t), u∗ , T )
lim
δ→0
 2 δ

ρ −1
∗T
R − Q x∗ (t + T )≤ −x (t + T )4−x∗T (t)Qx∗ (t) − u∗T Ru∗ .
(31)
Considering that the control input during ﬁrst δ is
uRH = u∗ , by the assumption ρ2 I≥4QR, the derivative ofJ(x∗ (t), uRH , T ) is negative deﬁnite. Therefore, we haveshown that J(x∗ (t), uRH , T ) qualiﬁes as a Lyapunov function and asymptotic stability is guaranteed.
(Q.E.D)
Theorem 4 guarantees the stability of the receding horizon
control using a control Lyapunov function for the 3D
eye-in/to-hand visual feedback system (17) which is ahighly nonlinear and relatively fast system. Since thestabilizing receding horizon control design is based on
optimal control theory, the control performance should be
improved compared to the simple passivity based control
uk (19), under the condition of adequate gain assignment
in the cost function. It should be noted that the error
function φ(eξ̂θ ) of the rotation matrix can be directly usedin the stage cost (22). Compared with the previous work
(Murao et al. [2006]), the assumption ρ2 I≥4QR becomesvery simply, and it is quite easy to set the value of ρ, byvirtue of the fact that N becomes a block diagonal matrix
in the case of the new dynamic visual feedback system(17). Moreover, the main advantage is that the proposed
control law can be applied to not only the eye-in-handvisual feedback system but also the eye-to-hand one. This
allows us to extend the technological application area.

0.1
0

0

1

2

3

N^G ehz [rad]

0.5

0

1

0

1

2

3

2

3

0
-1

0.1

0

1

2

3

0

1

2

3

0

1

2

3

0

-0.1
0.2
0

N^G ecz [rad]

-0.3

0

-0.1
pecy [m]pehy [m]0.10pecx [m]pehx [m]-0.3

Table 1. Values of the integral cost.

0.1

time [s]-0.6time [s]Fig. 3. Hand control error (Left) and camera one (Right).
(Solid: stabilizing receding control law, Dashed: simple passivity based one)
4. SIMULATIONSControl Scheme
Passivity based Control
Receding Horizon Control (T = 0.02 [s], ρ = 1)
Receding Horizon Control (T = 0.02 [s], ρ = 1.2)
Receding Horizon Control (T = 0.02 [s], ρ = 1.5)
Receding Horizon Control (T = 0.005 [s], ρ = 1)
Receding Horizon Control (T = 0.04 [s], ρ = 1)

cost
8014
701
878
1194
710
691

to improve the performance of the estimation. It is shownthat the stability of the receding horizon control scheme
is guaranteed by using the terminal cost derived from theenergy function of the visual feedback system. Simulation
results are presented to verify the control performance ofthe stabilizing receding horizon control law.
REFERENCES

In this section, we present simulation results for thepredictive visual feedback control, compared with thesimple passivity based control law uk (19). The simulation
results on two 2DOF manipulators are shown in order tounderstand our proposed method simply, though it is valid
for 3D dynamic visual feedback systems. The weights ofthe cost function (21)–(23) and the controller parameters
for the simple passivity based control law (19) are selected
in order not to exceed the limit of the input torques for the
manipulators. To solve the real time optimization problem,the software C/GMRES (Ohtsuka [2004]) is utilized. The
control input with the receding horizon control is updated
every 1 [ms]. It must be calculated by the receding horizon
controller within that period. The horizon was selected as
T = 0.02 [s].
The simulation results are presented in Fig. 3. The handcontrol error eh and the camera one ec are shown in
the left side and the right one, respectively. The solidlines denote the errors applying the proposed stabilizing
receding horizon control, and the dashed lines denote thosefor the passivity based control law uk (19). In Fig. 3,
the asymptotic stability can be conﬁrmed by steady state
performance. Moreover, the rise time applying the receding
horizon control is shorter than that for the passivity based
control.
The performance for parameter value T and ρ is comparedin terms of the integral cost in Table 1. Since the costof the stabilizing receding horizon method is smaller thanthe passivity based control method under conditions of theadequate cost function, it can be easily veriﬁed that thecontrol performance is improved. With increasing weightof the terminal cost from ρ = 1 to ρ = 1.5 the cost
increases, too. With higher terminal cost the state value
is reduced more strictly, using a large control input. As
the horizon length increases from T = 0.005 to T = 0.04,
the cost is reduced. In the case of T = 0.1, the calculation
cannot be completed within one sampling interval, due tolimited computing power.
5. CONCLUSIONSThis paper proposes a stabilizing receding horizon control for a reconstructed 3D eye-in/to-hand visual feedback system, which is a highly nonlinear and relatively
fast system, as a predictive visual feedback control. The
dynamic visual feedback system is reconstructed in order

F. Chaumette and S. Hutchinson. Visual Servo Control,
Part I: Basic Approaches. IEEE Robotics & Automation
Magazine, 13:82–90, 2006.
F. Chaumette and S. Hutchinson. Visual Servo Control,
Part II: Advanced Approaches. IEEE Robotics & Automation Magazine, 14:109–117, 2007.
G. Flandin, F. Chaumette and E. Marchan. Eye-inHand/Eye-to-Hand Cooperation for Visual Servoing.
Proc. of the 2000 IEEE International Conference on
Robotics and Automation, pages 2741–2746, Apr. 2000.
M. Fujita, H. Kawai and M. W. Spong. Passivity-based
Dynamic Visual Feedback Control for Three Dimensional Target Tracking: Stability and L2 -gain Performance Analysis. IEEE Trans. Control Systems Technology, 15:40–52, 2007.
A.E. Hunt and A.C. Sanderson. Vision-based Predictive
Robotic Tracking of a Moving Target. Technical Report,
Carnegie Mellon University, 1982.
A. Jadbabaie, J. Yu and J. Hauser. Unconstrained
Receding-Horizon Control of Nonlinear Systems. IEEE
Trans. Automatic Control, 46:776–783, 2001.
V. Lippiello, B. Siciliano and L. Villani. Position-based
Visual Servoing in Industrial Multirobot Cells using a
Hybrid Camera Conﬁguration. IEEE Trans. Robotics,
23:73–86, 2007.
D.Q. Mayne, J.B. Rawlings, C.V. Rao and P.O.M.
Scokaert. Constrained Model Predictive Control: Stability and Optimality. Automatica, 36:789–814, 2000.
M.B. Milam, R. Franz, J.E. Hauser and R.M. Murray.
Receding Horizon Control of Vectored Thrust FlightExperiment. IEE Proc. Control Theory & Applications,
152:340–348, 2005.
T. Murao, H. Kawai and M. Fujita. Passivity-based Control of Visual Feedback Systems with Dynamic Movable
Camera Conﬁguration. Proc. of the 44th IEEE Conference on Decision and Control and European Control
Conference, pages 5360–5365, Dec. 2005.
T. Murao, T. Yamada and M. Fujita. Predictive VisualFeedback Control with Eye-in-Hand System via Stabilizing Receding Horizon Approach. Proc. of the 45th
IEEE Conference on Decision and Control, pages 1758–
1763, Dec. 2006.
R. Murray, Z. Li and S. S. Sastry. A Mathematical
Introduction to Robotic Manipulation, CRC Press, 1994.
T. Ohtsuka. A Continuation/GMRES Method for Fast
Computation of Nonlinear Receding Horizon Control.
Automatica, 40:563–574, 2004.