Electronics and Communications in Japan, Vol. 94, No. 8, 2011
Translated from Denki Gakkai Ronbunshi, Vol. 129-C, No. 4, April 2009, pp. 630–638

Stabilizing Predictive Visual Feedback Control for Fixed Camera Systems
TOSHIYUKI MURAO,1 HIROYUKI KAWAI,2 and MASAYUKI FUJITA3
1

Advanced Institute of Industrial Technology, Japan
2
Kanazawa Institute of Technology, Japan
3
Tokyo Institute of Technology, Japanas position-based visual feedback control, which is normally treated as both an estimation problem and a control
problem. However, the control task has a negative effect on
the estimation of relative rigid body motion in Ref. 6.
Furthermore, the desired control performance cannot always be guaranteed explicitly, since the control method ofRef. 6 is not based on optimization.
Receding horizon control, also recognized as model
predictive control, is a well-known control strategy inwhich the current control action is computed by solving afinite horizon optimal control problem on-line [7]. A large
number of industrial applications using model predictive
control can be found in chemical industries where theprocesses have relatively slow dynamics. For receding horizon control, many researchers have addressed the problem
of stability guarantees. In contrast, for nonlinear and relatively fast systems such as in robotics, few implementationsof receding horizon control have been reported. Jadbabaie
and colleagues [8] showed that closed-loop stability is
assured by the use of a terminal cost consisting of a control
Lyapunov function. Furthermore, this result was applied to
the Caltech Ducted Fan to perform aggressive maneuvers
in Ref. 9. In Ref. 10, high effectiveness of stabilizing
receding horizon control for a direct drive manipulator
system is reported on the basis of experimental studies.
However, visual feedback is not considered here.
Predictive control could be of significant benefit
when used in conjunction with visual servoing. With theincorporation of visual information, the system could anticipate the target’s future position and be waiting there tointercept it [11]. Ginhoux and colleagues [12] proposed arepetitive generalized predictive controller to cancel respiratory motions in robotized surgery. Lange and Hirzinger[13] presented a predictive visual tracking method using
geometrical information to compute a desired path for
high-speed industrial robots. Although good predictive
control approaches using visual information are reported inthose papers, stability is not addressed. In Ref. 14, the

SUMMARY
This paper investigates vision-based robot control via
a receding horizon control strategy for fixed camera systems, as stabilizing predictive visual feedback control.
First, a visual motion robot error system with a fixed camera
configuration is reconstructed in order to improve estimation performance. Next, stabilizing receding horizon control for three-dimensional visual feedback systems, which
are highly nonlinear and relatively fast systems, is proposed. The stability of the receding horizon control scheme
is guaranteed by using a terminal cost derived from anenergy function of the visual motion robot error system.
Furthermore, simulation and actual nonlinear experimental
results are assessed with respect to stability and performance. © 2011 Wiley Periodicals, Inc. Electron Comm Jpn,
94(8): 1–11, 2011; Published online in Wiley Online Library (wileyonlinelibrary.com). DOI 10.1002/ecj.10357

Key words: visual feedback control; receding horizon control; passivity; stability; control Lyapunov function.

1. Introduction
Visual feedback control is a very flexible and effective method for the autonomous performance of various
tasks for robotic systems [1, 2]. It is currently employed ina wide range of applications, including not only the fieldsof robotics and factory automation, but also automatic
guidance of surgical instruments [3], control of an ultrasound probe held by a medical robot [4], injection of
biological cells [5], and others. The authors have discussed
passivity-based control for a moving target object in athree-dimensional (3D) workspace with a fixed camera
configuration [6]. This control approach can be classified© 2011 Wiley Periodicals, Inc.
1authors proposed stabilizing receding horizon control foran eye-in-hand planar visual feedback system. However,the visual feedback system proposed in Ref. 14 is restricted
to planar manipulators, and this method can treat only adesired position problem.
In this paper, we propose stabilizing receding horizon
control for 3D visual feedback systems with a fixed camera
configuration, which are highly nonlinear and relatively
fast systems, as a method of predictive visual feedback
control. First, a visual motion robot error system is reconstructed in order to improve estimation performance. Next,
stabilizing receding horizon control for 3D visual feedback
system using a control Lyapunov function is proposed.
Since the proposed stabilizing visual feedback control law
for the systems is based on receding horizon control theory,
control performance should be improved relative to that ofa nonoptimal control law. Compared with previous work[14], the proposed control law can treat not only position
but also orientation, so that the range of possible application
areas will undoubtedly be increased. The main idea is theuse of a terminal cost derived from an energy function of
the visual robot motion error system. Then, the control
performance of the proposed control scheme is evaluatedthrough simulation and experimental results.
The paper is organized as follows. In Section 2, wederive a visual motion robot error system for 3D fixedcamera systems and discuss an energy function and a stabilizing control law. In Section 3, stabilizing receding horizon control for fixed camera systems using a control
Lyapunov function is proposed. Section 4 describes simulation and experimental results that show the advantages ofthe proposed control laws, followed by conclusions in
Section 5.

Fig. 1. Visual feedback system with a fixed camera
configuration.

a skew-symmetric matrix corresponds to an axis of rotation(via the mapping a $ a^). We use the 4 × 4 matrix
(1)
as
the
homogeneous
representationof^gij = (pij, eξθij) ∈ SE(3) describing the configuration of aframe Σj relative to a frame Σi. For example, the positionand orientation in the object frame Σo relative to the handframe Σh are expressed by gho.
The control objective is to bring the actual relative
rigid body motion gho into agreement with a given constantreference one gd. We now define a control error as follows:(2)
which represents the error between the relative rigid body
motion gho and the reference one gd. We next define the error^^^
vector of the rotation matrix eξθei as eR(eξθei) := sk(eξθei)∨,
^ξθ
^ξθ
^
where sk(e ei) denotes 1 / 2(e ei − e−ξθei). Using this notation, the vector of the control error is given by^
ec := [pTeceR(eξθec)T]T.
The relative r igid body motion gho can be der ived byusing the composition r ule for r igid body tr ansfor mations[15] as follows:

2. Visual Motion Robot Error System andStabilizing Control Law
2.1 Vision camera model and estimation error
system
A visual feedback system with a fixed camera configuration uses four coordinate frames, which consist of aworld frame Σw, a hand (an end-effector of a manipulator)
frame Σh, a camera frame Σc, and an object frame Σo, asshown in Fig. 1. Throughout this paper, we use the notation^eξθij ∈ R3×3 to represent the rotation matrix of a frame Σjrelative to a frame Σi. ξij ∈ R3 specifies the direction ofrotation and θij ∈ R is the angle of rotation. For simplicity^ θ to denote ξ^ θ . The notation “∧” (wedge) is thewe use ξij
ij ij
skew-symmetric operator such that a^b = a × b for the vector
cross-product × and any vector a, b ∈ R3, that is, a^ is a 3 ×
3 skew-symmetric matrix. The notation “∨” (vee) denotes
the inverse operator to “∧,” that is, so(3) → R3. Recall that(3)
Here, gch = g−wc1gwh can be obtained directly, because therigid body motions gwc and gwh are known from the structureof the system and the angles of the manipulator. We assume
that the relative rigid body motion gch can be measured
exactly in this paper. In contrast, the relative rigid body
motion gco can be derived as(4)
The relative rigid body motion involves the velocity of eachrigid body. We define the body velocity of the camera

2relative to the world frame Σw as Vbwc = [vTwcωTwc]T, where vwc
and ωwc represent the velocity of the origin and the angularvelocity from Σw to Σc, respectively [15]. Differentiating
Eq. (4) with respect to time, the body velocity of the relative
rigid body motion gco can be written as follows (see Ref.
16):

motion gho into agreement with a given constant reference
gd. In this subsection, let us consider a control error system
in order to establish a visual robot motion error system.
In our previous work [6], using the estimated relative
rigid body motion(11)_
we defined the control error as gec = g−1
d gho. However, theestimation input ue has directly affected not only estimation
but also control; as a result, this leads to degraded estimation performance from the practical viewpoint. In order toremove the above negative effect in this paper, we define
the control error as gec = g−1
d gho (2) between the actual
relative rigid body motion gho and the desired one gd.
Next, we derive the control error gec (2) without thenonmeasurable
value gho. Using the estimation error matrix_
gee = g−co1gco, the control error can be transformed as(5)
where Ad(gab) is the adjoint transformation [15] associatedwith gab and Vbwo is the body velocity of the target object
relative to Σw.
In the case of a fixed camera configuration, that is,
Vbwc = 0, the body velocity Vbco can be rewritten as(6)
In visual feedback control, the relative rigid body
motion gco cannot be utilized directly, but an image featuref ∈ R2m can be measured through a pinhole camera withperspective projection. Hence, we consider a nonlinear
observer in order to estimate the relative rigid body motion
gco from the image feature f. Using the body velocity of therelativegco (6), we choose estimates____ body motion__ rigid^
gco = (pco, eξθco) and Vbco of the relative rigid body motion
and velocity, respectively, as(12)_
In Eq. (12), gd and gho are available information. While theestimation error vector ee can be given by Eq. (9), theestimation error matrix gee, which is defined using thenonmeasurable value gco as in Eq. (8), cannot be directly
obtained. Focusing on the definition of the estimation error^
vector ee, that is, ee := [pTeeeTR(eξθee)]T, the position estimation
error pee can be obtained directly from ee. In the case of the^rotation estimation error eξθee, if we assume that the regionof the attitude estimation error is restricted to−π / 2 ≤ θee ≤ π / 2, then ξθee can be obtained as follows:(7)
The new input ue_is to be__determined in order to drive theestimated values gco and Vbco to their actual values.
Here, we _define the estimation error between theestimated value gco and the actual relative rigid body motion
gco as

(13)

(8)
^

Therefore, it is possible to_derive the control error gec using
available information gd, gho, and ee by Eq. (12), since gee
can be derived from ee through ξθee by using Eq. (1). A
diagram showing the relationship between ee and gec isgiven in Fig. 2. It should be noted that the assumption−π / 2 ≤ θee ≤ π / 2 will not be a new constr aint, since wehavealreadymadetheassumption that theattitudeestimation error θee is small enough in developing the estimation
error vector ee in Section 2.1.
Differentiating Eq. (2) with respect to time, the control error system can be represented as^Using the notation eR(eξθei) = sk(eξθei)∨, the vector of the^estimation error is given by ee := [pTeeeR(eξθee)T]T. Supposethat the attitude estimation error θee is small enough that we^^
can let eξθee u I + sk(eξθee). Then using a first-order Taylor
expansion approximation, the estimation error vector ee canbe obtained from the image feature_ f and the estimated valueof the relative rigid body motion gco (i.e., the measurement
and the estimate) as follows:
(9)
_
where
f is the estimated value of the image feature and_J(gco) is an image Jacobian-like matrix [16]. In the same
way as Eq. (5), the estimation error system can be represented by

(14)

(10)
2.2 Control error system
The objective of visual feedback control for fixedcamera systems is to bring the actual relative rigid bodyFig. 2. Relationship diagram between ee and gec.

3

where n := Nx, N := diag{I, –I, –I}, and β0 is a positive
scalar.
Due to space limitations, the proof is only sketched.
The proof can be completed by using the following energyfunction:

2.3 Visual motion robot error system
The manipulator dynamics can be written as(15).
where M(q) ∈ Rn×n is the inertia matrix, C(q, q) ∈ Rn×n isnthe Coriolis matrix, g(q) ∈ R is the gravity vector, and q,..
.
q, and q are the joint angle, velocity, and acceleration,
respectively. τ is the vector of the input torque, and τdrepresents the disturbance input [17]. Since the manipulator
dynamics is considered, the hand body velocity is given by.
Vbwh = Jb(q)q, where Jb(q) is the body manipulator Jacobian[15]. Let us define the error vector with respect to the joint. .
.
velocity as ξ := q − qd, where qd represents the desired joint
velocity.
Next, we propose control laws for the manipulator as

(19)
^

^

where E(gei) := 1 / 2||pei||2 + φ(eξθei), and φ(eξθei) := 1 / 2tr(I −^eξθei ) is an error function of the rotation matrix (see, e.g., Ref.
18).
2.4 Energy function and stabilizing control law
Based on the above passivity property of the visualmotion robot error system, we now propose the following
control input for the interconnected system:
(20)

(16)
..
where qd represents the desired joint acceleration. The new
input uξ is to be determined in order to reduce the joint
velocity error ξ. Here, we design the reference of the joint.
velocity as qd := J†b(q)ud, where ud is the desired body
velocity, obtained from the control error system part. Thus,
Vbwh in Eq. (14) should be replaced by ud. Using Eqs. (10)and (14) to (16), a visual feedback error system with themanipulator dynamics (which we call the visual motion
robot error system) can be derived as follows:

where Kξ := diag{kξ1, . . . , kξn} denotes the positive gain
matrix for each joint axis of the manipulator. Kc :=diag{kc1, . . . , kc6} and Ke := diag{ke1, . . . , ke6} are thepositive gain matrices of the x, y, and z axes of the translation and rotation for the control case and the estimationcase, respectively. Considering the passivity of the visualmotion robot error system shown in Lemma 1, the following theorem can be derived for stability.
Theorem 1 If w = 0, then the equilibrium point x= 0 for closed-loop system (17) and (20) is asymptoticallystable.
Proof Differentiating Eq. (19) with respect to timeand using control input (20), we obtain(21)
This completes the proof.
(Q.E.D.)
Theorem 1 can be proved using the energy function(19) as a Lyapunov function. Here we compare proposed
control law (20) with the previous one in Ref. 6. Block(17)
where u := [uTξ(Ad(g−d1)ud)TuTe ]T and w := [τTd (Vbwo)T]T. We define the state of the visual motion robot error system as
x := [ξTeTc eTe ]T, which consists of the joint velocity error ξ,
the control error vector ec, and the estimation error vector
ee. It should be noted that if x = 0, then the actual relative
rigid body motion gho tends to the reference one gd.
In the visual motion robot error system (17), thefollowing important lemma concerning passivity between
the input u and the output n holds.
Lemma 1 If w = 0, then visual motion robot error
system (17) satisfies

Fig. 3. Block diagram of part of the control law: left
side, with proposed control law; right side, with previous
control law [6].

(18)

4

visual feedback control, we propose the stabilizing receding horizon control based on optimization in this paper.
3.1 Control Lyapunov function for visualfeedback systems
In this section, a finite horizon optimal control problem (FHOCP) for visual motion robot error system (17) is
considered. The FHOCP for visual motion robot error
system (17) at time t consists of minimization with respect
to the input u(τ, x(τ)), τ ∈ [t, t + T] of the following costfunction:Fig. 4. Block diagram of the closed-loop system: left
side, with proposed control law; right side, with previous
control law [6].

diagrams of the part of the control law in the case of the
proposed method and the previous one [6] are shown at theleft and right sides of Fig. 3, respectively. In Ref. 6, thecontrol error ec has interfered with the input for the estimation ue = Ke(ee − AdT(eξ^θec)ec). In the sequel, the presence ofec has negatively_ affected the estimation of the relative rigid
body motion gco. On the other hand, with proposed visualfeedback control law (20), the feedback inputs are separated
by the error vectors, that is, uξ = −Kξξ, ud = Ad(gd)Kcec, andue = Keee. Thus, the proposed control law can overcome theproblem of the above undesirable influence.
Similarly to Fig. 3, block diagrams of the closed-loopsystem in the case of the proposed method and the previous
one [6] are shown on the left and right sides of Fig. 4,
respectively. Differently from Ref. 6, the observer of the
proposed system can be unaffected by the reference gd, andthus it is clear that the estimation task is not directly
impacted by the control task.
This is why the block matrix with respect to the inputof the visual motion robot error system (17) is diagonal. It
should be interpreted that the structure of the passivitybased visual feedback control can be separated into theestimation part and the control one in some way. This allows
us to easily set the gains of control law (20) and the weightsof the cost function (22) to (24) described below from apractical point of view.
However, visual feedback control law (20) is notbased on optimization, and the desired control performancecannot be guaranteed explicitly. In the next section, stabilizing receding horizon control based on optimal control
theory is proposed.

(22)
(23)
(24)

where R(t) is a positive diagonal matrix, and Eqi(gei(t)) :=^qpi(t)||pei(t)||2 + qRi(t)φ(eξθei(t))(i ∈ c, e), with the state x(t) =x0. The special property of cost function (22) to (24) is thatthe terminal cost is derived from the energy function of thevisual motion robot error system. Furthermore, the rotation
error-related part of the stage cost is derived from the error^
function φ(eξθei) instead of the commonly used quadraticξ^θei 2
form ||eR(e )|| . For a given initial condition x0, we denote
this solution of the FHOCP as u∗(τ, x(τ)), τ ∈ [t, t + T]. In
receding horizon control, at each sampling time δ, theresulting feedback control at state x0 is obtained by solving
the FHOCP and setting(25)
A control Lyapunov function closely related to stability is defined as follows:
Definition 1 [8] The control Lyapunov function
S(x) is given by(26)
where l(x, u) is a positive definite function. The following
lemma concerning the control Lyapunov function is important in the proof of stabilizing receding horizon control.
Lemma 2 Suppose that w = 0, ||θec|| ≤ π / 2, ||θee|| ≤π / 2, and the design parameter ρ satisfies

3. Stabilizing Predictive Visual Feedback Control
The objective of this section is to propose predictive
visual feedback control based on optimal control theory. A
camera can provide more information than the current
derivation from a nominal position at the sample instant.
This property can be exploited to predict the target’s future
position and improve control performance. As predictive(27)
where Q := diag{qξIn, qpcI3, qRc I3, qpeI3, qReI3}. Then, theenergy function ρV(x) of visual motion robot error system(17) can be regarded as a control Lyapunov function.

5

Proof Using Eq. (21), which is the time derivativeof V along the trajectory of system (17), the positive definite
function l(x(t), u(t)) (23) and the stabilizing control law uk(20) with K = ρ / 2R−1 for the system, Eq. (25) can betransformed into

(30)
where x∗ is the optimal state trajectory. This cost, which isan upper bound for J(x∗(t + δ), u∗, T), satisfies(31)
Using the positive definite function l(x(t), u(t)) (23) and thestabilizing control law uk (20) for the system, dividing bothsides by δ, and taking the limit as δ → 0, Eq. (31) can betransformed into

(28)
^

^

where we have used the fact that φ(eξθei) ≤ ||eR(eξθei)||2 for. all||θei|| ≤ π / 2. Therefore, the condition infu[S(x) +l(x, u)] ≤ 0 will be satisfied if the assumption ρ2I ≥ 4QR is
valid.
(Q.E.D.)
Lemma 2 shows that the energy function ρV(x) of thevisual motion robot error system (17) can be regarded as acontrol Lyapunov function in the case of ρ2I ≥ 4QR.

(32)
Considering that the control input during the first δ is
uRH = u∗, by the assumption ρ2I ≥ 4QR, the derivative ofJ(x∗(t), uRH, T) is negative definite. Therefore, we haveshown that J(x∗(t), uRH, T) qualifies as a Lyapunov function
and asymptotic stability is guaranteed.
(Q.E.D.)
Theorem 2 guarantees the stability of the receding
horizon control using a control Lyapunov function for the3D visual motion robot error system (17), which is a highly
nonlinear and relatively fast system. Since the stabilizing
receding horizon control design is based on optimal control
theory, the control performance should be improved compared to the passivity-based visual feedback control uk (20),under the condition of adequate gain assignment in the cost^function. It should be noted that the error function φ(eξθei)of the rotation matrix can be directly used in the stage cost(23). The assumption ρ2I ≥ 4QR is very simple and it isquite easy to set the value of the design parameter ρ, byvirtue of the fact that the matrix N is a block diagonal
matrix, contrary to Ref. 6. Compared with the previous
work [14], the main advantage of this approach is that the3D visual feedback system is not restricted to a planar
manipulator, and can treat not only the position but also theorientation. This allows us to extend the range of technological applications.
In this paper, the receding horizon controller predicts
the states ξ, gec, and gee, using the visual motion robot error

3.2 Stabilizing receding horizon control for
visual feedback systems
We are now in a position to state the main result of
this paper.
Theorem 2 Consider the cost function (22) to (24)for the visual motion robot error system (17). Suppose thatw = 0, ||θec|| ≤ π / 2, ||θee|| ≤ π / 2, and ρ2I ≥ 4QR; then thereceding horizon control for the visual motion robot error
system is asymptotically stabilizing.
Proof Our goal is to prove that J(x∗(t), uRH, T),
which is the cost-to-go applying receding optimal control
uRH, will qualify as a Lyapunov function for the closed-loopsystem. We construct the following suboptimal control
strategy for the time interval [t + δ, t + T + δ]:(29)
where uk is stabilizing control law (20) with K = ρ / 2R−1 forthe visual motion robot error system. The associated cost is6system (17) as the internal model. Considering that gd is agiven reference value, this implies that predictive visual.
_feedback control anticipates the physical values q, gho, andgco from visual information. Also, it is interesting thatreceding horizon control is applied to the visual motion
observer.
4. Simulations and Experiments
In this section, the validity of the proposed control
law is confirmed by simulation results in the case of a static
target object and by experimental results in the case of amoving target object. The simulation and experimental
results on a two-degree-of-freedom manipulator, as depicted in Fig. 5, are shown as a simple illustration of our
proposed method, although it is valid for 3D visual feedback systems. The target object has four feature points.

Fig. 6. Norm of the state.

C/GMRES software [19] is utilized. The control input withthe receding horizon control is updated every 1 ms. It must
be calculated by the receding horizon controller within that
period. The horizon is selected as T = 0.02 s. The norm of
the state x applying the proposed control law is shown in
Fig. 6. Asymptotic stability can be confirmed by steadystate performance.
Next, we compare the performance of the receding
horizon control law uRH (25) and the visual feedback control law uk (20). The controller parameters for the passivity-based control law uk (20) were empirically selected as
Kξ = diag{10, 3}, Kc = diag{2, 2, 1, 1, 1, 2}, Ke = diag{1,1, 1, 1, 1, 1}. The aforementioned weights and these gains
are selected in order not to exceed the limit of the input
torque for the manipulator. Figure 7 shows the actual con^trol error er := [pTer eTR(eξθer)]T, which is the error vector

4.1 Simulation results
First, we present results for the stability analysis withthe static target object. The simulation is carried out with
the initial conditions q1(0) = π / 4 rad, q2(0) = π / 12 rad, pwc= [0.4732 0.1 0.1] T m, ξθwc = [0 0 0]T rad, pwo = [0.3986 0–0.9] T m, ξθwo = [0 0 −0.5087] T rad. We use the referencesof the relative rigid body motion as constant values, that is,
pd = [0 0 −0.9] T m, ξθd = [0 0 0]T rad. The initial error condition x(0) is ξ(0) = [0 0]T rad/s, pec(0) = [−0.294 −0.293 0]Tm, ξθec (0) = [0 0 −1.556] T rad, pee (0) = [0 −0.004 0] T m,ξθee(0) = [0 0 −0.015]T rad. The weights of cost function(22) to (24) were selected as qξ = 0.001, qpc = 0.1,
qRc = 0.05, qpe = 0.3, qRe = 0.1, and R = diag{0.1, 3.2, 0.7,
2, 2, 1, 1, 2, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3} and ρ = 1 satisfy
ρ2I ≥ 4QR. To solve the real-time optimization problem, theFig. 7. Actual control error: solid, with predictive
control law; dashed, with passivity-based control law.

Fig. 5. Experimental arm and fixed camera.

7

Real-Time Workshop and dSPACE software, which includes ControlDesk, Real-Time Interface, and so on. We set
up a Sony XC-HR57 camera which always kept all featurepoints of the target object in the field of view (see Fig. 5).
The video signals were acquired by a PicPort-Stereo-H4D
frame graver board and the HALCON image processing
software. The sampling time of the controller and the frame
rate provided by the camera were 1 ms and 60 fps, respectively. Hence, the image feature f was renewed every 16.7
ms. The control law τ was interpolated every 5.6 ms using
the most recent available data from the vision system. The
difference between the sampling rate of the robot control
and of the image grabbing system can be decreased by using
commercially available cameras with superior performance.
The target object, projected on the liquid crystal
display, had four feature points and moved for two time
periods (0 ≤ t < 0.85 and 2.15 ≤ t < 3.4), as depicted in Fig.
8. The experiment was carried out with the initial conditions
q1(0) = π / 6 rad, q2(0) = −π / 6 rad, pwc = [0.4732 0.1 0.1]Tm, ξθwc = [0 0 0]T rad. We set the initial error condition
x(0) = 0. The weights of the cost function (22) to (24) wereselected as qξ = 0.1, qpc = 3, qRc = 1, qpe = 30, qRe = 10,
and R = diag{0.1, 0.5, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05,
0.008, 0.008, 0.008, 0.01, 0.01, 0.01} and ρ = 1 satisfiedρ2I ≥ 4QR. The control input with receding horizon control
was updated every 5.6 ms. The horizon was selected as
T = 11.2 ms.
Figure 9 shows the rigid body motion of the end-effector of the manipulator gwh. The solid lines represent therigid body motion gwh applying the proposed stabilizing
receding horizon control law, and the dashed lines denote
the rigid body motion of the actual moving target object
gwo. In Fig. 9, it can be verified that the manipulator tracks
the moving target object. Although a slight error remains in
steady state because of the friction force with the manipulator, it decreases with increasing horizon length. Figure 10
illustrates the rigid body motion gwh of the translation of yfrom 3.5 s to 3.6 s when the target object has just stopped.
In Fig. 10, the actual trajectory is different from the reced-

between the actual gho and gd. In Fig. 7, the solid lines
represent the errors when applying the proposed stabilizing
receding horizon control law uRH (25), and the dashed lines
represent those for the control law uk (20). The rise time
applying the receding horizon control is shorter than thatfor the passivity-based control law. These results suggestthat the controller predicts the movement of the target
object using the visual information, and as a result, themanipulator moves more aggressively. This validates oneof the expected advantages of the stabilizing receding horizon control for visual feedback systems.
Next, the performance for the horizon length T and
the design parameter value ρ is compared in terms of theintegral cost in Table 1. The cost is calculated by thefollowing function:(33)
We set the integral interval I = 3 in this simulation. Since
the cost of the stabilizing receding horizon method issmaller than the passivity-based visual feedback control
method (20) under the conditions of an adequate costfunction, it can be easily verified that the control performance is improved. With increasing weight of the terminal
cost from ρ = 1 to ρ = 5 the cost too increases. With higher
terminal cost, the state value is reduced more strictly, usinga large control input. In this simulation, since the weightsof the control input are larger than those of the state, thecost increases. As the horizon length increases from T =0.02 to T = 0.5, the cost is reduced. In the case of T = 1, thecalculation cannot be completed within one sampling interval, due to limited computing power.
4.2 Experimental results
In this subsection, we present experimental results forthe case of a moving target object. The manipulator wascontrolled by a digital signal processor (DSP) fromdSPACE Inc., which utilizes a powerPC 750 running at 480
MHz. The control problems were written in MATLAB and
SIMULINK, and were implemented on the DSP using theTable 1. Values of the cost function in simulation

Fig. 8. Trajectory of the target object.

8

Table 2. Values of the cost function in experiment

5. Conclusions
This paper proposes stabilizing receding horizon
control for 3D fixed camera visual feedback systems, which
are highly nonlinear and relatively fast systems, as a methodof predictive visual feedback control. First, the visual motion robot error system is reconstructed in order to improvethe estimation performance. Next, stabilizing receding horizon control for the visual feedback systems based on
optimal control theory is designed. It is shown that thestability of the receding horizon control scheme is guaranteed by using the terminal cost derived from the energyfunction of the visual motion robot error system. It should
be noted that the proposed stabilizing receding horizon
control approach can be applied to visual feedback systems
with a movable camera configuration [20]. In the simulation and experimental results, the control performance ofthe stabilizing receding horizon control is improved compared to that of passivity-based control.

Fig. 9. Rigid body motion gwh.

ing horizon reference values. This indicates that the manipulator can track the target object
with re-prediction of_
the physical values ξ, gho, and gco, even if its movement
suddenly changes.
Finally, the performance for the horizon length T iscompared in terms of the integral cost, calculated using Eq.
(33) with I = 5 in Table 2. Similarly to the simulation results,
the cost is reduced as the horizon length increases from T= 0.0056 to T = 0.056. For values greater than T = 0.056,
the control law cannot be implemented due to limited
computing power.

REFERENCES
1. Chaumette F, Hutchinson S. Visual servo control, Part
I: Basic approaches. IEEE Robot Automation Mag
2006;13:82–90.
2. Chaumette F, Hutchinson S. Visual servo control, Part
II: Advanced approaches. IEEE Robot Automation
Mag 2007;14:109–117.
3. Omote K et al. Self-guided robotic camera control for
laparoscopic surgery compared with human camera
control. Am J Surg 1999;177:321–324.
4. Krupa A, Chaumette F. Control of an ultrasoundprobe by adaptive visual servoing. Proc of the 2005
IEEE/RSJ International Conference on Intelligent
Robots and Systems, p 2007–2012.
5. Yu S, Nelson BJ. Autonomous injection of biologicalcells using visual servoing. In: Rus D, Singh S (editors). Experimental robotics VII. Springer-Verlag;
2001. p 169–178.
6. Kawai H, Murao T, Fujita M. Passivity-based control
and estimation of dynamic visual feedback systems
with a fixed camera. Proc of the 43rd IEEE Conference on Decision and Control, p 4022–4027, 2004.

Fig. 10. Horizontal rigid body motion pwhy: solid, withactual trajectory; dashed, with receding horizon reference
trajectory.

9

7. Mayne DQ, Rawlings JB, Rao CV, Scokaert POM.
Constrained model predictive control: Stability andoptimality. Automatica 2000;36:789–814.
8. Jadbabaie A, Yu J, Hauser J. Unconstrained recedinghorizon control of nonlinear systems. IEEE Trans
Automatic Control 2001;46:776–783.
9. Milam MB, Franz R, Hauser JE, Murray RM. Receding horizon control of vectored thrust flight experiment. IEE Proc Control Theory Appl 2005;
152:340–348.
10. Kawai Y, Nakaso Y, Azuma T, Fujita M. Experiments
on stabilizing receding horizon control of a directdrive manipulator. Proc of the 16th IFAC World Congress on Automatic Control, Paper Code Tu-E04TO/2, 2005.
11. Hunt AE, Sanderson AC. Vision-based predictive
robotic tracking of a moving target. Technical Report,
Carnegie Mellon University, 1982.
12. Ginhoux R, Gangloff J, de Mathelin M, Soler L,
Sanchez MMA, Marescaux J. Active filtering ofphysiological motion in robotized surgery using predictive control. IEEE Trans Robot 2005;21:67–79.
13. Lange F, Hirzinger G. Predictive visual tracking oflines by industrial robots. Int J Robot Res 2003;
22:889–903.

14. Fujita M, Murao T, Kawai Y, Nakaso Y. An experimental study of stabilizing receding horizon controlof visual feedback system with planar manipulators.
In: Findeisen R, Allgöwer F, Biegler L (editors).
Assessment and future directions of nonlinear model
predictive control. Springer-Verlag; 2007. p 573–
580.
15. Murray R, Li Z, Sastry SS. A mathematical introduction to robotic manipulation. CRC Press; 1994.
16. Fujita M, Kawai H, Spong MW. Passivity-based dynamic visual feedback control for three dimensionaltarget tracking: Stability and L2-gain performance
analysis. IEEE Trans Control Syst Technol
2007;15:40–52.
17. Spong MW, Hutchinson S, Vidyasagar M. Robotmodeling and control. John Wiley & Sons; 2006.
18. Bullo F, Lewis AD. Geometric control of mechanicalsystems. Springer-Verlag; 2004.
19. Ohtsuka T. A continuation/GMRES method for fastcomputation of nonlinear receding horizon control.
Automatica 2004;40:563–574.
20. Murao T, Kawai H, Fujita M. Passivity-based controlof dynamic visual feedback systems with movable
camera configuration. Electron Commun Japan
2009;92:1–11.

AUTHORS (from left to right)

Toshiyuki Murao received a B.E. degree from the Department of Electrical and Computer Engineering, Kanazawa
University, in 2003, an M.E. degree from the Division of Electronics and Computer Science, Kanazawa University, in 2005,
and a D.Eng. degree from the Department of Mechanical and Control Engineering, Tokyo Institute of Technology, in 2008.
Since 2006, he has been an assistant professor at the Advanced Institute of Industrial Technology. His research focuses on
passivity-based visual feedback control of robot systems.

Hiroyuki Kawai received a B.S. degree from the Department of Electrical and Computer Engineering, Kanazawa
University, in 1999, and M.S. and Ph.D. degrees from the Graduate School of Natural Science and Technology, Kanazawa
University, in 2001 and 2004. From 2004 to 2005, he was a postdoctoral scholar at the Information Technology Research Center,
Hosei University. In 2005, he joined Kanazawa Institute of Technology as an assistant professor. Since 2010, he has been an
associate professor in the Department of Robotics. His research focuses on visual feedback control of robot systems andnetworked cellular vision systems. In particular, he is interested in passivity-based control. He received an IEEE Transactions
on Control Systems Technology Outstanding Paper Award from the IEEE Control Systems Society (CSS) in 2008.

10

AUTHORS (continued)

Masayuki Fujita is a professor in the Department of Mechanical and Control Engineering, Tokyo Institute of Technology.
He received a D.Eng. degree in electrical engineering from Waseda University in 1987. Prior to his appointment at Tokyo
Institute of Technology, he held faculty appointments at Kanazawa University and the Japan Advanced Institute of Science andTechnology. He also held a visiting position at the Technical University of Munich, Germany. His research interests include
passivity-based visual feedback, cooperative control, and robust control with its industrial applications. He has served as General
Chair of the 2010 IEEE Multi-Conference on Systems and Control. He is currently a Vice President of the IEEE CSS and amember of the IEEE CSS Board of Governors, and was a Director of SICE. He has served/been serving as an associate editor
for IEEE Transactions on Automatic Control, IEEE Transactions on Control Systems Technology, Automatica, and the Asian
Journal of Control, and as an Editor for the SICE Journal of Control, Measurement, and System Integration. He is a recipientof a 2008 IEEE Transactions on Control Systems Technology Outstanding Paper Award. He has also received Outstanding Paper
Awards from the SICE and the ISCIE, and a 2010 SICE Education Award.

11