Electronics and Communications in Japan, Vol. 96, No. 10, 2013
Translated from Denki Gakkai Ronbunshi, Vol. 132-C, No. 5, May 2012, pp. 721–729

Stabilizing Predictive Visual Feedback Control via Image Space Navigation
Function
TOSHIYUKI MURAO,1 HIROYUKI KAWAI,2 and MASAYUKI FUJITA3
1

Advanced Institute of Industrial Technology, Japan
2
Kanazawa Institute of Technology, Japan
3
Tokyo Institute of Technology, Japan

camera field of view since the work of Chaumette [5].
Navigation functions that are globally convergent potentialfunctions are reported to yield very good results for thecamera field of view problem [6–8]. Cowan and colleagues[6] proposed a visual feedback controller to bring a robot
to rest at a desired configuration for the field of viewproblem by using navigation functions. In Ref. 7, a modelbased geometric visual servoing framework capable ofocclusion-free servoing through the navigation function
method was presented. Chen and colleagues [8] developed
an off-line path planner based on an image space navigation
function with an adaptive 2.5D visual servoing controller.
However, the desired control performance cannot always
be guaranteed explicitly, since these useful control methods[6–8] are not based on optimization.
On the other hand, receding horizon control, also
recognized as model predictive control, is a well-known
control strategy in which a current control action is computed by solving a finite horizon optimal control problem
on-line [9]. For visual feedback systems, a few implementations of receding horizon control have been reported.
Sauvée and colleagues [10] proposed an image-based visual servoing scheme based on nonlinear model predictive
control for robot systems. In Ref. 11, nonlinear model
predictive control of instrument motion based on ultrasound images is investigated. Recently, the results of comparison between the two image prediction models—anonlinear global model and a local model based on theinteraction matrix—were given in Ref. 12. Although good
receding horizon control approaches for the visual feedback
system considering mechanical and visibility constraints
are reported in Refs. 10–12, stability is not addressed. In
Refs. 13 and 14, the authors proposed stabilizing receding
horizon control for 3D visual feedback systems, as predictive visual feedback control. However, this position-based
visual feedback control method through the nonlinear re-

SUMMARY
This paper investigates stabilizing receding horizon
control via an image space navigation function for three-dimensional (3D) visual feedback systems. First, we describe
the representation of a relative pose and a camera model.
Next, a visual motion error system which can handle timevarying desired motion is constructed. Then, visual motion
observer-based stabilizing receding horizon control for 3D
visual feedback systems is proposed. Moreover, a path
planner appropriate for the visual motion error system isdesigned through an image space navigation function tokeep all features in the camera field of view. The main
contribution of this paper is to show that the path planner
which always remains in the camera field of view during
servoing is designed for position-based visual feedback
receding horizon control based on optimal control theory.
Finally, we present simulation and nonlinear experimental
results in order to verify control performance with visibility
maintenance of the proposed control scheme. © 2013
Wiley Periodicals, Inc. Electron Comm Jpn, 96(10): 12–21,
2013; Published online in Wiley Online Library (wileyonlinelibrary.com). DOI 10.1002/ecj.11493

Key words: visual feedback control; receding horizon control; navigation function; passivity; stability.

1. Introduction
Visual feedback control is a very flexible and effective method for the autonomous performance of various
tasks for robotic systems [1]. Many researchers have tackled the various problems of visual feedback control [2–4].
In particular, there has been an increase of interest in
problems in which all feature points remain within the© 2013 Wiley Periodicals, Inc.
12

the rotation matrix from a frame Σa to a frame Σb. Then, therelative pose from Σa to Σb can be represented byceding horizon approach can allow feature points to leave
the field of view.
In this paper, we propose stabilizing receding horizon
control via an image space navigation function for 3D
visual feedback systems with an eye-in-hand configuration
as shown in Fig. 1. First, a visual motion error system which
can handle time-varying desired motion is constructed.
Second, stabilizing receding horizon control for the visualmotion error system using a control Lyapunov function is
proposed. Then, a path planner appropriate for the visualmotion error system is designed through an image space
navigation function in order to keep all features within thecamera field of view. Path planning on an image space could
be of significant benefit when used in conjunction with theproposed position-based receding horizon control by using
an error defined on a Cartesian space. Finally, the control
performance with visibility maintenance of the proposed
control scheme and with the previous scheme [13] is evaluated through simulation and nonlinear experimental results.
The paper is organized as follows. In Section 2, we
propose stabilizing predictive visual feedback control for atime-varying desired relative pose. In Section 3, we designa path planner based on an image space navigation function.
Sections 4 and 5 describe simulation and experimental
results that show the advantages of the proposed control
law, followed by conclusions in Section 6.

(1)
where the notation “∧” denotes the skew-symmetric operator [15]. In contrast, the notation “∨” denotes the inverse
operator to “∧.”The relative pose from Σc to Σo can be derived byusing the composition rule for rigid body transformations[15] as follows:(2)
Differentiating Eq. (2) with respect to time, the body velocity of the relative pose gco can be written as follows (seeRef. 16):(3)
where Vbwc ∈ R6 and Vbwo ∈ R6 are the body velocities of
the camera and the target object relative to Σw, respectively.
The notation Ad(gab) represents the adjoint transformation
associated with gab [15].
^The relative pose gco = (pco, eξθco) cannot be immediately obtained in the visual feedback system, because thetarget object velocity Vbwo is unknown and furthermore
cannot be measured directly. To control the camera using
visual information provided by a computer vision system,
we use the pinhole camera model with a perspective projection as shown in Fig. 2. Here, we consider four featurepoints on the rigid target object in this paper. Let λ be thefocal length, αx and αy be the scaling factors along the x andy axes (in pixels/meter), and poi ∈ R3 and pcoi ∈ R3 be theposition vectors of the target object’s i-th feature point

2. Stabilizing Predictive Visual Feedback Control
2.1 Vision camera model
Visual feedback systems with an eye-in-hand configuration use three coordinate frames: the world frame
Σw, the camera frame Σc, and the object frame Σo, as in Fig.
^
1. Let pab ∈ R3 and eξθab ∈ SO(3) be the position vector andFig. 1. Visual feedback system with an eye-in-handconfiguration. [Color figure can be viewed in the online
issue, which is available at wileyonlinelibrary.com.]

Fig. 2. Pinhole camera model. [Color figure can beviewed in the online issue, which is available atwileyonlinelibrary.com.]13

relative to Σo and Σc, respectively. Using a transformationof coordinates, we have pcoi = gcopoi, where pcoi and poi
should be regarded, with a slight abuse of notation, as[pTcoi1]T and [pToi1]T via the well-known homogeneous coordinate representation in robotics, respectively (see, e.g.,
Ref. 15).
The perspective projection of the i-th feature point
onto the image plane gives us the image plane coordinate
fcoi := [fcoxi fcoyi]T ∈ R2 as_
where _fco ∈ R8 is the estimated value of the image features
and J(gco) ∈ R8 × 6 is an image Jacobian-like matrix [16].
The pose control error vector ec can also be calculated byusing the desired relative pose gcd and the estimation error
vector ee [14].
Differentiating Eqs. (6) and (7) with respect to time,
we construct the visual motion observer-based pose control
error system (which we call the visual motion error system)as follows:

(4)

(9)

where pcoi = [xcoi ycoi zcoi]T. It is straightforward to extend
this model to four image points by simply stacking thevectors of the image plane coordinate, that is,
fco = [fTco1. . . fTco4]T ∈ R8. Hereafter, fab means f(gab ) for simplicity.

where
(10)
and Vbcd ∈ R6 is the body velocity of the desired relativepose gcd. Although the visual motion error system (9) in thispaper has the same form as in Refs. 13 and 14, the input u
is different in the two systems. Let us define the error vectorof the visual motion error system as x := [eTc eTe ]T, which
consists of the pose control error vector ec and the estimation error vector ee. It should be noted that if the vectors of
the pose control error and the estimation error are equal tozero, then the actual relative pose gco tends to the desired
one gcd when x → 0. In the case that the target object is static(that is, Vbwo = 0), the visual motion error system (9) is
passive from the input u to the output x. The passivity
property can be proved by using the following positive
definite function:

2.2 Visual motion error system
The objective of position-based visual feedback control is, in general, to bring the actual relative pose gco to thedesired one gcd. Although we handle the time-varying desired pose gcd, the final desired one is assumed to beconstant from the practical point of view in this paper. The
visual feedback control task requires information on therelative pose gco. Since the only measurable information is
the image features fco in the visual feedback system, weconsider the following visual motion observer based on Eq.
(3) in order to estimate the relative pose gco from the image
features fco:

(11)
ξ^θei

ξ^θei

where E(gei ) := 1 / 2|pei |2 + φ(e ) ∈ R and φ(e ) := 1/2 tr(I –^eξ θei) ∈ R is the error function of the rotation matrix [17].

(5)
__
_
where gco and Vbco ∈ R6 are the estimated values of therelative pose and body velocity, respectively. The new inputue ∈ R_6 is to be__ determined in order to drive the estimatedvalues gco and Vbco to their actual values.
Here we define the pose control error gec and theestimation error gee as follows:

2.3 Stabilizing predictive visual feedback
control
In this subsection, the finite horizon optimal control
problem (FHOCP) for visual motion error system (9) is
considered. The FHOCP for system (9) at time t consists ofthe minimization with respect to the inputu(τ, x(τ)), τ ∈ [t, t + T], of the following cost function:(6)(7)
ξ^θei ∨

Using the notations rei := sk(e ) ∈ R3 and^^^sk(eξθei) := 1 / 2(eξθei − e−ξθei) ∈ R3 × 3, we next define thevectors of the pose control error and the estimation error as
ec := [pTec rTec]T ∈ R6 and ee := [pTee rTee]T ∈ R6, respectively.
The estimation error vector ee can be obtained from theimage_ features fco and the estimated value of the relativepose gco (i.e., the measurement and the estimate) as follows:

(12)

(13)

(8)

(14)

14

where R(t) is a positive diagonal matrix, and^Eqi(gei(t)) := qpi(t)|pei(t)|2 + qri(t)φ(eξθei(t)), with the statex(t) = x0. For a given initial condition x0, we denote thissolution of the FHOCP as u∗(τ, x(τ)), τ ∈ [t, t + T]. In receding horizon control, at each sampling time δ, the resulting feedback control at state x0 is obtained by solving the
FHOCP and setting

From the proposed stabilizing receding horizon control law (15) and the input to the visual motion error system(10), the input to the vision camera is designed as follows:(16)
RH T T
T
where uRH = [(uRHc ) (ue ) ] . The desired relative pose
gcd can be obtained by solving the equation

(15)
(17)
Assuming that the target object is static, we have thefollowing theorem concerning the convergence of the stabilizing receding horizon control for the visual feedback
system.
Theorem 1 Consider cost function (12)–(14) for
visual motion error system (9). Suppose that
Vbwo = 0, |θec| ≤ π / 2, |θee| ≤ π / 2, and ρ2I ≥ 4QR; then thereceding horizon control for the visual motion error system
is asymptotically stabilizing.
Theorem 1 guarantees the stability of receding horizon control by using the fact that the energy function
ρV(x) of the visual motion error system (9) can be regarded
as a control Lyapunov function in the case of ρ2I ≥ 4QR.
One of the contributions of this paper is that the proposed
control law can be applied to the time-varying desired
relative pose gcd(t), similar to the case of a constant one [13,14]. Since the stabilizing receding horizon control designis based on optimal control theory, the control performance
should be improved under the condition of adequate gain
assignment in the cost function.

Hence, the vision camera input Vbwc needs only the body
velocity Vbcd.
Here we introduce the navigation function-based
method as a technique for constructing artificial potential
fields in order to design Vbcd which can achieve the control
objective. First, we define the desired image feature vector
and the final one as fcd := f(gcd) and fcdf := f(gcdf), respectively. The navigation function used in this paper is defined
as follows:
Definition 1 [7, 8] Let D be a space where all featurepoints of the target object remain visible, and let fcdf be in
the interior of D. A smooth Morse function
ϕ(fcd) : D → [0, 1] is a navigation function if
1. a unique minimum exists at fcdf;
2. it is admissible on D, that is, uniformly maximal
on the boundary of D.
To develop the desired body velocity Vbcd, we derivea relationship between fcd defined on the image space andVbcd defined on the Cartesian space. Using the desired feature point pcdi = [xcdi ycdi zcdi]T, the desired image features
can be obtained as follows:

3. Image Space Navigation Function-Based Path
Planning(18)

3.1 Design of desired body velocity
The inherent problem of position-based visual feedback control using only an error defined on a Cartesian
space has been stated, namely, that it is difficult to assurethat the target object will always remain in the camera fieldof view during servoing [5]. Because the proposed stabilizing receding horizon control in the previous section is aposition-based method, it may leave the camera field ofview. In this section, a path planner appropriate for thevisual motion error system is designed through an image
space navigation function to guarantee that all features willremain in the camera field of view. The control objective inthis paper is stated as follows:
Control Objective: The vision camera follows thetarget object, that is, the relative pose gco(t) coincides withthe time-varying desired one gcd(t) which is generated tokeep all features within the camera field of view, and whichconverges to the final desired one gcdf.

Differentiating Eq. (18), the relation between the desired
image features fcdi and the desired feature point pcdi can be
expressed as(19)
Moreover, we have the following relation between thedesired feature point pcdi and the body velocity Vbcd throughpcdi = gcdpoi:(20)
Hence, the desired image feature vector and the desired
body velocity can be related as
(21)

15

where JL(gcd) : SE(3) → R8 × 6 is defined as follows:

where we use the property k|a|2 ≤ kcdaT(JTLJL)−1a, Wa ∈ R6,
and k denotes a positive constant. It is clear from Eq. (27)that Vn(fcd) is a nonincreasing function in the sense that

(22)

(28)
From Eqs. (26) to (28), the condition fcd(t) ∈ D, Wt > 0 is
satisfied for any initial condition fcd(0) ∈ D. By LaSalle’s
Theorem, it can be proved that the only invariant set thatsatisfies |JTL(fcd)∇ϕ(fcd)| = 0 is the origin. Considering theassumption ∇ϕ(fcd) ∉ NS(JTL(fcd)), we have shown that|∇ϕ(fcd)| = 0. By the argument in the Appendix, it can beshown that fcd(t)→ fcdf. Since only one geometric solutionof gcd(t) exists due to the fact that we set four feature points
on the target object, it can be concluded that gcd(t) → gcdf.
"
Theorem 2 guarantees the convergence of the timevarying desired image feature vector fcd(t) to the final one
fcdf. The path planner can be designed to keep all features
within the camera field of view based on the image space
navigation function. A block diagram of the visual motion
stabilizing receding horizon control with the image space
navigation function-based path planner is shown in Fig. 3.
Although position-based control can allow featurepoints to leave the field of view, its principal advantages arethat it is possible to describe tasks in terms of Cartesianpose, as is common in many applications, such as robotics[1], and that it does not need a desired image a priori. Thus,
the proposed method which combines the position-based
receding horizon control and the image-based path planner
allows us to extend the technological application area. The
main contribution of this paper is to present the path planner
which always remains in the camera field of view during
servoing, which is designed for the position-based visualfeedback receding horizon control based on optimal control
theory.
It is also interesting to note that the Jacobian JL(⋅)
between the image feature vector and the body velocity has
exactly the same form as the Jacobian Je(⋅) for the estimation error, which is derived by using a first Taylor expansion
approximation. It leads to application to visual feedback
systems with a panoramic camera using the Jacobian J(⋅) in
Ref. 18.

(23)
Inspired by Eq. (18) and the definition of the navigation function, the desired body velocity Vbcd is designed asfollows:(24)
where ∇ϕ(fcd) := (∂ϕ(fcd) / ∂fcd)T ∈ R8 denotes the gradientvector of ϕ(fcd) and kcd ∈ R is the positive gain. The designof the image navigation function ϕ(fcd) is stated in the
Appendix.
The stabilizing predictive visual feedback control law
can be applied to the system by using the desired body
velocity Vbcd given by Eq. (24) and the desired relative pose
gcd which is obtained by solving Eq. (17).
3.2 Convergence analysis for path planner
Substituting Eq. (20) into Eq. (18), we obtain for thevelocity of the desired image feature vector(25)
Similarly to Ref. 8, it is assumed that∇ϕ(fcd) ∉ NS(JTL(fcd)), where NS(⋅) denotes the null space
operator. Since fcd is chosen a priori via the off-line path
planning routine in Eq. (25), this assumption can be satisfied.
Assuming that ∇ϕ(fcd) is not a member of the nullspace JTL(fcd), the following theorem concerning the convergence of the path planner holds.
Theorem 2 Suppose that ∇ϕ(fcd) ∉ NS(JTL(fcd)) andthe initial desired image feature vector fcd(0) satisfies
fcd(0) ∈ D. Then, the desired image feature vector (25)ensures that fcd(t) ∈ D and the desired relative pose gcd(t)has the asymptotically stable equilibrium point gcdf.
Proof: Consider the following positive definite function:(26)
Evaluating the time derivative of Vn(fcd) along the trajectories of Eq. (25) gives

Fig. 3. Block diagram of stabilizing predictive visualfeedback control with image space navigation
function-based path planner.

(27)

16

Our previous work has proposed stabilizing receding
horizon control for visual feedback systems with manipulator dynamics [13, 14]. In a similar way, our proposed
approach can be applied to robot systems which must be
controlled with a small sampling period.
4. Simulation Results
In this section, we present simulation results for thevisual feedback control with the path planner via the image
space navigation function, compared with the constantdesired motion proposed in Ref. 13.
The simulation was carried out with the initial conditions pco = [0.2 0.2 −1.35]T m, ξθco = [0 0 0]T rad. The
final desired relative pose was pcdf = [−0.22 −0.31 −1.35]Tm, ξθcdf = [0 0 π / 2]T rad. This condition means that thevision camera moves from end to end of the field of view
diagonally with optical axis rotation. We set the maximum
and minimum pixel values of the vision camera as fxM = 240
pixels, fxm = –240 pixels, fyM = 320 pixels, fym = –320 pixels.
The weights of the cost function (12)–(14) were selected as
qpc = 0.003, qrc = 0.001, qpe = 0.0003, qre = 0.0001, and R= diag{0.105, 0.105, 15, 15, 15, 0.105, 300, 300, 30, 30, 30,300} and ρ = 1 satisfied ρ2I ≥ 4QR. To solve the real-time
optimization problem, the software C/GMRES [19] wasutilized. The control input with receding horizon control
was updated every 2 ms and had to be calculated by thereceding horizon controller within that period. The horizon
was selected as T = 0.04 s. The parameters for the trajectory
generation were selected as kcd = 6000000, Ks = 0.1I, κ =2.
The simulation results are presented in Figs. 4 to 6.
In Fig. 4, the norm of the state x applying the proposed
control law is shown. The asymptotic stability can be confirmed by steady-state performance. Figure 5 shows theFig. 5. Trajectory of image feature points (solid: withproposed method; dashed: with previous one [13]).
[Color figure can be viewed in the online issue, which isavailable at wileyonlinelibrary.com.]trajectory of the image feature points fco. In Fig. 5, the solidlines denote the trajectory applying the proposed stabilizing
receding horizon control with the path planner, and thedashed lines denote those for the constant desired value[13], respectively. fco(0) := [fT1 (0) fT2 (0) fT3(0) fT4(0)]T andfco(1.5) := [fT1 (1.5) fT2(1.5) fT3(1.5) fT4 (1.5)]T show the valuesof the image feature vector in the case of the initial conditions and in the case of t = 1.5 s, respectively. The control
method must be designed so that the feature points do notleave the camera field of view, which is shown by a rectangle in Fig. 5. From Fig. 5 it is concluded that the proposed

Fig. 4. Norm of the state. [Color figure can be viewed
in the online issue, which is available atwileyonlinelibrary.com.]

Fig. 6. Pose control error. [Color figure can be viewed
in the online issue, which is available atwileyonlinelibrary.com.]17

Table 1. Values of the integral cost with simulation
results

method can make the vision camera keep all feature points
in the field of view. Although convergence to the desired
values was also achieved by the previous method [13] in thesimulation, it corresponds to failure in the actual experiment because the vision camera misses the target object.
Figure 6 shows the actual pose control error er, whichis the error vector between the current relative pose gco(t)and the final desired one gcdf, instead of the time-varying
desired one gcd(t). It can be confirmed that all errors converge to zero by steady-state performance. It should be
noted that the position error with the z-axis and the rotation
error with the x-axis and y-axis increases, while the others
are monotonically decreasing. This means that the vision
camera moves away and changes orientation once in orderto keep the target object in the camera field of view. Thisvalidates one of the expected advantages of stabilizing
receding horizon control with the path planner for the visualfeedback system.
Next the performance for the horizon length T iscompared in terms of the integral cost in Table 1. The costis calculated by the function

Fig. 7. Mobile robot and target object. [Color figure
can be viewed in the online issue, which is available atwileyonlinelibrary.com.]controlled by a digital signal processor (DSP) DS1104(dSPACE Inc.) whose sampling time was 10 ms. The control signal was transmitted through XBees.
Due to environmental constraints, we performed theexperimental verification with non-optical-axis rotation,
but the simulation verification with optical axis rotation. The
experiment was carried out with the following conditions,parameters, and weights: pco = [0.2 0 −1.3]T m, ξθco =[0 π / 12 0]T rad, pcdf = [0.03 0 −0.6] T m, ξθcdf = [0 0 0]T rad,fxM = 350 pixels, fxm = –350 pixels, fyM = 200 pixels, fym =–200 pixels, qpc = 0.008, qrc = 0.005, qpe = 0.2, qre = 0.15,
R = diag{1.5, 1.5, 1.75, 0.8, 0.8, 0.8, 0.05, 0.05, 0.05, 0.06,
0.06, 0.06}, ρ = 1, T = 0.05 s, kcd = 140000, Ks = diag{0.1,0.05, 0.1, 0.05, 0.1, 0.05, 0.1, 0.05}, κ = 2.
The experimental results are presented in Figs. 8 to
10. The pose control error and the estimation error are
shown on the left and right sides of Fig. 8, respectively. We
focus on the errors in the translation of x and z and therotation of y, because the errors in the translation of y andthe rotations of x and z are ideally zero in the defined(29)
We set the integration interval I = 1.5 in this simulation.
Since the cost of the stabilizing receding horizon method issmaller than that of the passivity-based visual feedback
control method under conditions of the adequate cost function, it can be easily verified that the control performanceis improved. As the horizon length increases from T = 0.04
s to T = 1 s, the cost is reduced. In the case of T = 10 s, thecalculation cannot be completed within one sampling interval, due to limited computing power.

5. Experimental Results
In this section, we present experimental results withan omnidirectional mobile robot from Tosa Denshi, Ltd., asshown in Fig. 7. We set up a KMT-1607N (30 fps) cameraon the mobile robot. The video signals were acquired by a
PicPort-Stereo-H4D frame grabber board and the HALCON image processing software. The mobile robot wasFig. 8. Pose control error and estimation one. [Color
figure can be viewed in the online issue, which isavailable at wileyonlinelibrary.com.]18

Table 2. Values of the integral cost with experimental
resultsfunction. It can be verified that the relative pose tracks thedesired one, and that a time delay occurs. Figure 10 shows
the trajectory of the image features fco. In Fig. 10, the top
and bottom graphs show the trajectories obtained by theprevious method [13] and the proposed method, respectively. In the case of the previous method [13], the mobile
robot is controlled so as to reduce the errors in the Cartesian
space. However, the vision camera misses the target object
at 1.2 s because the control approach is not proposed on thebasis of the image space. As a result, the experiment fails
under this condition. On the other hand, it can be confirmedthat the vision camera moves to the desired value while
keeping all feature points in the camera field of view in thecase of the proposed method.
Finally, the performance with respect to the horizon
length T is compared in terms of the integral cost calculated
by Eq. (29) with I = 15 in Table 2. Similarly to the simulation results, the cost is reduced as the horizon length increases. Therefore, the validity of the stabilizing predictive
visual feedback control which can keep all feature points inthe camera field of view is confirmed by the experimental
results.

Fig. 9. Estimated relative pose. [Color figure can beviewed in the online issue, which is available atwileyonlinelibrary.com.]coordinates in Fig. 7. Since we set the weights and gains inorder not to reach the voltage limits of the mobile robot, theconvergence time of the experimental results is longer thanthat of the simulation results. Although a slight error remains in the steady state because of image noise and thefriction force with the robot, it can be confirmed that allerrors converge to zero.
In Fig. 9, the solid and dashed lines respectively
represent the estimated relative pose and the desired relativepose generated on the basis of the image space navigation

6. Conclusions
This paper proposes stabilizing receding horizon
control via an image space navigation function for 3D
eye-in-hand visual feedback systems. Regarding the image
space navigation function as a Lyapunov function, convergence analysis for the proposed path planner is provided.
The main contribution of this paper is to show that a path
planner which always remains in the camera field of view
during the servoing is designed for the position-based visual feedback receding horizon control based on optimal
control theory. Simulation and nonlinear experimental results are presented to verify the control performance withvisibility maintenance of the proposed control scheme.
Fig. 10. Trajectory of image feature points (top: withprevious method [13]; bottom: with proposed one).
[Color figure can be viewed in the online issue, which isavailable at wileyonlinelibrary.com.]

REFERENCES
1. Chaumette F, Hutchinson SA. Visual servoing andvisual tracking. In Siciliano B, Khatib O (editors).

1917. Bullo F, Lewis AD. Geometric control of mechanicalsystems. Springer-Verlag; 2004.
18. Kawai H, Murao T, Fujita M. Passivity-based visualmotion observer with panoramic camera for posecontrol. J Intell Robot Syst 2011;64(3–4):561–583.
19. Ohtsuka T. A continuation/GMRES method for fastcomputation of nonlinear receding horizon control.
Automatica 2004;40:563–574.

Springer handbook of robotics. Springer-Verlag;
2008. p 563–583.
2. Gans NR, Hutchinson SA. Stable visual servoing
through hybrid switched-system control. IEEE Trans
Robot 2007;23:530–540.
3. Lippiello V, Siciliano B, Villani L. Position-based
visual servoing in industrial multirobot cells using a
hybrid camera configuration. IEEE Trans Robot
2007;23:73–86.
4. Mansard N, Chaumette F. Directional redundancy for
robot control. IEEE Trans Automatic Control
2009;54:1179–1192.
5. Chaumette F. Potential problems of stability and convergence in image-based and position-based visual
servoing. In Kriegman DJ, Hager GD, Morse AS(editors). The confluence of vision and control. Lect
Notes Control Inf Sci 1998;237:66–78.
6. Cowan NJ, Weingarten JD, Koditschek DE. Visual
servoing via navigation functions. IEEE Trans Robot
Automation 2002;18:521–533.
7. Cowan NJ, Chang DE. Geometric visual servoing.
IEEE Trans Robot 2005;21:1128–1138.
8. Chen J, Dawson DM, Dixon WE, Chitrakaran VK.
Navigation function-based visual servo control.
Automatica 2007;43:1165–1177.
9. Mayne DQ, Rawlings JB, Rao CV, Scokaert POM.
Constrained model predictive control: Stability andoptimality. Automatica 2000;36:789–814.
10. Sauvée M, Poignet P, Dombre E, Courtial E. Image
based visual servoing through nonlinear model predictive control. Proc 45th IEEE Conference on Decision and Control, p 1776–1781, 2006.
11. Sauvée M, Poignet P, Dombre E. Ultrasound imagebased visual servoing of a surgical instrument
through nonlinear model predictive control. Int J
Robot Res 2008;27:25–40.
12. Allibert G, Courtial E, Chaumette F. Predictive control for constrained image-based visual servoing.
IEEE Trans Robot 2010;26:933–939.
13. Murao T, Kawai H, Fujita M. Predictive visual feedback control with eye-in/to-hand configuration via
stabilizing receding horizon approach. Proc 17th
IFAC World Congress on Automatic Control, p
5341–5346, 2008.
14. Murao T, Kawai H, Fujita M. Stabilizing predictive
visual feedback control for fixed camera systems.
Electron Commun Jpn 2011;94:1–11.
15. Murray R, Li Z, Sastry SS. A mathematical introduction to robotic manipulation. CRC Press; 1994.
16. Fujita M, Kawai H, Spong MW. Passivity-based dynamic visual feedback control for three dimensionaltarget tracking: Stability and L2-gain performance
analysis. IEEE Trans Control Syst Technol
2007;15:40–52.

APPENDIXThe Image Space Navigation Function
This section presents the image space navigation
function ϕ(fcd) [6, 8]. The image space navigation function
is designed so that all image features remain in the visible
set.

(30)
(31)
First, we define two auxiliary functions
η(fcd) : R8 → [−1, 1]8 and s(η) : [−1, 1]8 → R8 as follows:
where η(fcd) = [η1(fcd) η2(fcd) . . . η8(fcd)]T, and κ > 0 ∈ R
is an additional parameter to change the potential field.
η(fcdf) = [ηf1(fcd) ηf2(fcd) . . . ηf8(fcd)]T : R8 → [−1, 1]8 is defined as in Eq. (30). fxM, fxm, fyM and fym ∈ R denote themaximum and minimum pixel values along the x- andy-axes, respectively. η and s are functions used to normalize
the current pixel value for the maximum and minimum
pixel values, and to define the error between the current
image features and the final one. The model space naviga~
tion function φ(η) ∈ R8 → [0, 1] is then defined as(32)__
The objective function ϕ(η) ∈ R8 → R is defined as(33)
where Ks ∈ R8 × 8 is a positive definite symmetric matrix.
The image space navigation function by
ϕ(fcd) ∈ D → R can be developed as follows:(34)
where ! is the composition operator. The gradient vector∇ϕ(fcd) can be represented as

20

It should be noted that fcd → fcdf from Eqs. (30) to (35) when∇ϕ(fcd) → 0.

(35)

AUTHORS (from left to right)

Toshiyuki Murao received his B.E. degree from the Department of Electrical and Computer Engineering of Kanazawa
University in 2003, M.E. degree from the Division of Electronics and Computer Science of Kanazawa University in 2005, andD.Eng. degree from the Department of Mechanical and Control Engineering of Tokyo Institute of Technology in 2008. Since
2006, he has been an assistant professor at the Advanced Institute of Industrial Technology. His research focuses on
passivity-based visual feedback control of robot systems.
Hiroyuki Kawai received his B.S. degree from the Department of Electrical and Computer Engineering of Kanazawa
University in 1999, and M.S. and Ph.D. degrees from the Graduate School of Natural Science and Technology of Kanazawa
University in 2001 and 2004. From 2004 to 2005 he was a postdoctoral scholar at the Information Technology Research Center,
Hosei University. In 2005 he joined Kanazawa Institute of Technology as an assistant professor. Since 2010 he has been an
associate professor in the Department of Robotics, Kanazawa Institute of Technology. His research focuses on visual feedback
control of robot systems and networked cellular vision systems. In particular, he is interested in passivity-based control. He
received an IEEE Transactions on Control Systems Technology Outstanding Paper Award from the IEEE Control Systems
Society (CSS) in 2008.
Masayuki Fujita is a professor in the Department of Mechanical and Control Engineering of Tokyo Institute of Technology.
He is also a program officer of the Japan Science and Technology Agency (JST) Core Research for Evolutional Science andTechnology (CREST). He received his D.Eng. degree in electrical engineering from Waseda University in 1987. Prior to his
appointment at Tokyo Institute of Technology, he held faculty appointments at Kanazawa University and the Japan Advanced
Institute of Science and Technology. His research interests include passivity-based control in robotics and applied robust control.
He is currently the CSS Vice President for Conference Activities. He was a member of the CSS Board of Governors. He serves
as Head of the SICE Technical Division on Control and previously served as chair of the SICE Technical Committee on Control
Theory and as a director of SICE. He served as General Chair of the 2010 IEEE Multi-conference on Systems and Control. His
past and present positions include associate editorships of IEEE Transactions on Automatic Control, IEEE Transactions on
Control Systems Technology, Automatica, and the Asian Journal of Control, and the editorship of the SICE Journal of Control,
Measurement, and System Integration. He is a recipient of a 2008 IEEE Transactions on Control Systems Technology
Outstanding Paper Award. He also received a 2010 SICE Education Award and the Outstanding Paper Award from the SICE.

21