13th International Workshop on
Dynamics & Control
DaimlerChrysler Training Center, Wiesensteig, Germany
May 22 – 26, 2005

Passivity-based Control of Visual Feedback
Systems with a Movable Camera
Masayuki Fujita
Division of Electrical Engineering and Computer Science, Kanazawa University,Japanfujita@t.kanazawa-u.ac.jp

Hiroyuki Kawai
Information Technology Research Center, Hosei University,
Japan

Toshiyuki Murao
Division of Electrical Engineering and Computer Science, Kanazawa University,
Japan
Abstract. This paper deals with the visual feedback control with a movable camera insteadof a fixed camera in the fixed camera configuration. Firstly the brief summary of the visualfeedback system with a fixed camera is given with the fundamental representation of a relative
rigid body motion. Secondly we construct the new error system in order to enlarge the field ofview. Next, we derive the passivity of the visual feedback system. Finally, stability and L2 -gain
performance analysis are discussed based on the passivity and the dissipative systems theory.

1

Introduction

Robotics and intelligent machines need many information to behave autonomously under dynamical environments. Speciﬁcally, the combination of mechanical control withvisual information, so-called visual feedback control or visual servoing, should become
extremely important, when we consider a mechanical system working under dynamical
environments [1].
In classical visual servoing, many practical methods are reported by two well known approaches with two camera conﬁgurations, i.e., position-based visual feedback control andimage-based one with an eye-in-hand conﬁguration or a ﬁxed camera one (see, e.g. [1]).
Kelly et al. [2] considered a simple image-based controller for visual feedback system inthe three dimensional(3D) workspace under the assumption that the objects’ depths areM. Fujita, H. Kawai, and T. Murao

Image

Cameraxi

Robot
Manipulator

gch

5c

fyi

gwc
gco

zo

5w

P
poi

World Frame

xo

Target Object

So

Target Object

Figure 1: Visual feedback system in the ﬁxed
camera conﬁguration.

Sc
yc
yi

pci
gco

gho
5o

zc

l

fi

5h
gwh

gwo

Camera Frame
xc

yo

fxi
zw

Image Plane
xw
yw

Sw

World Frame

Figure 2: Pinhole camera

known. Zergeroglu et al. developed an adaptive control law for the position tracking andthe camera calibration problems of the visual feedback system with parametric uncertainties in [3]. Cowan et al. [4] addressed the problems of the ﬁeld of view for the visualfeedback system by using the navigation functions. Although the good solutions to the
set-point problems are reported in those papers, few results have been obtained for the
tracking problems of the moving target object in the visual feedback system. Additionally, most of the previous works are discussed for the camera conﬁgurations separately,while the position-based visual feedback control and the image-based one are combinedin some issues [5][6].
In this paper, we discuss the visual feedback control for the target tracking problem
with a movable camera instead of a ﬁxed camera in the ﬁxed camera conﬁguration as inFig. 1. While the objective of this system is obviously to control the end-eﬀector of themanipulator, we also control the camera in order to enlarge the ﬁeld of view. Moreover, we
can derive that the visual feedback system with a movable camera preserves the passivityof the visual feedback system which is obtained in our previous works [7][8]. Stability
and L2 -gain performance analysis are discussed based on the passivity and the dissipative
systems theory with the energy function.
ˆ
Throughout this paper, we use the notation eξθab ∈ R3×3 to represent the change of theprinciple axes of a frame Σb relative to a frame Σa . The notation ‘∧’ (wedge) is the skewˆ = ξ × θ for the vector cross-product × and any vector
symmetric operator such that ξθ3
θ ∈ R . The notation ‘∨’ (vee) denotes the inverse operator to ‘∧’: i.e., so(3) → R3.
ˆ abξab ∈ R3 speciﬁes the direction of rotation and θab ∈ R is the angle of rotation. Here ξθdenotes ξˆab θab for the simplicity of notation. We use the 4 × 4 matrix ˆeξθab pab
(1)
gab =
0
1
ˆ

as the homogeneous representation of gab = (pab , eξθab ) ∈ SE(3) which is the descriptionof the conﬁguration of a frame Σb relative to a frame Σa . The adjoint transformation
associated with gab is denoted by Ad(gab) [9]. Let us deﬁne the vector form of the rotation
ˆ
ˆ
ˆ
ˆ
ˆ
matrix as eR(eξθab ) := sk(eξθab )∨ where sk(eξθab ) denotes 12 (eξθab − e−ξθab ).

Passivity-based Control of Visual Feedback Systems with a Movable Camera

2

Passivity-based Visual Feedback System in the Fixed
Camera Configuration

2.1

Fundamental Representation for Visual Feedback System

Visual feedback systems typically use four coordinate frames which consist of a world
frame Σw , a target object frame Σo , a camera frame Σc and a hand (end-eﬀector) frameΣh as in Fig. 1. Then, gwh , gwc and gwo denote the rigid body motions from Σw to Σh ,from Σw to Σc and from Σw to Σo , respectively. Similarly, the relative rigid body motions
from Σc to Σh , from Σc to Σo and from Σh to Σo can be represented by gch , gco andgho , respectively, as shown in Fig. 1. Here, it is supposed that the relative rigid body
motion from Σw to Σc , i.e. gwc can be measured exactly. Since gwh is known by the angleof manipulator, gch can be also available from using the composition rule for rigid body−1
gwh . Thus, gwc , gwh and gch
transformations ([9], Chap. 2, pp. 37, eq. (2.24)) as gch = gwc
are known information in the visual feedback system as in Fig. 1.
−1
gwo . Then, theThe relative rigid body motion from Σc to Σo can be led as gco = gwc
fundamental representation of the relative rigid body motion gco is described as follows
[7].
b
b
−1 V
Vcob = −Ad(gco) wc + Vwo

(2)

b
b
and Vwo
are the body velocity of the camera and the target object relative towhere Vwc
Σw , respectively. Roughly speaking, the relative rigid body motion gco will be derivedbband the target object velocity Vwo.
from the diﬀerence between the camera velocity Vwc

2.2

Camera Model and Estimation Error System

Next, we derive the model of a pinhole camera with a perspective projection as shownin Fig. 2. Let λ be a focal length, poi ∈ R3 and pci ∈ R3 be coordinates of the target
object’s i-th feature point relative to Σo and Σc , respectively. Using a transformation of
the coordinates, we have pci = gco poi where pci and poi should be regarded as [pTci 1]T and[pToi 1]T via the well-known representation in robotics, respectively (see, e.g., [9]). The
perspective projection of the i-th feature point onto the image plane gives us the image
plane coordinate fi := [fxi fyi ]T ∈ R2 as follows


λ xci
(3)
fi =
zci yci
where pci := [xci yci zci ]T . It is straightforward to extend this model to the m image
points case by simply stacking the vectors of the image plane coordinate, i.e. f(gco ) :=T T
] ∈ R2m . We assume that multiple point features on a known object can[f1T · · · fm
be used.
The visual information f(gco ) which includes the relative rigid body motion can be
exploited, while the relative rigid body motion gco can not be obtained directly in thevisual feedback system. In order to bring the actual relative rigid body motion gho to agiven reference gd in Fig. 1, in addition to the control problem, we consider the estimation

M. Fujita, H. Kawai, and T. Murao

one in the visual feedback system. Firstly, we shall consider the following model whichjust comes from the fundamental representation (2).
b
−1 VV̄cob = −Ad(ḡco) wc + ue

(4)

ˆ
¯

where ḡco = (p̄co , eξθ̄co ) and V̄cob are the estimated value of the relative rigid body motion
and the estimated body velocity from Σc to Σo , respectively. ue is the input in order toconverge the estimated value to the actual relative rigid body motion. Because the designof ue needs a property of the whole visual feedback system, we will propose ue in Section 3.
The estimated image feature point f¯i (i = 1, · · · , m) should have the same form as (3).
In order to establish the estimation error system, we deﬁne the estimation error between−1
gco . Using
the estimated value ḡco and the actual relative rigid body motion gco as gee = ḡcoˆˆξθTthe notation eR(e ), the vector of the estimation error is given by ee := [pee eTR(eξθee )]T .
The vector of the estimation error ee can be derived from the following relationee = J † (ḡco )(f − f¯)(5)where J (ḡco ) is deﬁned in [7] and † denotes the pseudo-inverse. Therefore the estimation
error ee can be exploited in the 3D visual feedback control law using image information
f obtained from the camera. Hence, the nonlinear observer is constructed by (4) and theestimation input ue which can be determined from ee in (5) with an estimation gain in
Section 3.3. Then, the estimation error system is represented byb−1 ue + V
Veeb = −Ad(gee
wo .
)

(6)

It should be noted that if the vector of the estimation error is equal to zero, then theestimated relative rigid body motion ḡco equals the actual one gco .

2.3

Visual Feedback System in the Fixed Camera Configuration

Similar to the estimation error system, we consider the control error system. Because gco
can not be obtained directly, we represent the relative rigid body motion from Σh to Σo−1with the estimated one ḡco as ḡho = gchḡco . Diﬀerentiating ḡho with respect to time, theestimated body velocity from Σh to Σo can be obtained as follows.
bbV̄ho= −Ad(ḡ−1 ) Vwh+ ue .
ho(7)bis the body velocity of the hand relative to Σw . Similarly, we deﬁne the error
where Vwh
between gd and ḡho , which is called the control error, as gec = gd−1 ḡho . The vector of theˆcontrol error is deﬁned as ec := [pTec eTR(eξθec )]T . The control error system is described bybb−1 V
Vecb = −Ad(ḡ−1 ) Vwh+ ue − Ad(gec
) d
ho

(8)

where Vdb is the desired body velocity of the relative rigid body motion gho .
Combining (6) and (8), the visual feedback system in the ﬁxed camera conﬁgurationis constructed as follows b    bb
−Ad(ḡ−1 )
I
+
Ad
V
Vec
0
V
(g
)
b
wh
d
d
ho
=
uce +
.
(9)
V , uce :=
Veeb
−1
ue
I wo
0
−Ad(gee)Passivity-based Control of Visual Feedback Systems with a Movable CameraT
Let us deﬁne the error vector of the visual feedback system as ece := eTc eTe
whichcontains of the control error vector ec and the estimation error vector ee . Here, we deﬁne
the output of the visual feedback system (9) as follows
νce :=−AdT(g−1 )dAd(e−ξ̂θec )

0
−I


ece .

T
Then the visual feedback system (9) satisﬁes 0 uTce νce dτ ≥ −βce where βce is a positive
scalar [8]. This would suggest that the visual feedback system (9) is passive from the
input uce to the output νce just formally as in the deﬁnition in [10].

3Visual Feedback System with a Movable Camera

3.1

Camera Field Error System

In this section, we construct the error system of the movable camera in the ﬁxed camera
conﬁguration, we call the camera ﬁeld error system, in order to increase the available
workspace for the robot hand. Here we deﬁne the camera ﬁeld error between the estimated−1ḡco . If ḡco is equal tovalue ḡco and a given reference gcd for the camera motion as gev = gcd
gcd , then the target object can be kept in the center of the camera ﬁeld of view. Using theˆˆnotation eR (eξθ ), the vector of the camera ﬁeld error is deﬁned as ev := [pTev eTR(eξθev )]T .
ˆNote that ev = 0 iﬀ pev = 0 and eξθev = I3. Similarly to (6) and (8), the camera ﬁeld
error system can be obtained as
b
b
−1 V
−1 V
Vevb = ue − Ad(ḡco) wc − Ad(gev
) cd

(10)

where Vcdb is the desired body velocity of the relative rigid body motion gco .

3.2

Property of Visual Feedback System

Combining (6), (8) and (10), we construct the visual feedback system with a movable
camera in the ﬁxed camera conﬁguration as follows
⎤

 ⎡

 

b
Vec
b
Vee
b
Vev

=⎣

−Ad(ḡ−1 )I
00ho⎦ucev + I
−1
0 −Ad(gee0)
00
I −Ad(ḡco
−1
)

b
Vwo
,

ucev :=b
Vwh+ Ad(gd ) Vdb
ue
b
b
Vwc
+ Ad(gcd ) Vcd. (11)


T
Let us deﬁne the error vector of the visual feedback system (11) as e := eTc eTe eTv . Itshould be noted that if the vector of the estimation error is equal to zero, not only ḡcoequals gco but also ḡho equals gho . Moreover, if the vectors of the control error and thecamera ﬁeld error are equal to zero, then ḡho and ḡco equal gd and gcd , respectively. Thus,
when e → 0, gho and gco tend to gd and gcd , respectively. This states that the control
objective can be achieved, in addition, the available workspace for the robot hand will
be increased by moving of the camera.

M. Fujita, H. Kawai, and T. Muraob
Lemma 1 If Vwo= 0, then the visual feedback system (11) satisﬁes⎡
T
T0

uTcev νcev dτ ≥ −βcev , ∀T > 0,

−Ad(g−1 )d⎣νcev := Ncev e = Ad(e−ξ̂θec )
0

0
−I
0

⎤

0

Ad(e−ξ̂θev ) ⎦e
−AdT(g−1 )

(12)

cd

where βcev is a positive scalar.
Proof Consider the following positive deﬁnite function
Vcev = E(gec ) + E(gee ) + E(gev )
ˆ

ˆ

(13)

ˆ

where E(g) := 12 p2 + φ(eξθ ) and φ(eξθ ) := 12 tr(I − eξθ ) which is the error function of therotation matrix. Diﬀerentiating (13) with respect to time yields
⎡

⎤⎡
⎤
b
Ad(eξ̂θec )
0
0
Vec
⎢
⎥⎣ b ⎦0Ad(eξ̂θee )0V̇cev = eT ⎣
⎦ Vee
b
Vev
0
0
Ad(eξ̂θev )
ˆ

(14)

ˆ

where we use the property φ̇(eξθ ) := eξθ ω. Observing the skew-symmetry of the matricesp̂ec , p̂ee and p̂ev , the above equation along the trajectories of the system (11) can betransformed intoV̇cev = uTcev νcev .

(15)

Integrating (15) from 0 to T , we can obtainT0

uTcev νcev dτ ≥ −Vcev (0) := −βcev(16)where βcev is the positive scalar which only depends on the initial states of gec , gee and
2
gev .
ˆ

Remark 1 In the visual feedback system, pTec (e−ξθd ωec )∧ pec = 0, pTee ω̂ee pee = 0,
ˆ
pTev (e−ξθcd ωev )∧ pev = 0 hold. Let us take ucev as the input and νcev as its output in Fig.
3. OMFC and HMFC represent the object motion relative to the camera frame Σc andthe hand motion relative to the camera frame Σc , respectivery. Lemma 1 suggests thatthe visual feedback system (11) is passive from the input ucev to the output νcev as in thedeﬁnition in [10].

3.3

Stability Analysis for Visual Feedback System

It is well known that there is a direct link between passivity and Lyapunov stability.
Thus, we propose the following control input.

ucev = −Kcev νcev = −Kcev Ncev e, Kcev :=

Kc
0
0

0
Ke
0

0
0
Kv

(17)

Passivity-based Control of Visual Feedback Systems with a Movable Camera

+

b
Vwh

Ad(gd )Vdb
ue

ucev

Visual Feedback System

Estimated
HMFC

gho +

gho

Estimated
OMFC

gd
gco , f

gco

b
Vwo

+

b
Vwc

Ad(gcd )Vcdb

OMFC

gco

Image Jacobian
JO

ec
ee
ev

ncev

gco +
f

Ncev

gcd

Figure 3: Block diagram of the visual feedback system

where Kc := diag{kc1 , · · · , kc6 }, Ke := diag{ke1 , · · · , ke6 } and Kv := diag{kv1 , · · · , kv6 }are the positive gain matrices of x, y and z axes of the translation and the rotation forthe control error, the estimation one and the camera ﬁeld one, respectively. The resultwith respect to asymptotic stability of the proposed control input (17) can be establishedas follows.
b= 0, then the equilibrium point e = 0 for the closed-loop system (11)Theorem 1 If Vwo
and (17) is asymptotic stable.

Theorem 1 can be proved using the energy function (13) as a Lyapunov function. It
is interesting to note that stability analysis is based on the passivity as described in (12).

3.4

L2 -gain Performance Analysis for Visual Feedback System

Based on the dissipative systems theory, we consider L2 -gain performance analysis for thevisual feedback system (11) in one of the typical problems, i.e. the disturbance attenuation
problem. Now, let us deﬁne
T
Kcev Ncev −P := Ncev

1
1
W− I
2
2γ
2

(18)

where γ ∈ R is positive and W := diag{0, I, 0}. Then we have the following theorem.
Theorem 2 Given a positive scalar γ and consider the control input (17) with the gains
Kc , Ke and Kv such that the matrix P is positive semi-deﬁnite, then the closed-loopsystem (11) and (17) has L2 -gain ≤ γ.
Theorem 2 can be proved using the energy function (13) as a storage function forL2 -gain performance analysis. The L2 -gain performance analysis of the visual feedback
system is discussed via the dissipative systems theory. In H∞ -type control, we can consider
some problems by establishing the adequate generalized plant. This paper has discussed
L2 -gain performance analysis for the disturbance attenuation problem. The proposed
strategy can be extended for the other-type of generalized plants of the visual feedback
systems.

M. Fujita, H. Kawai, and T. Murao

4

Conclusions

This paper dealt with the visual feedback control with a movable camera instead of a ﬁxed
camera in the ﬁxed camera conﬁguration in order to increase the available workspace forthe robot hand. Moreover, we derived that the visual feedback system preserved thepassivity of the visual feedback system by the same strategy in our previous works, [7],[8]. Stability and L2 -gain performance analysis for the visual feedback system have been
discussed based on passivity with the energy function.

References[1] S. Hutchinson, G. D. Hager and P. I. Corke, “A Tutorial on Visual Servo Control,”IEEE Trans. Robotics and Automation, Vol. 12, No. 5, pp. 651–670, 1996.
[2] R. Kelly, R. Carelli, O. Nasisi, B. Kuchen and F. Reyes, “Stable Visual Servoing of
Camera-in-Hand Robotic Systems,” IEEE Trans. Mechatronics, Vol. 5, No. 1, pp. 39–
48, 2000.
[3] E. Zergeroglu, D. M. Dawson, M. S. de Queiroz and A. Behal, “Vision-Based Nonlinear Tracking Controllers With Uncertain Robot-Camera Parameters,” IEEE/ASME
Trans. Mechatronics, Vol. 6, No. 3, pp. 322–337, 2001.
[4] N. J. Cowan, J. D. Weingarten and D. E. Koditschek, “Visual Servoing via Navigation
Functions,” IEEE Trans. Robotics and Automation, Vol. 18, No. 4, pp. 521–533, 2002.
[5] E. Malis, F. Chaumette and S. Boudet, “2-1/2-D Visual Servoing,” IEEE Trans.
Robotics and Automation, Vol. 15, No. 2, pp. 238–250, 1999.
[6] P. I. Corke and S. A. Hutchinson, “A New Partitioned Approach to Image-Based
Visual Servo Control,” IEEE Trans. on Robotics and Automation, Vol. 17, No. 4, pp.
507–515, 2001.
[7] H. Kawai and M. Fujita, “Passivity-based Dynamic Visual Feedback Control for Three
Dimensional Target Tracking:Stability and L2 -gain Performance Analysis,” Proc. ofthe 2004 American Control Conference, pp. 1522–1527, 2004.
[8] H. Kawai, T. Murao and M. Fujita, “Passivity-based Control and Estimation of Dynamic Visual Feedback Systems with a Fixed Camera,” Proc. of the 43rd IEEE Conf.
on Decision and Control, pp. 4022–4027, 2004.
[9] R. Murray, Z. Li and S. S. Sastry, A Mathematical Introduction to Robotic Manipulation, CRC Press, 1994.
[10] A. van der Schaft, L2 -Gain and Passivity Techniques in Nonlinear Control (2nd ed.),Springer–Verlag, 2000.