Noname manuscript No.
(will be inserted by the editor)

Passivity-based Visual Motion Observer with Panoramic
Camera for Pose Control
Hiroyuki Kawai ´ Toshiyuki Murao ´
Masayuki Fujita

Received: date / Accepted: date

Abstract This paper considers the vision-based estimation and pose control with apanoramic camera via passivity approach. First, a hyperbolic projection of a panoramic
camera is presented. Next, using standard body-attached coordinate frames (the world
frame, mirror frame, camera frame and object frame), we represent the body velocityof the relative rigid body motion (position and orientation). After that, we propose avisual motion observer to estimate the relative rigid body motion from the measured
camera data. We show that the estimation error system with a panoramic camera has
the passivity which allows us to prove stability in the sense of Lyapunov. The visualmotion error system which consists of the estimation error system and the pose control
error system preserves the passivity. After that, stability and L2 -gain performance
analysis for the closed-loop system are discussed via Lyapunov method and dissipative
systems theory, respectively. Finally, simulation and experimental results are shown inorder to conﬁrm the proposed method.
Keywords Visual Feedback Control · Panoramic Camera · Passivity · Lyapunov
Stability

1 Introduction
Vision gives us rich information in order to respond to surrounding motions. Mechanical systems also need many information in an eﬃcient manner autonomously. The
H. Kawai
Department of Robotics, Kanazawa Institute of Technology, Ishikawa, Japan
E-mail: hiroyuki@neptune.kanazawa-it.ac.jp
T. Murao
Master Program of Innovation for Design and Engineering, Advanced Institute of Industrial
Technology, Tokyo, Japan
E-mail: murao-toshiyuki@aiit.ac.jp
M. Fujita
Department of Mechanical and Control Engineering, Tokyo Institute of Technology, Tokyo,JapanE-mail: fujita@ctrl.titech.ac.jp

2
MirrorCameraMirrorMirrorCameraCameraMobile Robot

Mobile Robot

Mobile Robot

Fig. 1 Omnidirectional vision-based formation of mobile robots.

combination of mechanical control with visual information, so-called visual feedback
control or visual servoing, is important when we consider a mechanical system working
under dynamical environments [1]. Recently, Lippiello et al. [2] presented a positionbased visual servoing for a hybrid eye-in-hand/eye-to-hand multicamera conﬁgurationby using the extended Kalman ﬁlter and a multiarm robotic cell. Gans and Hutchinson[3] proposed hybrid switched-system control which utilizes image-based and positionbased visual feedback control. Hu et al. [4] considered homography-based robust visual
servo control for the uncertainty of the camera calibration. In our previous works, we
discussed dynamic visual feedback control for 3D target tracking based on passivity[5]–[7]. Although these previous works give us the new vision-based robot control theory systematically, most of the works use a simple perspective projection by a pinhole
camera. Thus, it was implicitly supposed that the target object exists on the directionof the optical center of the camera.
On the other hand, omnidirectional cameras are useful for recognizing unknown
surroundings widely. Geyer and Daniilidis [8] presented a unifying theory for all central panoramic systems, i.e., an equivalence of catadioptric and spherical projections.
Hadj-Abdelkader et al. [9] proposed a catadioptric image-based control strategy fornonholonomic robot in order to follow a 3D line. Mariottini et al. [10] reviewed theseveral epipolar geometry estimation algorithms by using an omnidirectional camera
and gave us Epipolar Geometry Toolbox which is a simulation environment with MATLAB. Fomena and Chaumette [11] considered improvements on modeling features forvisual servoing using a spherical projection. However, the relative pose between the
robot and the target object can not be obtained except at the desired pose, while thepose of a robot with a panoramic camera coincides with a desired one in these methods.
For the vision-based formation control of mobile robots with central panoramic cameras, Das et al. [12] discussed cooperative control of a group of nonholonomic mobile
robots by using an extended Kalman ﬁlter-based estimation algorithm. Vidal et al. [13]used motion segmentation techniques to estimate the position of each leader. However,stability analysis have not been discussed for both pose control and pose estimationin these works which dealt with the pose control by using a panoramic camera, while
practical methods have been proposed for the set point control problem. Although thevision-based pose synchronization has been proposed in our previous work [14], a perspective projection model of a pinhole camera is used in order to the estimate relative
pose.

3
Mirror
Frame

gmo

5m
gcm

(constant)
CameraFrame 5c

gco
gwm

gwo

5o
Object Frame(Target Robot)

gwc
Mobile Robot

5w
World Frame

Fig. 2 Coordinate frames for mobile robots.

This paper deals with visual motion observer by using a panoramic camera for posecontrol of mobile robot systems as depicted in Fig. 1. The advantage of our proposed
method is that the relative pose between the leader and the follower robot can beobtained by using a single panoramic camera in the around direction. Moreover, adesired pose between the leader and the follower robot can be given directly, while our
proposed method does not need a desired image a priori. The estimation error systemwith a panoramic camera has the passivity which allows us to prove stability in thesense of Lyapunov. The visual motion error system which consists of the estimation
error system and the pose control error system preserves the passivity. After that,stability and L2 -gain performance analysis for the closed-loop system are discussed.
In the proposed control law, we can design both the estimation and the control gain
independently in the same framework. Finally, simulation and experimental results areshown in order to conﬁrm the proposed method.

2 Panoramic Camera
In our previous works [5]–[7], it was implicitly supposed that the target object existedon the direction of the optical center of the camera. It was reported that there existeda case of missing a target object in the position-based visual servoing [15]. In orderto catch the leader robot in the whole space where the controlled follower robot will
move, we use a panoramic camera instead of a simple pinhole camera. Hereafter, theleader and the follower robot are regarded as a target object and a controlled robot
which has a panoramic camera, respectively, as shown in Fig. 2
Visual feedback systems by using a panoramic camera use four coordinate frames
which consist of a world frame Σw , a mirror frame Σm , a camera frame Σc , and an
object frame Σo as in Fig. 2. Let pmo ∈ R3 and eξ̂θmo ∈ SO(3) be the position vector
and the rotation matrix from the mirror frame Σm to the object frame Σo . Then, therelative rigid body motion (position and orientation) from Σm to Σo can be represented
by gmo = (pmo , eξ̂θmo ) ∈ SE(3) 1 . Similarly, gwm = (pwm , eξ̂θwm ), gwc = (pwc , eξ̂θwc )and gwo = (pwo , eξ̂θwo ) denote the rigid body motions from the world frame Σw to1

The notation of the homogeneous transform is denoted in Appendix A4zoHyperbolic
Mirror

yo

poi
zm

pmi

ym

pmh

5m

Mirror Frame

gcm

xm

h

gmo
zw

5o

gwo

xo

Object Frame
yw

gwm
5w

pch

xw

World Frame

2r
PinholeCameraszc

5c

Camera
Frame

yc
xc

Fig. 3 Panoramic camera model (pinhole camera and hyperbolic mirror).

the mirror frame Σm , from the world frame Σw to the camera frame Σc and from theworld frame Σw to the object frame Σo , respectively.

2.1 Hyperbolic Projection of Panoramic Camera
In this paper, we consider a panoramic camera which consists of a pinhole camera anda hyperbolic mirror as shown in Fig. 3. So, the pinhole camera catches reﬂected images
through the hyperbolic mirror. We ﬁrst review the panoramic camera model [10] inorder to represent an image feature in our framework.
Here, pwo is projected at s through the origin of the camera frame Σc , after being
projected at pmh through the origin of the mirror frame Σm as shown in Fig. 3. Let a
and b be the hyperbolic mirror parameters which satisfy
x2 + y 2(zmh + r)2− mh 2 mh = 1
2
a
b

(1)

√
with eccentricity r = a2 + b2 [10]. Eq. (1) means a constraint with respect to pmh .
From the well-known perspective projection, the projection s in the camera frame (see,Fig. 3) is given by
s=

1
Λpch
zch

(2)

where s := [ fx fy 1 ]T ∈ R3 , fx and fy are the coordinates of x-axis and y-axis
onto the image plane, respectively. In this paper, Λ is assumed as the ideal internal
calibration matrix of the pinhole camera and deﬁned as Λ := diag{λ, λ, 1} where λ isa focal length.

5From Fig. 3, the relation between the camera frame Σc and the mirror frame Σmcan be represented as
2 3
0
(3)
pcm = 4 0 5 eξ̂θcm = I3
2r
and it is assumed that these parameters are known. Moreover, pmh can be represented
as follows:
pmh = αpmo

(0 < α < 1)

(4)

Because αpmo has to satisfy the constraint (1), the following relation holds
2(αzmo + r)2
α2 x2mo + α2 ymo
−
= 1.
a2
b2

(5)

Solving Eq. (5) for α, we obtainα(pmo ) =b2 (rzmo + apmo )
2 − b2 z 2
a2 x2mo + a2 ymomo(6)where α(pmo ) represents that α depends on pmo explicitly.
From Eqs. (2)(3) and zch = zcm +zmh = 2r+α(pmo )zmo , the hyperbolic projectionof the panoramic camera can be represented as
“
”
1
s=Λ
α(pmo )pmo + pcm(7)2r + α(pmo )zmo
where we exploit the composition rule [16], i.e., pch = eξ̂θcm pmh + pcm .

2.2 Image Feature for Panoramic Camera
In order to know the relative rigid body motion gmo from the visual information, we
consider the feature points on the rigid target object 2 . From the hyperbolic projectionof the panoramic camera, we consider the obtainable visual information.
Let poi ∈ R3 and pmi ∈ R3 be the position vectors of the target object’s i-thfeature points (i = 1, · · · , n, (n ≥ 4)) relative to Σo and Σm , respectively (see Fig. 4).
Using a transformation of the coordinates, we havepmi = gmo poi(8)T
where pmi and poi should be regarded, with a slight abuse of notation, as [pT
mi 1]
Tand [pT1]
via
thewell-known
homogeneous
coordinate
representationinrobotics,
oirespectively (see, e.g., [16]).
The hyperbolic projection of the i-th feature point onto the image plane gives us
the image plane coordinate fi := [ fxi fyi ]T ∈ R2 asλα(pmi )fi =2r + α(pmi )zmi

»

xmi
ymi

–
(9)

2 The feature points on the rigid object are suitable for the estimation of the position andthe orientation in 3D workspace.

6
zo

zm

ym

gmo

5m

xm

Mirror Frame

yo

poi

pmi

xo
5o
Object Frame
zw
yw
5w
xw
World Frame

Fig. 4 Feature points on target object.

Panoramic Camera Projectionpoi (i=1,...,n)gmo

Feature pm
PointsHyperbolic
ProjectionfFig. 5 Block diagram of panoramic camera projection.

where α(pmi ) means that pmo in Eq. (6) is replaced with pmi = [xmi ymi zmi ]T . It is
straightforward to extend this model to n image points by simply stacking the vectorsof the image plane coordinate, i.e.,f (gmo ) := [f1T · · · fnT ]T ∈ R2n

(10)

T T
∈ R3n . We assume that multiple feature points on aand pm := [pT
m1 · · · pmn ]known object are given. Although the problem of extracting the feature points fromthe target object is interesting in its own right, we will not focus on this problem andmerely assume that the image feature are obtained by well-known techniques. From
Eq. (8) and Fig. 5 which shows the block diagram of the panoramic camera projection,
the image feature f only depends on the relative rigid body motion gmo .

2.3 Body Velocity for Panoramic Camera
The relative rigid body motion gmo is discussed in this section, because the image
feature f only depends on gmo . We recall that visual feedback systems by using apanoramic camera use four coordinate frames and the relative rigid body motion fromΣm to Σo can be represented by gmo = (pmo , eξ̂θmo ) as shown in Fig. 2.
The relative rigid body motion from Σm to Σo can be led by using the composition
rule for rigid body transformations as follows:−1
gmo = gwm
gwo .

(11)

The relative rigid body motion involves the velocity of each rigid body. To this aid,
let us consider the velocity of a rigid body as described in [16]. We deﬁne the bodybTT= [vwmωwm]T , where
velocity of the mirror relative to the world frame Σw as Vwm
vwm and ωwm represent the velocity of the origin and the angular velocity from Σw
to Σm , respectively.

7
b
Vwo
b
Vwc

Ad(gcm1 )b+ Vmo

Ad(gmo1 )bg_ mo =gmoV^mogmo

Relative Rigid Body Motion
for Panoramic Camera (RRBM-PC)

Panoramic Camera
ProjectionfPanoramic Camera

Fig. 6 Block diagram of panoramic camera which consists of relative rigid body motion andpanoramic camera projection.

Diﬀerentiating Eq. (11) with respect to time, the body velocity of the relative rigid
body motion gmo can be written as follows (See [6]):bbb= −Ad(g−1 ) Vwm
+ Vwo
Vmo
mo

(12)

b
where Vwois the body velocity of the target object relative to Σw . Because the camerabbis adequate as an input rather than the mirror velocity Vwm
in thisvelocity Vwc
framework, we lead the body velocity of the relative rigid body motion with the camera
velocity.
The body velocity of the mirror frame relative to Σw will be denoted as
b
−1
−1
−1
−1
−1 b
= gwm
ġwm = gwm
ġwc gcm = gwm
gwc gwcġwc gcm = gcm
V̂wc gcm .
V̂wm

(13)

b
can be transformed
From the property concerning the adjoint transformation, Vwm
intobb= Ad(g−1 ) Vwc
.
Vwm
cm

(14)

Thus, Eq. (12) can be transformed intobbb
= −Ad(g−1 ) Ad(g−1 ) Vwc
+ Vwo
.
Vmo
mo

cm

(15)

This is the body velocity of the relative rigid body motion for the panoramic camera.
bband Vwo, are
While gcm is known information from Eq. (3), gmo and gwo , i.e., Vmo
unknown information in the visual feedback system. Fig. 6 shows the block diagramof the panoramic camera which consists of the relative rigid body motion and thepanoramic camera projection. Then, the control objective is described as follows.
Control Objective: The controlled mobile robot follows the target robot, i.e., therelative rigid body motion gco is coincided with the desired one gd .
Because gcm is known a priori, gco can be obtained from gmo by the composition rule gco = gcm gmo . Thus, we consider the estimate of gmo for the above control
objective.

3 Visual Motion Observer
In this section, we propose a visual motion observer by using a single panoramic camera
in order to estimate the relative rigid body motion from the mirror frame to the object
frame.

8

3.1 Image Jacobian for Panoramic Camera
Since the measurable information is only the image feature f from the panoramic
camera, we consider a visual motion observer in order to estimate the relative rigid
body motion gmo from the image feature f . Using the body velocity of the relative rigidˆbbody motion (15), we choose estimates ḡmo = (p̄mo , eξ̄θ̄mo ) and V̄moof the relative
rigid body motion and velocity, respectively asbb= −Ad(ḡ−1 ) Ad(g−1 ) Vwc+ ue .
V̄mo
mo

cm

(16)

T
T T
ωue
] is to be determined in order to drive the estimated
The new input ue = [vuebvalues ḡmo and V̄mo to their actual ones.
Similarly to Eqs. (8) and (9), the estimated image feature f¯i (i = 1, · · · , n) is
deﬁned asp̄mi = ḡmo poiλα(p̄mi )f¯i =2r + α(p̄mi )z̄mi

»

x̄mi
ȳmi

–

(17)
(18)

where p̄mi := [x̄mi ȳmi z̄mi ]T . f¯(ḡmo ) := [f¯1T · · · f¯nT ]T ∈ R2n means the n image
points case.
In order to establish the estimation error system, we deﬁne the estimation error
between the estimated value ḡmo and the actual relative rigid body motion gmo as−1
gee := ḡmo
gmo
ˆ

(19)
ˆ

in other words, pee = e−ξ̄θ̄mo (pmo − p̄mo ) and eξ̂θee = e−ξ̄θ̄mo eξ̂θmo . Note that pmo =ˆp̄mo and eξ̂θmo = eξ̄θ̄mo if and only if gee = I4 , i.e., pee = 0 and eξ̂θee = I3 . We next
deﬁne the error vector of the rotation matrix eξ̂θab aseR (eξ̂θab ) := sk(eξ̂θab )∨(20)

where sk(eξ̂θab ) denotes 12 (eξ̂θab − e−ξ̂θab ). Using this notation, the vector of the estimation error is given byiThT ξ̂θee.
ee := pT)ee eR (e(21)

From the above, we derive a relation between the actual and the estimated image
feature. Suppose the attitude estimation error θee is small enough that we can leteξ̂θee  I + sk(eξ̂θee ). Then we have the following relation between the actual featurepoint pmi and the estimated one p̄mi
#
"
˜
pee
ξ̄ˆθ̄mo ˆ
pmi − p̄mi = e.
(22)
I −p̂oieR (eξ̂θee )Using a ﬁrst-order Taylor expansion approximation, the relation between the actual
image feature and the estimated one can be expressed as–»∂fi ˛˛∂fi ˛˛∂fi ˛˛(pmi − p̄mi )(23)
fi − f¯i =∂xmi pmi =p̄mi ∂ymi pmi =p̄mi ∂zmi pmi =p̄mi

9where the partial diﬀerentiations ∂fi /∂xmi , ∂fi /∂ymi and ∂fi /∂zmi are represented
as follows.
»–» –2rλαx (pmi )∂fiλα(pmi )
xmi
1
=
+
(24)
∂xmi
(2r + α(pmi )zmi )2 ymi
2r + α(pmi )zmi 0»–» –λα(pmi )2rλαy (pmi )
∂fi
xmi
0
+
(25)
=
∂ymi
(2r + α(pmi )zmi )2 ymi
2r + α(pmi )zmi 1
»
–
»
–
∂fi
2rλαz (pmi )λα2 (pmi )
xmi
xmi
=
−
(26)
∂zmi
(2r + α(pmi )zmi )2 ymi(2r + α(pmi )zmi )2 ymi
where
αx (pmi ) =2−2a2 b2 rxmi zmi pmi  − ab2 xmi (r2 zmi+ a2 pmi 2 )∂α(pmi )
=
`
´
2
∂xmi
pmi 
a2 x2 + a2 y2 − b2 z 2
mi

mi

mi

2
−2a2 b2 rymi zmi pmi  − ab2 ymi (r2 zmi+ a2 pmi 2 )∂α(pmi )αy (pmi ) ==`´
2 − b2 z 2 2 p∂ymi
a2 x2mi + a2 ymi
mi 
mi
`
´
2
2 2
2
2
2
ab zmi r (xmi + ymi ) + b pmi ∂α(pmi )αz (pmi ) =
= `
´2
∂zmi
pmi 
a2 x2 + a2 y2 − b2 z 2
mi

mi

mi

2

2
2
b r(b2 zmi+ a2 x2mi + a2 ymi)pmi 
+ `
.
´2
2
2
2
a2 xmi + a2 ymi − b2 zmi pmi Let us deﬁne the image feature error as fe := f (gmo ) − f¯(ḡmo ). Hence, the relation
between the actual image feature and the estimated one can be given byfe = J (ḡmo )ee ,(27)

where J (ḡmo ) : SE(3) → R2n×6 is deﬁned as
J (ḡmo ) := [J1T (ḡmo ) J2T (ḡmo ) · · · JnT (ḡmo )]T»∂fi ˛˛∂fi ˛˛Ji (ḡmo ) :=∂xmi pmi =p̄mi ∂ymi pmi =p̄mi–ˆ˜ˆ
∂fi ˛˛× eξ̄θ̄mo I −p̂oi .
∂zmi pmi =p̄mi(i = 1, · · · , n)

(28)

(29)

We assume that the matrix J (ḡmo ) is full column rank for all ḡmo ∈ SE(3). Then,the relative rigid body motion can be uniquely deﬁned by the image feature vector.
Because this may not hold in some cases when n = 3, it is known that n ≥ 4 is desirable
for the full column rank of the image Jacobian [17].
The above discussion shows that we can derive the vector of the estimation error
ee from image feature f and the estimated value of the relative rigid body motion ḡmo ,ee = J † (ḡmo )fe(30)

where † denotes the pseudo-inverse. Fig. 7 shows the relation between the image featureerror fe and the estimation error ee . Therefore the estimation error ee can be exploitedin the visual feedback control law using image feature f obtained from the panoramic
camera.

10
b
Vwo
b
Vwc

RRBM-PC

gmo

Panoramic Camera
Panoramic Camera f
Projection
ee

ue

Image
Jacobianfe +Panoramic Camera Model
RRBM-PC gmo Panoramic Camera f
Model
Projection Model

Fig. 7 Block diagram of estimation error vector obtained from panoramic camera andpanoramic camera model.

Remark 1 If we select one and the imaginary unit as a and b numerically, i.e., a = 1,b = i, then the image Jacobian for the panoramic camera (29) equals to the pinhole’s
∂fi
∂fi
∂fi
λ
λ
one [6] as follows: ∂x= zmi[1 0]T , ∂y= zmi
[0 1]T , ∂z
= − zλ[xmi ymi ]T .
2
mi
mi
mi
mi

Thus our previous work [6] can be regarded as a special case of this study, although
the pinhole camera has diﬀerent applications from the panoramic one in the practicalpoint of view.

3.2 Passivity of Estimation Error System
In the same way as Eq. (12), the estimation error system can be represented bybb= −Ad(g−1 ) ue + Vwo
.
Vee
ee

(31)

Then, we have the following lemma relating the input ue to the vector form of theestimation error ee .
bLemma 1 If Vwo= 0, then the following inequality holds for the estimation error
system (31).
Z T
uT
(32)
e (−ee )dt ≥ −βe0where βe is a positive scalar.
Proof Consider the positive deﬁnite function
Ve =1pee 2 + φ(eξ̂θee )2(33)

where φ(eξ̂θab ) := 12 tr(I − eξ̂θab ) is the error function of the rotation matrix [18].
Evaluating the time derivative of Ve along the trajectories of Eq. (31) gives usξ̂θee −ξ̂θeeξ̂θee ξ̂θeeV̇e = pTeṗee + eT
)e
ωee
ee e
R (e
b
T
T
= eT
e Ad(eξ̂θee ) Vee = −ee ue − pee ω̂ue pee= uT
e (−ee ).

(34)

11

Integrating Eq. (34) from 0 to T yields
Z T
Z T
uT
V̇ dτ ≥ −V (0) ≥ −βee (−ee )dτ =
0

0

(35)

where βe is the positive scalar which only depends on the initial state of gee .
Remark 2 Let us consider the vector form of the estimation error −ee as its output.
Then, Lemma 1 says that the estimation error system (31) is passive from the input
ue to the output −ee . In fact, the body velocity of the relative rigid body motion (15)has passivity, the estimation error system preserves its passivity.

3.3 Stability Analysis of Visual Motion Observer
Based on the above passivity property of the estimation error system, we consider thefollowing control law.
ue = Ke ee(36)where Ke := diag{ke1 , · · · , ke6 } is the positive gain matrix of x, y and z axes of thetranslation and the rotation for the estimation error.
b= 0, then the equilibrium point ee = 0 for the closed-loop system
Theorem 1 If Vwo(31) and (36) is asymptotic stable.

Proof In the proof of Lemma 1, we have already shown that the time derivative of Ve
along the trajectory of the system (31) is formulated as Eq. (34). Using the control
input (36), Eq. (34) can be transformed intoV̇e = −eTe K e ee .

(37)This completes the proof.
It should be noted that if the vector of the estimation error is equal to zero, thenthe estimated relative rigid body motion ḡmo equals the actual one gmo . Fig. 8 shows a
block diagram of the visual motion observer with a panoramic camera. By the proposed
visual motion observer, the unmeasurable motion gco will be exploited as the part of thecontrol law. The relative pose between the robot and the target object can be obtainedby using a single panoramic camera in the around direction. This is the advantage ofour proposed method. Our proposed visual motion observer is composed just as the
Luenberger observer for linear systems.
Remark 3 The estimation error vector is conﬁgured by available information (i.e.,themeasurement and the estimate) though it is deﬁned by unavailable one. This is one of
the main contributions of this paper.

4 Visual Motion Observer-based Pose Control
Let us consider the dual of the estimation error system, which we call the pose control
error system, in order to achieve the control objective in this section.

12
b
Vwo
b
Vwc

RRBM-PC

gmo

Panoramic Camera
Panoramic Camera f
Projection

Ke
ue

ee

Image
Jacobianfe +Panoramic Camera Model
RRBM-PC gmo Panoramic Camera f
Model
Projection Model

Visual Motion Observer for Panoramic Camera
Fig. 8 Block diagram of visual motion observer.

4.1 Pose Control Error System
In previous work [6], we deﬁned the pose control error as gec = gd−1 ḡco , which represents
the error between the estimated value ḡco and the reference of the relative rigid body
motion gd . However, the estimation input ue has aﬀected directly the camera control
error system, because the pose control error was deﬁned using the estimated value ḡco .
This has deteriorated the performance of the estimation in the visual feedback control.
In this paper, we deﬁne the pose control error as follows:
gec = gd−1 gco ,(38)

which represents the error between the relative rigid body motion gco and the reference
one gd . By using the above pose control error (38), we can construct the pose control
error system in order to overcome the deterioration of the estimation.
Next, we have to derive the pose control error gec without non-measurable value
gco in order to use it in a control law. Using the composition rule, the pose control
error gec can be transformed as−1
gmo = gd−1 gcm ḡmo gee .
gec = gd−1 gco = gd−1 gcm gmo = gd−1 gcm ḡmo ḡmo

(39)

In Eq. (39), gd , gcm and ḡmo are available information. While the estimation error
vector ee can be obtained as Eq. (30), the estimation error matrix gee , which is deﬁned
using non-measurable value gmo as Eq. (19), cannot be utilized directly.
T ξ̂θee T)] ,Focusing on the deﬁnition of the estimation error vector ee , i.e., ee = [pTee eR (ethe position estimation error pee can be given directly from ee . As for to the rotation
estimation error eξ̂θee , if we assume that the region of the attitude estimation error isrestricted to − π2 ≤ θee ≤ π2 , then ξθee can be gained byξθee =

sin−1 eR (eξ̂θee )
eR (eξ̂θee )

eR (eξ̂θee ).

(40)

Thus it is possible to obtain the pose control error gec from the available information
by Eq. (39), since gee can be derived from ee through ξθee Here, it should be noted thatthe assumption − π2 ≤ θee ≤ π2 will not be a new constraint, since we have already set

13

the assumption that the attitude estimation error θee is small enough in Section 3.1.
Using the notation eR (eξ̂θ ), the vector of the pose control error is deﬁned ashiTT ξ̂θec.
ec := pT)ec eR (e(41)

Note that ec = 0 iﬀ pec = 0 and eξ̂θec = I3 .
The reference of the relative rigid body motion gd is assumed to be constant in this
b
b
= Vco
. Thus, the pose control error system can bepaper, i.e., ġd = 0 and hence Vec
represented as
“
”
b
b
b
+ Vwo
Vec
= −Ad(g−1 ) Ad(g−1 ) Vwc
.
(42)
ec

d

This is dual to the estimation error system (31). Similar to the estimation error system,
the pose control error system also preserves the passivity property.

4.2 Passivity of Visual Motion Error System
Combining the estimation error system (31) and the pose control one (42), we construct
the visual motion observer-based pose control error system (we call the visual motion
error system) as follows:
# "
#
"
» –
b
0
−Ad(g−1 )
I
Vec
b
ec
(43)
=
u+
Vwo
b
0
−Ad(g−1 )
I
Vee
ee

ˆ

˜
b T T T
where u := (Ad(g−1 ) Vwc) ue . Let us deﬁne the error vector of the visual motionderror system ashiTTx := eT(44)c eewhich consists of the pose control error vector ec and the estimation error vector ee . Itshould be noted that if the vectors of the pose control error and the estimation one are
equal to zero, then the actual relative rigid body motion gco tends to the reference one
gd when x → 0. It is noted that the visual motion error system (43) has block diagonal
matrix with respect the input u, while the pose control error system was aﬀected by
the estimation input ue in our previous work [6]. Next, we show an important relation
between the input and the output of the visual motion error system.
b
Lemma 2 If Vwo= 0, then the visual motion error system (43) satisfies

Z T
0uT (−x)dt ≥ −β, ∀T > 0(45)where β is a positive scalar.
Proof Consider the following positive deﬁnite function
V =11pec 2 + φ(eξ̂θec ) + pee 2 + φ(eξ̂θee ).
2
2

(46)

The positive deﬁniteness of the function V results from the property of the error
function φ. Diﬀerentiating Eq. (46) with respect to time and using the skew-symmetry

14

gmo
ee

e to ggd
gcm
gee++ gmo+ +gco + gec

Pose
Computation

g to eec

Kc

Ad( gd )b
Vwc

Pose Controller for Panoramic Camera

Fig. 9 Block diagram of pose controller for panoramic camera.

f

b
Vwc

gmo
Visual Motionee
Observer

b
Vwo

gd
Pose
Controllerb
Vwc

Panoramic
Cameraf

Visual Motion Observer based Pose Controller

Fig. 10 Block diagram of closed-loop system.

TTof the matrices p̂ec and p̂ee , i.e., pT
ec p̂ec ωwc = −pec ω̂wc pec = 0 and pee p̂ee ωue =T−pee ω̂ue pee = 0 yields

V̇ = xT»–−Ad (−pec )0
u = −xT u = uT (−x).
0−Ad(−pee )Integrating Eq. (47) from 0 to T , we obtain
Z T
uT (−x)dt = V (T ) − V (0) ≥ −V (0) := −β
0

(47)

(48)

where β is the positive scalar which only depends on the initial states of gec =(pec , eξ̂θec ) and gee = (pee , eξ̂θee ).
Remark 4 Let us take u as the input and −x as its output. Thus, Lemma 2 implies
that the visual motion error system (43) is passive from the input u to the output −x.

4.3 Stability Analysis of Visual Motion Observer-based Pose Control
Based on the above passivity property of the visual motion error system, we consider
the following control law.
»–
Kc 0
u = Kx, K :=
(49)
0 Ke
where Kc := diag{kc1 , · · · , kc6 } is the positive gain matrix of x, y and z axes of thetranslation and the rotation for the pose control error.
b= 0, then the equilibrium point x = 0 for the closed-loop system
Theorem 2 If Vwo(43) and (49) is asymptotic stable.

Proof In the proof of Lemma 2, we have already shown that the time derivative of V
along the trajectory of the system (43) is formulated as Eq. (47). Using the control
input (49), Eq. (47) can be transformed intoV̇ = −xT Kx.
This completes the proof.

(50)

15

Theorem 2 shows Lyapunov stability for the closed-loop system. If the camera
velocity is decided directly, the control objective is achieved by using the proposed
control law (49). Because a panoramic camera gives us a wide ﬁeld of view, it is adequate
to mobile robots. As for the visual motion error system with a panoramic camera, wehave proposed the stabilizing control law based on the passivity. This is one of the main
contributions of this research. Moreover, both the estimation and the control gain canbe designed independently in the same framework. Although the dead angle problem[11] is not considered explicitly in this framework, our proposed method will overcome
this problem by dealing with a collision avoidance as in [14]. Fig. 9 shows the posecontroller for the panoramic camera. It should be noted that the desired image is notneeded in the proposed controller which only entails the given desired relative rigid
body motion gd as shown in Fig. 9. Fig. 10 shows the block diagram of the closed-loopsystem which consists of the panoramic camera and the visual motion observer-basedpose controller.

4.4 L2 -Gain Performance Analysis
In this subsection, we utilize L2 -gain performance analysis to evaluate the tracking
performance of the control scheme in the presence of a moving target robot. The
motion of the target robot is regarded as an external disturbance.
In order to derive a simple and practical gain condition, we redeﬁne Ke = ke I
where ke is a positive scalar.
Theorem 3 Given a positive scalar γ, assume
kc,min >ke >γ 2 (2ke − 1) + 2(ke − 1)
2 {γ 2 (2ke − 1) − 1}

γ2 + 1
2γ 2

(51)
(52)

where kc,min means the minimum value in Kc . Then the closed-loop system (43) and(49) has L2 -gain ≤ γ.
Proof Diﬀerentiating the positive deﬁnite function V deﬁned in Eq. (33) along the
trajectory of the closed-loop system yields
V̇ =

1
γ2
1
γ2
b 2
b 2
 − x2 − + x2
Vwo
Vwo
2
2
2
2
ˆ
˜T b
−xT u + xT AdT
AdT
Vwo(eξ̂θec )(eξ̂θee )(53)

By completing the squares, we have1γ2b 2V̇ + x2 −

Vwo
2
2
‚
i ‚
‚2
γ2 ‚
1h T
b
T
‚Vwo
‚
Ad
=−
−
Ad
ξ̂θec )
ξ̂θee ) x‚
‚
2
(e
(e
2
γ
‚
i ‚2
1 ‚h
1
‚
T
x‚ − xT u + x2
+ 2 ‚ AdT
Ad
ξ̂θ
ξ̂θ
ec
ee
(e
(e
)
)
2γ
2
1
1
2
T
2
≤
W x − x u + x
2
2γ 2

(54)

16

where
"
W :=

I
AdT ξ̂θec −ξ̂θee(eAd(eξ̂θec −ξ̂θee )

#

I

)

.

Substituting the control input (49) into Eq. (54), we obtain
γ2
1
b 2
Vwo
 ≤ −xT P x ≤ 0V̇ + x2 −
2
2

(55)

holds if P := K − 2γ1 2 W − 12 I is positive semi-deﬁnite. Integrating Eq. (55) from 0 to
T and noticing V (T ) ≥ 0, we have
Z T0x2 dt ≤ γ 2

Z T
0

b 2
Vwo
 dt + 2V (0), ∀T > 0.

(56)

From the Schur complement,
2
P =4

Kc − 2γ1 2 I − 12 I

− 2γ1 2 AdT ξ̂θec −ξ̂θee(e)

3− 2γ1 2 Ad(eξ̂θec −ξ̂θee )“” 5ke − 2γ1 2 − 12 I

(57)

can be modiﬁed as the conditions (51)(52) in Theorem 3 by using AdT(eξ̂θec −ξ̂θee )Ad(eξ̂θec −ξ̂θee ) = I.
The estimation gain does not have to be restricted to a scalar, although it causesslightly complicated gain conditions compared with Eqs. (51) and (52) for the reason
that Schur complement can not be used. In this framework, γ can be considered as anindicator of the tracking performance.
Remark 5 Our previous work has extended the eye-in-hand system with a pinhole
camera to an eye-to-hand one which estimates both a robot hand and a target object[5]. In a similar way, this approach can be extended to estimate and control the poses ofmulti-robots by considering the ﬁxed panoramic camera as surveillance systems which
has it on the ceiling.
Remark 6 For the L2 -gain performance analysis of the visual motion observer without
the pose control, the following condition is only needed.
ke,min >γ2 + 12γ 2(58)

where ke,min means the minimum value in Ke .

5 Simulation and Experimental Results
Both the proposed estimation and control methods are conﬁrmed by the simulation,
while the experiment is carried out by using the proposed visual motion observer only.

17
x 10-3

2

4

6

0

-5
0
0.02

2

4

6

0

-0.02
0

2

time [s]4Fig. 11 Estimation error ee
Dashed, Gain B: Solid).

6(Gain A:x 10-3

0

-5
0
0.01
pecy [m]peey [m]pecx [m]0-40x 10-35

NGeez [rad]

5

2

4

6

2

4

6

4

6

0

-0.01
0
0.04

NGecz [rad]peex [m]

4

0

-0.04
0

2

time [s]Fig. 12 Pose control error ec (Gain A:
Dashed, Gain B: Solid).

5.1 Simulation Results
In this subsection, we present simulation results for stability and L2 -gain performance
analysis in the case of a moving target robot. The target robot has four feature points
and moves by t = 4.8 [s]. The gains for the control law u (49) were empirically selected
as follows:
Gain A)
Gain B)γ = 0.123, Kc = 50I, ke = 100
γ = 0.082, Kc = 100I, ke = 300.

The simulation results are presented in Figs. 11 and 12 which illustrate the estimation error ee and the pose control one ec , respectively. In these ﬁgures, we focus on
the errors of the translations of x and y and the rotation of z. In Figs. 11 and 12, thedashed line and the solid line are the errors in the case of γ = 0.123 and γ = 0.082,
respectively.
In the case of the static target robot, i.e., after t = 4.8 [s], all errors in Figs. 11
and 12 tend to zero. Therefore, asymptotic stability can be conﬁrmed through thesimulation. In the presence of the moving target robot as disturbances by t = 4.8 [s],the tracking performance is improved for the smaller values of γ from Figs. 11 and 12.
Thus the simulation results show that L2 -gain is adequate for the performance measureof the visual motion observer-based pose control.

5.2 Experimental Results
In this subsection, we describe experimental results with respect to the proposed visualmotion observer with a panoramic camera as shown in Fig. 13. A panoramic camera
consists of a MTV-7310 camera and a hyperbolic mirror. In order to evaluate theestimation error exactly, the target object moves on the rail whose radius is 0.27 [m]as Fig. 13 with about 8.0 [s] per round instead of a target mobile robot. Fig. 14 is acaptured image from the panoramic camera. The image feature points are acquired by
OpenCV which is a library of programming functions for real time computer vision.
The experiment with respect to the stability analysis is carried out with an appropriate initial estimation error and the static target object. The experimental results
are presented in Figs. 15 and 16. In Fig. 15, the dashed lines and the solid ones mean

18

TargetHyperbolic
MirrorCamera0.27 [m]Rail

Fig. 13 Panoramic camera and a target ob- Fig. 14 Captured image from the panoramic
ject.
camera.

50
0
0

0.25
time [s]0.5fy4, fy4 [pixel]

fy3, fy3 [pixel]

0.25
time [s]

0.5

150

150
100
50
0

0
0

0

0.25
time [s]

0.5

50
0
0.25
time [s]0.5

Fig. 15 Image features and estimated ones
w.r.t. the y-axis.

0.25

0.5

0.25

0.5

0.25
time [s]

0.5

0

-0.1
0

100

-50
0

0

-0.8
0

50

NGeez [rad]

-50

100

peey [m]100peex [m]150
fy2, fy2 [pixel]

fy1, fy1 [pixel]

150

2
0
0

Fig. 16 Estimation error ee for the static
target object with ke = 10.

the image features and the estimated ones with respect to the y-axis, i.e., fyi andf¯yi (i = 1, · · · , 4), respectively. The estimated image features coincide with the actual
ones. From Fig. 16 which illustrates the estimation error of the translations of x andy and the rotation of z, we can conﬁrm that the estimation error ee tends to zero byusing the proposed visual motion observer.
The experiments with respect to the L2 -gain performance analysis are carried out
with the moving target object. We selected ke = 1 and ke = 10 as the estimation
gain for γ = 1.001 and γ = 0.230, respectively. Fig. 17 shows the estimation error
ee . The error in the case of ke = 1 and ke = 10 are shown in the left side and theright one of this ﬁgure. The estimation error is reduced for the smaller values of γ in
Fig. 17. Fig. 18 depicts the estimated trajectory with ke = 10 and the course of themoving target object. The cross sign means the estimated value at both t = 0 [s] andt = 8 [s]. Because the average and q
the sample standard deviation of the norm of theestimated position from the origin p̄2mox + p̄2moy are 0.269 [m] and 2.87 × 10−3 [m],we can conﬁrm that the estimated trajectory coincides with the ideal trajectory of themoving target object perfectly. It is the advantage of this research that the relativepose can be obtained by using only single panoramic camera in the around direction.
The demonstration movie of the proposed visual motion observer can be seen on the
Web site [19].

19ke = 10

-0.3
0
0.05

8

0

1.5

0

4

8

0
0

4
time [s]

8

4

8

0

-0.05
0
1.5
NGeez [rad]-0.05
NGeez [rad]40.1
y-axis [m]peex [m]

0.05

0

t=0
and
t=8

0.2

peey [m]

-0.3

0.3

0

peey [m]peex [m]ke = 1
0

4

-0.1

8

-0.2

0
0

0.27 [m]0

4time [s]

-0.3
-0.4

8

Fig. 17 Estimation error ee for the moving
target object (left side: with ke = 1; rightside: with ke = 10)
.

-0.3

-0.2

-0.1

0
0.1
x-axis [m]

0.2

0.3

0.4

Fig. 18 Estimated trajectory of the moving
target object with ke = 10 (Ideal trajectory of
the moving target object: Dashed, Estimatedtrajectory: Solid).

6 Conclusions
This paper proposes the visual motion observer and pose controller with a panoramic
camera based on the passivity. The advantage of this research is that the relativepose between the leader and the follower robot can be obtained by using a single
panoramic camera in our proposed method in the around direction. Especially, ourproposed method does not need a desired image a priori. The estimation error systemwith a panoramic camera has the passivity which allows us to prove stability in the senseof Lyapunov. The visual motion error system which consists of the estimation error
system and the pose control one preserves the passivity. As for the visual motion error
system with a panoramic camera, we have proposed the stabilizing control law basedon the passivity. This is one of the main contributions of this research. In the stabilizing
control law, both the estimation and the control gain can be designed independently
in the same framework. Our previous work [6] can be regarded as a special case of thisstudy. Simulation results are presented to verify stability and L2 -gain performance inthe visual motion observer-based pose control. Because the experiment is carried out
by using the proposed visual motion observer only, we will realize pose control by usinga mobile robot with the omnidirectional camera in the future work.
Acknowledgements We would like to thank Mr. Masahiro Matsuzawa in Kanazawa Instituteof Technology for his eﬀort to build experimental systems.

A Notation of Homogeneous TransformIn this paper, pab = [xab yab zab ]T ∈ R3 represents the position vector of origin of frame Σbfrom the origin of frame Σa . We use the notation eξ̂θab ∈ R3×3 to represent the change ofthe principle axes of a frame Σb relative to a frame Σa . ξab ∈ R3 speciﬁes the direction ofˆ ab to denote ξˆab θab . The
rotation and θab ∈ R is the angle of rotation. For simplicity we use ξθˆ = ξ × θ13 for the vector
notation ‘∧’ (wedge) is the skew-symmetric operator such that ξθcross-product × and any vector θ ∈ R3 . The notation ‘∨’ (vee) denotes the inverse operator to‘∧’, i.e., so(3) → R3 . Recall that a skew-symmetric matrix corresponds to an axis of rotation

20(via the mapping a → â). We use the 4 × 4 matrix–»ξ̂θab pabgab = e01(59)

as the homogeneous representation of gab = (pab , eξ̂θab ) ∈ SE(3) describing the conﬁgurationof a frame Σb relative to a frame Σa . The adjoint transformation associated with gab is denotedby#"eξ̂θab p̂ab eξ̂θab.
(60)
Ad(gab ) =0eξ̂θab

References
1. F. Chaumette and S. A. Hutchinson, “Visual Servoing and Visual Tracking,” In:B. Siciliano
and O. Khatib (Eds), Springer Handbook of Robotics, Springer–Verlag, pp. 563–583 (2008)
2. V. Lippiello, B. Siciliano and L. Villani, “Position-Based Visual Servoing in Industrial
Multirobot Cells Using a Hybrid Camera Conﬁguration,” IEEE Trans. on Robotics, Vol. 23,
No. 1, pp. 73–86 (2007)
3. N. R. Gans and S. A. Hutchinson, “Stable Visual Servoing Through Hybrid SwitchedSystem Control,” IEEE Trans. on Robotics, Vol. 23, No. 3, pp. 530–540 (2007)
4. G. Hu, W. MacKunis, N. Gans, W. E. Dixon, J. Chen, A. Behal, and D. Dawson,“Homography-Based Visual Servo Control With Imperfect Camera Calibration,” IEEE
Trans. on Automatic Control, Vol. 54, No. 6, pp. 1318–1324 (2009)
5. H. Kawai, T. Murao and M. Fujita, “Passivity-based Dynamic Visual Feedback Control with
Uncertainty of Camera Coordinate Frame,” Proc. of the 2005 American Control Conference
pp. 3701–3706 (2005)
6. M. Fujita, H. Kawai and M. W. Spong, “Passivity-based Dynamic Visual Feedback Control
for Three Dimensional Target Tracking: Stability and L2 -gain Performance Analysis,” IEEE
Trans. on Control Systems Technology, Vol. 15, No. 1, pp. 40–52 (2007)
7. T. Murao, H. Kawai and M. Fujita, “Predictive Visual Feedback Control with Eye-in/toHand Conﬁguration via Stabilizing Receding Horizon Approach,” Proc. of the 17th IFAC
World Congress on Automatic Control, pp. 5341–5346 (2008)
8. C. Geyer and K. Daniilidis, “A Unifying Theory for Central Panoramic Systems and Practical Implications,” In:D. Vernon (Ed), Computer Vision - ECCV 2000, Springer–Verlag,
pp. 445–461 (2000)
9. H. Hadj-Abdelkader, Y. Mezouar, N. Andreﬀ and P. Martinet, “2 1/2D Visual Servoing with
Central Catadioptric Cameras,” Proc. of the 2005 IEEE/RSJ International Conference onIntelligent Robots and Systems, 2342–2347 (2005)
10. G. L. Mariottini and D. Prattichizzo, “EGT for Multiple View Geometry and Visual
Servoing,” IEEE Robotics & Automation Magazine, Vol. 12, No. 4, pp. 26–39 (2005)
11. R. T. Fomena and F. Chaumette, “Improvements on Visual Servoing From Spherical
Targets Using a Spherical Projection Model,” IEEE Trans. on Robotics, Vol. 25, No. 4,
pp. 874–886 (2009)
12. A. K. Das, R. Fierro, V. Kumar, J. P. Ostrowski, J. Spletzer and C. J. Taylor, “A VisionBased Formation Control Framework,” IEEE Trans. on Robotics and Automation, Vol. 18,
No. 5, pp. 813–825 (2002)
13. R. Vidal, O. Shakernia and S.Sastry, “Following the ﬂock,” IEEE Robotics & Automation
Magazine, Vol. 11, No. 4, pp. 14–20 (2004)
14. M. Fujita, T. Hatanaka, N. Kobayashi, T. Ibuki and M. W. Spong, “Visual Motion
Observer-based Pose Synchronization: A Passivity Approach,” Proc. of the 48th IEEE Conf.
on Decision and Control and 28th Chinese Control Conf., pp. 2402–2407 (2009)
15. F. Chaumette, “Potential Problems of Stability and Convergence in Image-Based andPosition-Based Visual Servoing,” In:D. J. Kriegman, G. D. Hager, and A. S. Morse (Eds),
The Confluence of Vision and Control, Springer-Verlag, pp. 66–78 (1998)
16. R. Murray, Z. Li and S. S. Sastry, A Mathematical Introduction to Robotic Manipulation,
CRC Press (1994)
17. H. Michel and P. Rives, “Singularities in the Determination of the Situation of a Robot
Eﬀector from the Perspective View of 3 Points,” Technical Report, INRIA (1993)

21
18. F. Bullo and A.D. Lewis, Geometric Control of Mechanical Systems, Springer–Verlag(2004)
19. http://wwwr.kanazawa-it.ac.jp/kawai/research/VMO/movies.html