Passivity-based Visual Force Feedback Control for Planar Manipulatorswith Eye-in-Hand Configuration
Hiroyuki Kawai, Toshiyuki Murao and Masayuki Fujita
Abstract— This paper investigates visual force feedback control for planar manipulators with the eye-in-hand configuration
based on passivity. The vision/force control is applied to horizontal/vertical direction for the environment which is thought
as a frictionless, elastically compliant plane. We show passivityof the visual force feedback system which allows us to provestability in the sense of Lyapunov. The L2 -gain performance
analysis for the disturbance attenuation problem is considered
via the dissipative systems theory. Finally simulation results areshown to verify the stability and L2 -gain performance of thevisual force feedback system.

Environment

Force
Sensor2DOF Robot
Manipulator

CameraTarget Object
Fig. 1.

Visual force feedback system with the eye-in-hand configuration.

I. I NTRODUCTION
Robotics and intelligent machines need sensory information to behave autonomously in dynamical environments.
Visual information is particularly suited to recognize unknown surroundings. Vision based control of robotic systems
involves the fusion of robot kinematics, dynamics, andcomputer vision to control the motion of the robot in an
efficient manner. The combination of mechanical control
with visual information, so-called visual feedback control orvisual servoing, is important when we consider a mechanicalsystem working under dynamical environments [1][2].
In the visual feedback control, many practical methods arereported. In [3], the 2 1/2-D visual servoing which incorporates the advantages of both position-based and image-based
visual servoing is proposed in order to guarantee robustness
with respect to calibration errors. Partitioned visual servoing
is considered in [4] in order to guarantee that all features
remain in the image. Cowan et al. [5] addressed the fieldof-view problem for 3D dynamic visual feedback systemusing navigation functions. More recently, an approach basedon switching between position-based visual servoing andbackward motion is investigated for dealing with the field ofview problem [6]. In our previous works, we discussed the
dynamic visual feedback control for three dimensional target
tracking based on passivity [7][8][9][10]. Recent applications
H. Kawai is with Department of Robotics, Kanazawa Institute of Technology, Ishikawa 921-8501, Japan hiroyuki@neptune.kanazawa-it.ac.jp
T. Murao is with Master Program of Information Systems Architecture,
Advanced Institute of Industrial Technology, Tokyo 140-0011, Japan
M. Fujita is with Department of Mechanical and Control Engineering,Tokyo Institute of Technology, Tokyo 152-8550, Japanof visual feedback control include the autonomous injectionof biological cells [11], laparoscopic surgery [12] and others.
Although visual information is necessary in order to recognize environments, visual information is inadequate to complete tasks in these applications. For example, not only visual
information but also force information are needed to inject
DNA to biological cells. Hence, integrating visual feedback
control with force control is important for the modern robot.
Most reported hybrid vision/force control considers the fixedcamera configuration with a calibrated camera [13][14][15].
In contrast to most hybrid vision/force research, Baeten
et al. [16] addressed a hybrid control structure for theeye-in-hand vision and force control. However few rigorous
results have been obtained in terms of the nonlinear control
aspects, although many practical methods are reported withexperimental results. For example, there exist no researches
that explicitly show Lyapunov function for the visual force
feedback system.
This paper deals with visual force feedback control forplanar manipulators with the eye-in-hand configuration as
depicted in Fig. 1. The vision/force control is applied tohorizontal/vertical direction for the environment which isthought as a frictionless, elastically compliant plane. We
show passivity of the visual force feedback system whichallows us to prove stability in the sense of Lyapunov. The
L2 -gain performance analysis for the disturbance attenuation
problem is considered via the dissipative systems theory. Finally simulation results are shown to verify the stability andL2 -gain performance of the visual force feedback system.

Target Object

Xi
Camera Frame
Yi
Xc

pco

fx

f

fy

Yc 5c

pwo

l
Zc

camera with the frame Σc = {Xc Yc Zc } is mounted on the
manipulator end-effector as depicted in Fig. 2. Hence, themanipulator kinematics gives the camera position pwc(q) :=[xwc(q) ywc (q)]T and the orientation θ(q) := q1 + q2 withrespect to Σw . A frame Σi = {Xi Yi } is defined in thecamera image plane and its origin is the intersection of the
optical axis with the image plane. Here it is assumed thatthe axes Xi and Yi parallel the axes Xc and Yc respectively.
Next the object point pwo is located at [xwo ywo zwo ]Twith respect to the frame Σw . We assume that the object
point motion is restricted on the Xw −Yw plane, i.e. żwo = 0.
pio = [xio yio ]T is the image coordinate of pwo through theperspective transformation with the frame Σi .
Taking the perspective transformation as the camera model(shown in Fig. 2) yields

pwc Yw
Image Plane

Manipulator

Fig. 2.

Zw

5w
World Frame

Xw

Pinhole camera model with perspective projection

II. V ISUAL F ORCE F EEDBACK S YSTE M
A. Manipulator Modelf =The dynamics of n-link rigid robot manipulators can be
written as
M (q)q̈ + C(q, q̇)q̇ + g(q) = τ(1)where q, q̇ and q̈ are the joint angle, velocity and acceleration,
respectively, τ is the vector of the input torque, and τd represents a disturbance input. M (q) ∈ Rn×n is the manipulator
inertia matrix, C(q, q̇) ∈ Rn×n is the Coriolis matrix andg(q) ∈ Rn is the gravity vector. The following properties
are well known [17].
Property 1: The inertia matrix M (q) is positive definite.
Property 2: Ṁ (q) − 2C(q, q̇) is skew-symmetric.
Property 2 is concerned with the passivity property. These
properties are important for the Lyapunov/passivity based
control design.
The relation between the vector of generalized forces F
and the vector of joint actuating generalized forces τF canbe expressed as
τF = JpT (q)F(2)

where Jp (q) is the manipulator Jacobian [18]. Then we
consider generalized forces F and the disturbance forces d,the manipulator dynamics (1) can be transformed into
M (q)q̈ + C(q, q̇)q̇ + g(q) = τ + JpT (q)(F + d).

(3)

Throughout this paper, the above equation is used as themodel of the manipulators with n = 2.
B. Camera Model
We consider a planar manipulator with the world frameΣw = {Xw Yw Zw }. It is assumed that the manipulator endeffector evolves in the Xw −Yw plane of Σw . Suppose that a

λ T
R (θ)(pwo − pwc )
zwo

(4)

where f := [fx fy ]T and λ > 0 is a focal length [7].
Then the differential kinematics of the manipulator gives
the relationship between the manipulator joint velocities q̇
and the velocities of the camera mounted on the end-effector.
The relation can be represented using the manipulator Jacobian Jp (q) ∈ R2×2 :ṗwc(q) = Jp (q)q̇.

(5)

The derivation of the equation (4) yields
λ T
λ T
f˙ = −R Jp q̇ − RT Ṙf +R ṗwo
zwo
zwo

(6)

where ṗwo is the unknown motion of the target object.
C. Visual Force Feedback System
Here, we define the coordinate frame ΣF as a force frame
whose direction coincides the world frame Σw as shown in
Fig. 3. Then, the generalized forces F can be separated Fx
normal and Fy tangential to the environment plane, i.e. F =[Fx Fy ]. Moreover, we define the virtual image plane cx −cy
whose origin and direction coincide the image plane Xi − Yiand the force frame ΣF , respectively.
Thus, the virtual image c := [cx cy ]T can be expressedasc = Rfcos(θ)=sin(θ)− sin(θ)
cos(θ)



fx
fy


.

(7)

The environment is thought of as a frictionless, elasticallycompliant plane. Hence the model of the contact force takeson the simple form
F = Ke (pwc − pwc0 )

(8)

Yw
-q2
cy,YF

q1

5F

5w
Fig. 3.

where Sθ and Cθ represent sin(θ) and cos(θ) for short,respectively. Using (4), (6) and (15), we obtain−αSθ −αCθ
θ̇fe
ṡ =
Cθ
−Sθ



kexCθ −kexSθ
Cθ Sθ
Jp q̇
−
λ
λ
−Sθ Cθz Sθ
zwo Cθ
 woαCθ −αSθ
0 −θ̇
−
fe
Sθ
Cθ
θ̇ 0



kexCθ −kexSθ
Cθ Sθ
ṗwo
+
λ
λ
−Sθ Cθz Sθ
zwo Cθ


 wo

0
0
kex
kex
Jp q̇ +
ṗwo . (16)
=−
λ
λ
0
0
zwo
zwo

Environment

Yi
cx,XF
Xi
Xw

Coordinate frames for planar manipulators

where pwc is the position of the contact point, pwc0 is apoint of the environment plane at rest. Ke is the constantsymmetric stiffness matrix, i.e. Ke = diag{kex, key }. It is
assumed that contact with the environment plane is not lost.
Let Fd = [Fdx Fdy ] be a desired force and fd = [0 0]T be
a desired image. By using the relation (7), the desired virtual
image cd := [cdx cdy ]T satisfies cd = [0 0]T . We define theforce error, the image error and the virturl image error as
Fe := F − Fd ,(9)fe := f − fd ,(10)

ce := c − cd ,(11)respectively. By using (4), (7) and (8), the force error Fe canbe transformed into
zwo
Fe =
Ke ce
λ 


zwo kex 0
cex
.
(12)
=
0 key
cey
λ
Our purpose is to design a stabilizing control law τ suchthat the force control and the visual feedback control holdfor the normal direction and the tangential direction to theenvironment plane, respectively.
In order to consider both the force control and the visualfeedback control, we define the visual force error as follows:


Fex
s=
.
(13)
cey
By using (7) and (12), the state s can be transformed into



α 0
cex
s=
0 1
cey


α cos(θ) −α sin(θ)
=
fe
(14)
sin(θ)
cos(θ)
zwo kex. Differentiating (14) with respect towhere α :=λtime, we obtainαCθ −αSθ−αSθ −αCθθ̇fe +f˙e (15)
ṡ =
Cθ
−Sθ
Sθ
Cθ

where we exploit that RT Ṙ is skew-symmetric, i.e.


0 −θ̇
T
R Ṙ =.
θ̇ 0
Combining (3) and (16), we construct the visual force
feedback system as follows:M q̈ + C q̇ + g = τ + JpT (F + d)(17)ṡ = −Kxy Jp q̇ + Kxy ṗwo

(18)

where


Kxy :=

kex
0

0


.

λ
zwo

The following assumption will be made throughout thispaper:
Assumption 1: The manipulator Jacobian Jp is nonsingular.
This assumption is required for technical reasons in thestability analysis. Under this assumption, we formulate themanipulator control problem as follows:
Control problem : For the visual force feedback systemwith the eye-in-hand configuration described by (17) and(18), design a control law τ such that s → 0 and q̇ → 0
as t → ∞.
III. V ISUAL F ORCE F EEDBACK C ONTROL
A. Passivity of Visual Force Feedback System
Now, we propose the control law for the manipulator asτ = uξ + JpT KF c s + M u̇d + Cud + g − JpT F
whereKF c :=

kF
0

0
kc

(19)


.

kF and kc denote the positive scalar gain for force error andimage error, respectively. The new inputs uξ and ud are to be
determined in order to achieve the control objective. Here,we define the error vector with respect to the joint velocityof the manipulator dynamics asξ := q̇ − ud .

uTd ]T . We define the state and the disturT T T

bance of visual force feedback system as x := [ξ s ] andw := [dT ṗTwo ]T , respectively.
Remark 1: In the visual force feedback system, the joint
angle q, velocity q̇, normal generalized force to the environment Fx and visual information f are measurable. Then,the normal visual information to the environment cy will
be obtained by using f and θ = q1 + q2 . Because oneof states s which is constructed by Fex = Fx − Fdx andcey = [cos(θ) sin(θ)]fe is calculated, the state of the visual
force feedback system x can be exploited for the control law.
Before constructing the visual force feedback control law,
we derive an important lemma.
Lemma 1: If w = 0, then the visual force feedback system(21) satisfies
 T
uT νdτ ≥ −β, ∀T > 0
(22)
0

where


ν := N x, N :=I
00
−JPT KF c



V =

1
1 T
−1
s.
ξ M ξ + sT KF c Kxy
2
2

(23)

Differentiating (23) with respect to time yields

B. Visual Force Feedback Control and Stability Analysis
We now propose the following control input for theinterconnected system:



Kξ 0
I
0
u=−
x
0 Kp
0 −JPT KF c
= −KN x
(26)
where


K :=

Kξ
0

0
Kp



and Kξ := diag{kξ1 , kξ2 } and Kp = diag{kp1 , kp2 } denotethe positive gain matrices for each joint axis. Using thecontrol input (26), s = 0 and q̇ = 0 if and only if x = 0.
Then, the stability of the visual feedback system can be
described by the following theorem.
Theorem 1: If w = 0, then the equilibrium point x = 0
for the closed-loop system (21) and (26) is asymptotic stable.
Proof: In the proof of Lemma 1, we have alreadyderived that the time derivative of V along the trajectoryof the system (21) is formulated as (24). Using the control
input (26), (24) can be transformed into
(27)

Because N is a nonsingular matrix and K is a positive
definite matrix, this completes the proof.
Theorem 1 shows the stability via Lyapunov method for thevisual force feedback system. It is interesting to note thatstability analysis is based on the passivity as described in(22).
C. L2 -Gain Performance Analysis

1−1ṡV̇ = ξ T M ξ˙ + ξ T Ṁ ξ + sT KF c Kxy2
 1= ξ T −Cξ + uξ + JpT KF c s + ξ T Ṁξ2−1−1−sT KF c Kxy
Kxy Jp ξ − sT KF c Kxy
Kxy Jp ud1= ξ T uξ + ξ T (Ṁ − 2C)ξ + ξ T JpT kF c s
2−sT KF c Jp ξ − sT KF c Jp ud= uT ν.

where β is a positive scalar that only depends on the initialstates of ξ and s.
Let us take u as the input and ν as its output. Thus,
Lemma 1 implies that the visual force feedback system (21)is passive from the input u to the output ν as in the definitionin [19].

V̇ = uT ν = xT N T u = −xT N T KN x.

and β is a positive scalar.
Proof: Consider the following positive definite function

(25)

0

(20)

Using (17), (18) and (19), the visual force feedback system
can be derived as follows:  −M −1 Cξ + M −1 JPT KF c sξ˙=−Kxy Jp ξ
ṡ


−1
0
M
u
+
0
−Kxy Jp0
M −1 JPT
w
(21)
+
0
Kxy
where u := [uTξ

Integrating (24) from 0 to T , we obtain T
uT νdτ = V (T ) − V (0) ≥ −V (0) = −β(24)In this subsection, we utilize L2 -gain performance analysis
to evaluate the tracking performance of the control scheme
in the presence of a moving target. The motion of the target
object is regarded as an external disturbance. Here, we define
the controlled outputs as


εξ 0
z :=
x
0 εs

T

(29)

By completing the squares, we haveγ21V̇ + z2 − w2
2
2

It can be verified that the inequality
γ2
1
(31)
V̇ + z2 − w2 ≤ 0
2
2holds if P is positive semi-definite. Integrating (31) from 0
to T and noticing V (T ) ≥ 0, we have
 T
 T
2
2
z ≤ γw2 + 2V (x(0))(32)
0This completes the proof.
The L2 -gain performance analysis of the visual force feedback system is discussed via the dissipative systems theory.
IV. S IMULATION
To illustrate the behavior of the visual force feedback
control, we apply the proposed control law to a two-link
planar manipulator. The stiffnesses of the contact plane arekex = 200 [N/m] and key = 0 [N/m]. The control objective
is Fx → Fdx and cy → cdy as t → ∞, i.e. Fy and cx are
not considered in this framework. We give the desired force
and the desired image as Fdx = 18 [N] and cdy = 0 [pixel],
respectively.

1.0
time [s]

1.5

2.0

0.5

1.0
time [s]

1.5

2.0

0
-2

x1 [rad/s]

Fig. 4.

Time response of ξ(t)

10
0
-10
-20

0

0.5

1.0
time [s]

1.5

2.0

0.5

1.0
time [s]

1.5

2.0

0

-200
-400
0

1
≤ −ξ T Kξ ξ − sT KFT c Jp Kp JpT s + (εξ ξ)T εξ ξ
2
1
1 T T
1
T
+ (εs s) εs s + 2 ξ Jp Jp ξ + 2 sT KFT c KF c s
22γ2γ= −xT P x
(30)

0.5

2

KFT c Jp Kp JpT s+ξ T JpT d + xT KF c ṗwo

0

0

-4
0

cey [pixel]V̇ = −ξ Kξ ξ − sT

20

4x2 [rad/s]

where γ ∈ R is positive.
Theorem 2: Given a positive scalar γ, consider the control
input (26) with the weight matrices εξ and εs and the gains
Kξ , Kp and KF c such that the matrix P is positive semidefinite. Then the closed-loop system (21) and (26) has L2 gain ≤ γ.
Proof: Differentiating the positive definite function V
defined in (23) along the trajectory of the closed-loop systemyields400Fex [N]

where εξ = diag{εξ1 , εξ2 } > 0 and εs = diag{εs1 , εs2 } >0. are weight matrices.
Here, let us defineKξ − 12 εTξ εξ − 2γ1 2 JPT JP
P :=
0

0
(28)
KFT c Jp Kp JpT KF c − 12 εTs εs − 2γ1 2 KFT c KF c

Fig. 5.

Time response of s(t)A. Stability Analysis
First, we present results for the stability analysis with astatic target object. The initial conditions of the manipulator
are set as follows: q1 (0) = π/2 [rad], q2 (0) = −π/2 [rad],q̇1 (0) = 0 [rad/s], q̇2 (0) = 0 [rad/s]. In this initial setting,the initial states of the visual force feedback system satisfy
ξ1 = 0 [rad/s], ξ2 = 0 [rad/s], Fex = −18 [N], cey = −310
[pixel], respectively. The controller parameters for Equation (26) were empirically selected as Kξ = diag{0.1, 0.1},
Kp = diag{1, 0.1} and KF c = diag{10, 0.1}. The simulation results are shown in Figs. 4 and 5. Figs. 4 and 5
illustrate the velocity error ξ and the force and image errors, respectively. From these figures, the asymptotic stabilityof the equilibrium point x = 0 can be also confirmed.
B. L2 -gain Performance Analysis
Next, we present simulation results for the L2 -gain performance analysis in the case of a moving target object. In
particular, we consider the disturbance attenuation problem
by regarding the target motion as the disturbance for thevisual force feedback system. The target object moves alonga straight line parallel to Yw -axis. The initial states of the

|| z || (g = 5.54)|| z || (g = 2.92)

0.2
0.1
0

0

2.0

4.0
time [s]

6.0

8.0

2.0

4.0
time [s]

6.0

8.0

0.2
0.1
0

0

Fig. 6.

Euclidean norm of the controlled output zvisual force feedback system is x(0) = 0. The weights forthe states are selected as εξ = diag{2.0, 2.0} and εs =diag{0.25, 0.01}. The following gains were selected in order
to confirm the adequacy of the L2 -gain performance for thevisual force feedback system.
Gain A : γ = 5.54, Kξ = diag{5, 5}, Kp = diag{80, 40},
KF c = diag{0.5, 0.015}
Gain B : γ = 2.92, Kξ = diag{5, 5}, Kp = diag{5, 2.5},
KF c = diag{5, 0.4}
In Fig. 6, the top graph and the bottom one show thenorm of controlled output z in the case of γ = 5.54 andγ = 2.92, respectively. It can be verified that the disturbanceattenuation is improved in the case of the smaller value ofγ from Fig. 6.
V. C ONCLUSIONSThis paper investigates the visual force feedback control
for planar manipulators. The proposed method is regarded
as an extension of the hybrid position/force control to thehybrid vision/force control. The main contribution of thispaper is to show that the visual force feedback system has thepassivity. Stability and L2 -gain performance analysis for thevisual force feedback system are discussed based on passivity
and dissipative systems theory. Finally simulation results arepresented to verify the stability and L2 -gain performanceof the visual force feedback system. In our future work,
we will discuss the visual force feedback control for three
dimensional task space based on passivity.
R EFERENCE S[1] S. Hutchinson, G. D. Hager and P. I. Corke, “A Tutorial on Visual
Servo Control,” IEEE Trans. Robotics and Automation, Vol. 12, No. 5,
pp. 651–670, 1996.

[2] H. I. Christensen and P. Corke, “Editorial: Special Issue on Visual
Servoing,” Int. Journal of Robotics Research, Vol. 22, No. 10–11,
pp. 779–780, 2003.
[3] E. Malis, F. Chaumette and S. Boudet, “2-1/2-D Visual Servoing,”
IEEE Trans. on Robotics and Automation, Vol. 15, No. 2, pp. 238–
250, 1999.
[4] P. I. Corke and S. A. Hutchinson, “A New Partitioned Approach to
Image-Based Visual Servo Control,” IEEE Trans. on Robotics andAutomation, Vol. 17, No. 4, pp. 507–515, 2001.
[5] N. J. Cowan, J. D. Weingarten and D. E. Koditschek, “Visual Servoing
via Navigation Functions,” IEEE Trans. Robotics and Automation,
Vol. 18, No. 4, pp. 521–533, 2002.
[6] G. Chesi, K. Hashimoto, D. Prattichizzo and A. Vicino, “Keeping
Features in the Field of View in Eye-In-Hand Visual Servoing: A
Switching Approach,” IEEE Trans. on Robotics, Vol. 20, No. 5, pp.
908–913, 2004.
[7] A. Maruyama and M. Fujita, “Robust Control for Planar Manipulatorswith Image Feature Parameter Potential,” Advanced Robotics, Vol. 12,
No. 1, pp. 67–80, 1998.
[8] T. Murao, H. Kawai and M. Fujita “Passivity-based Control of VisualFeedback Systems with Dynamic Movable Camera Configuration,”
Proc. of the 44th IEEE Conference on Decision and Control, pp. 5360–
5365, Seville, 2005.
[9] H. Kawai, T. Murao and M. Fujita, “Image-based Dynamic VisualFeedback Control via Passivity Approach,” Proc. of the 2006 IEEE
Conference on Control Applications, pp. 740–745, 2006.
[10] M. Fujita, H. Kawai and M. Spong, “Passivity-based Dynamic VisualFeedback Control for Three Dimensional Target Tracking: Stability
and L2 -gain Performance Analysis,” IEEE Transactions on Control
Systems Technology, Vol. 15, No. 1, pp. 40–52, 2007.
[11] S. Yu and B. J. Nelson, “Autonomous Injection of Biological Cells
Using Visual Servoing,” In:D. Rus and S. Singh (Eds), Experimental
Robotics VII, Springer–Verlag, pp. 169–178, 2001.
[12] K. Omote et al., “Self-Guided Robotic Camera Control for Laparoscopic Surgery Compared with Human Camera Control,” The
American Journal of Surgery, Vol. 177, No. 4, pp. 321–324, 1999.
[13] B. J. Nelson and P. K. Khosla, “Force and Vision Resolvability for
Assimilating Disparate Sensory Feedback,” IEEE Trans. Robotics andAutomation, Vol. 12, No. 5, pp. 714–731, 1996.
[14] K. Hosoda, K. Igarashi and M. Asada, “Adaptive Hybrid Control for
Visual and Force Servoing in an Unknown Environment,” IEEE Trans.
Robotics and Automation Magazine, Vol. 5, No. 4, pp. 39–43, 1998.
[15] D. Xiao, B. K. Ghosh, N. Xi and T. J. Tarn, “Sensor-based Hybrid
Position/Force Control of a Robot Manipulator in an Uncalibrated
Environment,” IEEE Transactions on Control Systems Technology,
Vol. 8, No. 4, pp. 635–645, 2000.
[16] J. Baeten, H. Bruyninckx and J. D. Schutter, “Integrated Vision/Force
Robotic Servoing in the Task Frame Formalism,” International Journalof Robotics Research, Vol. 22, No. 10–11, pp. 941–954, 2003.
[17] M. W. Spong, S. Hutchinson and M. Vidyasagar, Robot Modeling andControl, John Wiley & Sons, 2006.
[18] B. Siciliano and L. Villani, Robot Force Control, Kluwer Academic
Publishers, 1999.
[19] A. van der Schaft, L2 -Gain and Passivity Techniques in Nonlinear
Control (2nd ed.), Springer–Verlag, 2000.