2012 IEEE/RSJ International Conference onIntelligent Robots and Systems
October 7-12, 2012. Vilamoura, Algarve, Portugal

Robust Visual Servoing for Object Manipulation
with Large Time-Delays of Visual Information
Akihiro Kawamura, Kenji Tahara, Ryo Kurazume and Tsutomu Hasegawa
Abstract— This paper proposes a new visual servoing method
for object manipulation robust to considerable time-delays ofvisual information. There still remain several problems in visual
servoing methods although they are quite useful and effective
for dexterous object manipulation. For instance, time-delays
to obtain necessary information for object manipulation fromvisual images induce unstable behavior. The time-delays aremainly caused by low sampling rate of visual sensing system,computational cost for image processing, and latency of datatransmission from visual sensor to processor. The method makes
it possible to avoid such unstable behavior of the systems due toconsiderable time-delays using virtual object frame defined byonly each joint angle. Firstly, a new control scheme for object
manipulation using the virtual object frame is designed. Next,
numerical simulations are conducted to verify the effectivenessof the control scheme. Finally, experimental results are shownto demonstrate the practical usefulness of proposed method.

I. I NTRODUCTION
Visual information is useful for dexterous object manipulation by a sensory feedback controller [1–4]. However ingeneral, time-delays to obtain necessary information fromvisual images should be considered carefully since they
make a visual servoing system unstable. The time-delays
are mainly caused by low sampling rate, computational cost,and data transmission latency. It is known that the sampling
rate of a standard NTSC camera which is often used in
the visual sensing system is fixed at 30 [Hz]. However, it
is not enough to use directly in the visual servoing since
the servo-loop generally requires a few milliseconds orderedsampling rate. In addition, other inevitable time-delays arecaused by a latency of data transmission from a visual sensor
to a processor, and a computational cost to extract necessary
information from images.
To date, several methods have been proposed to improve
the robustness of the visual servoing against these timedelays. They are based on state estimation techniques, suchas Kalman filter [5], AR model [6], nonlinear observer [7]and multi-rate control [8], [9]. The state estimation techniques are based on an assumption that the overall dynamics
can be approximated as a linear system. However, the overall
dynamics of an object manipulation system is too complicated to be approximated as a linear system since manygeometrical and physical constraints have to be taken into
consideration. Also in these methods, reaching or moving
tasks of the end-effector are mainly addressed and the effectof the time-delays for dexterous object manipulation has notbeen discussed.
Recently, we have proposed a stable object grasping andmanipulation method for an arbitrary polyhedral object using
a multi-fingered hand-arm system [10], [11]. The advantageof the method is that only proprioceptive sensory information, such as joint angles or angular velocities, is required forstable object grasping and no other information such as theobject shape and grasping points is necessary. After that,it has been extended to a robust visual servoing method
for object manipulation against temporary loss of visual
information when occlusion occurs or the object goes outof-sight [12]. However, any time-delay of visual information
is not considered in this method explicitly.
In this paper, a new visual servoing method for object manipulation by a multi-fingered hand-arm system is proposed.
The new method is robust not only to the temporary loss ofthe visual information, but also the time-delays of the visual
information. This method utilizes a virtual object frame thatis defined by a position and attitude of each fingertip obtainedby proprioceptive sensors similar to the previous method[12]. The difference between the new one and previous one
is that information of an actual object frame obtained by avisual sensor is not directly used in a feedback controller,
but used in the design of desired virtual object frame in
the new one. Namely, the new controller is designed so as
to manipulate the position and attitude of the virtual object
frame toward its desired value designed by the informationof the actual object frame.
In what follows, the overall model composed of a multifingered hand-arm system and a grasped object is shownin section II. The virtual object frame is given in section
III. The new control scheme for robust object manipulation
using the information of both the virtual and actual object
frame is designed in section IV. Numerical simulation results
are shown to verify the performance in section V, andexperimental results to show the practical usefulness arepresented in section VI.

This work was partially supported by Grant-in-Aid for JSPS Fellows.
A. Kawamura is with Graduate School of Information Science andElectrical Engineering, Kyushu University, Fukuoka, 819-0395, JAPANII. A M ULTI -F INGERED H AND -A RM S YSTEM
AND A G RASPED OBJECT

kawamura@irvs.ait.kyushu-u.ac.jp

K. Tahara is with Faculty of Engineering, Kyushu University, Fukuoka,
819-0395, JAPAN tahara@ieee.org
R. Kurazume and T. Hasegawa are with Faculty of Information Scienceand Electrical Engineering, Kyushu University, Fukuoka, 819-0395, JAPAN{kurazume, hasegawa}@ait.kyushu-u.ac.jp
978-1-4673-1735-1/12/S31.00 ©2012 IEEE

In this section, a hand-arm system composed of an arm anda multi-fingered hand is given. In this study, assume that a
robotic hand-arm system has enough number of fingers and
DOFs to achieve stable object grasping and manipulation

4797

Capture

ObjectActual frame

Capture((t1+tsample )-tdelay= t1-t image )1/hx(t0 -timage ) x(t1 -timage ) = x(t-tdelay )xTime[s]Vision sensor

timageyz

xdvir(t0)01

timagetsamplexdvir(t1)Present

Controller
x vir

Fingertip
y vir

03

Fig. 1.

xvir(t0-timage ) xvir(t1-timage ) = xvir(t-tdelay )02

Joint encoder

z vir

Fig. 2. Time-line chart of the vision sensor, the servo-loop of the proposed
controller and the proprioceptive sensor at each joint (rotary encoder)of a grasped object. The symbol Na and Ni denote thenumber of DOFsof the arm and the ith finger respectively,N
and ND = Na + i=1 Ni is the total number of DOFs in thesystem where N is the number of the fingers. The subscriptof i in all variables and equations refers to the ith finger
hereinafter. Each fingertip is hemispheric and made of some
soft material. Then, assume that all fingertips maintain rolling
contact with the object surfaces, and do not slip and detachfrom the surfaces during manipulation. Also assume that
fingertips roll within the ranges of its hemispheric surfaces,
and they do not deviate from each initial contact surface.
Note that the gravity effect of a grasped object is ignored
for the sake of modeling easier at this stage.
III. V IRTUAL O BJECT F RAME
The virtual object frame has been reported by Wimböck
et al. [13] and Tahara et al. [14], independently. In thesemethods, the position and attitude of the virtual object frame
defined by a position of each fingertip is utilized as controlled
variables instead of an actual object frame. We also haveproposed another virtual object frame defined not only by aposition, but also an attitude of each fingertip in our previous
work [12]. In this section, the virtual object frame definedin our previous work [12] is introduced.
Firstly, the position of the virtual frame xvir is defined asN1x0i ∈ R3 ,
N i=1

(1)

where x0i ∈ R3 denotes the position of the center of eachfingertip. Equation (1) means that xvir is the centroid ofa polygon fixed by x0i . The attitude of the virtual object
frame located at xvir is expressed as a rotational matrix
Rvir = [r xvir , r yvir , r zvir ] ∈ SO(3) as shown in Fig. 1, andits components are given in the following way:Nr̃ xvir(2)∈ R3 ,
r̃ xvir = i=1 r xfi
r̃ xvir Nr̃ xvir × r̃ yvir
r yvir =∈ R3 ,r̃ yvir = i=1 r yfi (3)r̃ xvir × r̃ yvir r zvir = r xvir × r yvir ∈ R3 ,(4)

r xvir =Time[s]

Virtual frame

Actual and virtual object frame

xvir =

Time[s]xdvir(t1+tsample ) = xdvir(t)virwhere Rfi = [r xfi , r yfi , r zfi ] ∈ SO(3) indicates a local
frame located at the position of the center of each fingertip
x0i , and denotes its attitude in Cartesian coordinates.
IV. C ONTROL L AW
In this section, a control input to regulate position andattitude of a grasped object by visual servoing with considerable time-delays is designed. The basic idea of thedesign of the input is based on our previous method thatis robust to the temporary loss of the visual information
using the virtual object frame [12]. The advantage of the
new one compared with the previous one is that it is robust
not only to the temporary loss of the visual information,
but also to the considerable time-delays. This advantage isbrought by the use of the virtual object frame as a controlled
variable all of the time during manipulation. Its availability
is nothing to do with such temporary loss and time-delayof visual information because it is defined only using theinformation from proprioceptive sensors. Therefore, the new
method makes the overall system more stable than theprevious method. In addition, a desired virtual object frame
is continuously designed and updated using the informationof the actual object frame from the visual sensor. The error
between the actual object frame and the virtual object frame
is reduced in real-time by the updating manner.
A. Time-Delay
In this paper, assume that the time-delays are caused
by low sampling rate, computational cost for image processing, and data transmission latency, and that their timedelay can be measured roughly in real-time. Note that any
unstable behavior is not induced even if the measured delay
time is not accurate since the delay time is not related
to the current value of the virtual object frame. In thismethod, the time-delay due to the low sampling rate isconsidered independently of other time-delays induced by
the computational cost for image processing, and the datatransmission latency. This is because these features and theircountermeasures are different. The length of the time-delay
due to the low sampling rate is fluctuated by a timing of when
an instantaneous visual image is captured. For example, thetime-delay is fluctuated from 0 [ms] to 33 [ms] in the caseof using 30 [Hz] NTSC camera. The variance of it is now

4798

expressed as tsample , and it satisfies the following inequality.
0 ≤ tsample <

1
,
h

(5)

where h denotes the sampling rate.
The other time-delay caused by the computational cost forimage processing, and data transmission latency is expressed
as timage , and the total time-delay is expressed as tdelay . It
is given as
tdelay = tsample + timage .

(6)

The sampling rate h and the total time-delay tdelay is shownin Fig. 2.
B. Control Input
The proposed control input u(t) ∈ RND is given by thesummation of three control inputs, one is for stable object
grasping us (t), another is for position control up (t), andthe other is for attitude control uo (t). Eventually the total
control input u(t) is given asu(t) = us (t) + up (t) + uo (t).

N

J 0i (t)T (xvir (t) − x0i (t)) − C q̇(t) + g(t)

i=1

fd

Ks = N

j=1 rj,N

J 0i (t)T (xdvir (t) − xvir (t)) ,J Ωi (t)T {r xvir (t) × r xdvir (t)i=1+r yvir (t) × r ydvir (t) + r zvir (t) × r zdvir (t)} , (12)where Ko > 0 is a positive scalar constant, J Ωi (t) ∈R3×ND denotes the Jacobian matrix for the attitude angular velocity of each fingertip with respect to each joint
angular velocity q̇. The desired attitude of the virtual object frame is expressed as a rotational matrix Rdvir (t) =[r xdvir (t), r ydvir (t), r zdvir (t)] ∈ SO(3). The summation ofeach cross product in (12) implies a desired instantaneousrotational axis of the virtual object. Namely, the attitude
error between the present virtual object frame and the desired
virtual object frame can be reduced by rotating the virtual
object frame around the axis. The desired attitude of thevirtual object frame Rdvir (t) is designed in the following
manner:Rdvir (t) = Rd R(t − tdelay )T Rvir (t − tdelay ),(13)

where the desired attitude of the actual object frame is expressed as a rotational matrix Rd = [r xd , r yd , r zd ] ∈ SO(3).
The attitudes of both the actual and the virtual object frametdelay [s] one time-step before the present time are expressedas R(t − tdelay ) and Rvir (t − tdelay ) respectively, and they
are obtained at the same timing of obtaining x(t − tdelay ).
C. Conditions for convergence of a desired state(9)

In order to make the position and attitude of the actual
object frame converge to each desired value in the final
state by the control signal u(t), it is necessary to fulfill thefollowing inequalities. They are given as follows:
For the object position:(10)|{x(tB ) − xvir (tB )} − {x(tA ) − xvir (tA )}|≤ |x(tB ) − x(tA )|,For the object attitude:

 
 −1
Rα − 1
costr
2

where Kp is a positive scalar constant, xdvir (t) denotes adesired position of the virtual object frame and it is designedby considering the time-delays in the following way:
(11)

where xd denotes a desired position of the actual object
frame, x(t − tdelay ) indicates a position of the actual object
position which is obtained by a visual sensor at tdelay [s]before, and xvir (t − tdelay ) stands for a position of thevirtual object frame which is obtained at the same timingof obtaining x(t − tdelay ). The relationship between thesevalues is shown in Fig. 2.

    −1Rβ − 1 ≤ cos
tr
 
2

(14)


,

(15)

where

i=1

xdvir (t) = xvir (t − tdelay ) + (xd − x(t − tdelay )) ,

N


(8)

where J 0i (t) ∈ R3×ND denotes the Jacobian matrix forthe velocity of the position of the center of each fingertip
x0i with respect to each joint angular velocity q̇, C ∈RND ×ND > 0 is a positive definite diagonal matrix thatplays a role of the damping gain for each joint, g(t) ∈ RND
signifies gravity compensation term for the robot, and fd
indicates a nominal desired grasping force.
The control input to regulate the position of the virtual
object frame up (t) is designed asup (t) = Kp

uo (t) =Ko(7)

The control input for stable grasping us (t) is designed sothat the center of each fingertip approaches each other, andit accomplishes stable grasping of an arbitrary polyhedral
object [10]. It is given as follows:
us (t) = Ks

The control input to regulate the attitude of the virtual
object frame uo (t) is designed as

Rα = R(tB )Rvir (tB )T

R(tA )Rvir (tA )TRβ = R(tB )R(tA ) ,
T

T

(16)
(17)

and both tA and tB denote an instantaneous arbitrary time
between one time-step and the next time-step of an acquisition interval of the visual image respectively, and they satisfy
the following inequality.
1
.
(18)
h
Equations (14) and (15) mean that the error norm between
the difference of the actual object frame and the virtual object
frame at tA and that at tB is smaller than that between the4799

0 ≤ tB − t A ≤TABLE I
P HYSICAL PARAMETERS FOR THE SIMULATIONTABLE IIII NITIAL CONDITION OF THE SIMULATION
q̇
qa
q 01
q 02
q 03
ẋ
x
ω

Triple-fingered hand-arm systemst

1 link length
la11.300[m]li10.300[m]la2
1.000[m]li20.200[m]2nd link length
3rd link lengthla30.175[m]li30.140[m]1st mass center
lga1
0.650[m]lgi10.150[m]2nd mass center
lga20.500[m]lgi20.100[m]rdlga3
0.0875[m]lgi30.070[m]3 mass centerst
1 massma1
1.300[kg]mi1
0.250[kg]2nd mass
ma2
1.000[kg]mi2
0.150[kg]ma3
0.400[kg]mi3
0.100[kg]3rd mass
1st Inertia I a1
diag(7.453, 7.453, 0.260)×10−1 [kg·m2 ]nd−1diag(3.397, 3.397, 0.128)×10 [kg·m2 ]2 Inertia I a2
3rd Inertia I a3
diag(0.291, 0.291, 0.500)×10−1 [kg·m2 ]1st Inertia I i1
diag(7.725, 7.725, 0.450)×10−3 [kg·m2 ]diag(2.060, 2.060, 0.120)×10−3 [kg·m2 ]2nd Inertia I i2
diag(0.538, 0.538, 0.031)×10−3 [kg·m2 ]3rd Inertia I i3
Radius of fingertip ri
0.070[m]52
Stiffness coefficient ki
1.000×10
[N/m
]

 2
Damping function ξi
1.000× ri − Δri2 π[Ns/m2 ]R

Actual Position

x [m]

0.2

Object0.037[kg]0.092[m]0.048[m]0.048[m]1.309[rad]1.309[rad]0.524[rad]diag (1.273, 0.193, 1.148) × 10−3 [kg · m2 ]

2

4

6

8

10

Time[s]10.0[N]
4.762
0.238
diag(1.003, 0.651, 0.735, 0.278, 0.177)×10−1 [Ns·m/rad]diag(0.606, 0.687, 0.786, 0.642, 0.198)×10−2 [Ns·m/rad]diag(0.468, 0.780, 0.318, 0.099)×10−2 [Ns·m/rad]diag(0.648, 0.780, 0.318, 0.099)×10−2 [Ns·m/rad]T⎡(0.100, 0.500, 0.700) [m]⎤0.88 −0.32 −0.34⎣ 0.340.94
0.00 ⎦
0.32 −0.12
0.94

Fig. 3. Transient responses of the x-component of the position of theactual object frame x: The sampling rate is h = 20 [Hz], the time-delay
due to the computational cost for image processing, and data transmission
latency is timage = 50 [ms]40

actual object frame at tA and at tB , where the error norm of
the object attitude is represented by a rotational angle aroundthe equivalent rotational axis. Namely, a physical meaningof the conditions is that the change of the difference between
the actual object frame and the virtual object frame is smaller
than the discretization error due to the low sampling rate. The
actual object frame converges to the desired state finally if
the conditions are satisfied in every servo-loop.
V. N UMERICAL S IMULATIONThis section shows numerical simulation results to verify
the proposed controller. A triple-fingered hand-arm model isutilized in the simulations (the detail of its dynamic model
has been given in [10]). It consists of an arm part whichhas 5 DOFs and a triple-fingered hand part which has one 5
DOFs finger and two 4 DOFs fingers. A triangular prism isutilized for the grasped object as an example of polyhedral
objects. Specific parameters of the hand-arm system and thegrasped object are shown in Table I. In this Table, Yi is aperpendicular distance from the center of the object mass
Oc.m. to each surface of the object, and θti is the internalθz [deg]

Rd

0
-0.1
0

TABLE IIN OMINAL DESIRED GRASPING FORCE AND GAINS FOR THE SIMULATION
fd
Kp
Ko
Ca
C1
C2
C3
xd

xd
x new
x pre

0.1

20

Actual Attitude

Mass m
Y1
Y2
Y3
θt1
θt2
θt3
Inertia I

0[rad/s](−0.183, −1.369, 1.898, 1.343, −0.787)T [rad](1.007, 0.235, −0.771, 1.338, 0.328)T [rad](0.242, −0.733, 1.122, 0.754)T [rad](2.019, −0.924, 0.912, 1.088)T [rad]0 [m/s](0.158, 0.501, 0.681)T [m]0 [rad/s]⎡⎤
1.00 0.00 0.00
⎣ 0.00 1.00 0.00 ⎦0.00 0.00 1.00

0

-20
0

θzd
θz new
θz pre

2

4

6

8

10

Time[s]Fig. 4. Transient responses of the θz which means the actual rotational
angle around the z-axis expressed by XYZ Euler angles: The sampling rateis h = 20 [Hz], the time-delay due to the computational cost for image
processing, and data transmission latency is timage = 50 [ms]angle of a cross-sectional triangle of the object. Table IIshows the desired nominal grasping force and gains. In orderto focus on the visual servoing for object manipulation,
assume that the hand-arm system has already grasped theobject in the initial condition. The parameters of this system
in initial state are shown in Table III.
Two types of the simulations are conducted to show theadvantage of the newly proposed method. One is using
the newly proposed visual servoing method. The other isusing our previous visual servoing method which utilizes
the position and attitude of the actual object frame as acontrolled variable directly [11]. The previous method uses
control inputs for position control upreal and attitude control
uoreal of the actual object frame instead of up and uo . They

4800

Actual Position and Virtual Position vir [m]

x

0.2

xd
x
xd vir
xvir

0.1

x

0
-0.1

0

2

4

6

8

0.0[s]

5.6[s]

1.4[s]

7.0[s]

2.8[s]

8.5[s]

10

Time[s]Actual Attitude

θz and Virtual Attitude θz vir [deg]Fig. 5. Transient responses of the x-component of the position of theactual object frame x, the position of the virtual object frame xvir , and theposition of the desired virtual object frame xdvir when the control inputs
up and uo are used: The sampling rate is h = 20 [Hz], the time-delay
due to the computational cost for image processing, and data transmission
latency is timage = 50 [ms]

40

θzd
θz
θzd vir
θz vir20

04.2[s]

-20
0

2

4

6

8

10

Time[s]Fig. 6. Transient responses of the θz of the attitude of the actual object
frame R, and θzvir of the attitude of the virtual object frame Rvir expressedby XYZ Euler angles when the control inputs up and uo are used: The
sampling rate is h = 20 [Hz], the time-delay due to the computational costfor image processing, and data transmission latency is timage = 50 [ms]are given as follows:
upreal (t) = Kp
uoreal (t) = Ko

N

i=1
N


J 0i (t)T (xd − x(t − tdelay ))(19)

J Ωi (t)T {(r x (t − tdelay ) × r xd )

i=1

+ (r y (t − tdelay ) × r yd ) + (r z (t − tdelay ) × r zd ) } ,(20)
where the attitude of the actual object frame is expressed asa rotational matrix R (t) = [r x (t) , r y (t) , r z (t)] ∈ SO(3).
The sampling rate is h = 20 [Hz] and the time-delay due
to the computational cost for image processing, and datatransmission latency is timage = 50 [ms], respectively.
Figures 3 and 4 show the transient responses of the xcomponent of the actual x, and θz which means the actual
rotational angle around the z-axis expressed by XYZ Euler
angles respectively. In these figures, xnew and θznew indicate
the transient responses of x and θz in the case that the
newly proposed control inputs up (t) and uo (t) are utilized,
and xpre and θzpre indicate the transient responses of x andθz in the case that the previously proposed control inputs
upreal (t) and uoreal (t) are utilized. We see from these figuresthat in the case of using the newly proposed method, an

10.0[s]Actual frame

Desired real frame

Virtual frame

Desired virtual frameFig. 7. Snapshots of the simulation in the case that the sampling rate
h = 20 [Hz] and the time-delay timage = 50 [ms]oscillation is reduced and convergent rate is faster comparedwith the case of using the previous method, even with thelow sampling rate and large time-delays.
Figures 5 and 6 show the behavior of the position andattitude of the actual and virtual object frame when thecontrol inputs up (t) and uo (t) are used in the case thatthe sampling rate h = 20 [Hz] and the time-delay due
to the computational cost for image processing, and datatransmission latency is timage = 50 [ms]. Figure 5 shows
the x-component of the positions of the actual object framex, the virtual object frame xvir , and the desired virtual
object frame xdvir . Figure 6 shows the θz of the attitudesof the actual object frame R, θzvir of the attitude of thevirtual object frame Rvir , and θzdvir of the attitude of thedesired virtual object frame Rdvir expressed by XYZ Euler
angles. We see from these figures that xdvir and Rdvir are
actually updated with a step-like pattern in every intervalof visual image acquisition. Figure 7 shows the snapshotsof the simulation. In this figure, both the actual and virtualframes approach and finally converge to each desired value
simultaneously.
Additionally, it is confirmed that the conditions (14) and(15) to guarantee the convergence of the desired state aresatisfied in all the simulations.
VI. E XPERIMENTS
Experiments of object manipulation using a prototype are
performed to demonstrate the practical usefulness of the4801

TABLE IV
P HYSICAL PARAMETERS

Triple-fingered Robotic Hand

Three-fingered robotic hand
1st link length li1
2nd link length li2
3rd link length li3
1st mass center lgi1
2nd mass center lgi2
3rd mass center lgi3
1st mass mi1
2nd mass mi2
3rd mass mi3(Fingertip)
Radius ri
Physical properties si

Micron Tracker

0.064[m]0.064[m]0.030[m]0.023[m]0.035[m]0.010[m]0.038[kg]0.024[kg]0.054[kg]Marker

0.015[m]2.390 × 106 [N/m2 ]ObjectTABLE VD ETAILS OF THE GRASPED OBJECT
Triangular prism
Mass m0.0015[kg]Material
Styrene foam
(Figure)Length of side of triangle
0.060[m]Height0.039[m]Fig. 8. Object manipulation using the three-fingered robotic hand and thevisual sensing system

0.03

Actual Position

1.1
70.0
4.0 × 10−2diag(0.07, 0.07, 0.05, 0.03)×10−2 [Ns·m/rad]diag(0.07, 0.07, 0.05, 0.03)×10−2 [Ns·m/rad]diag(0.07, 0.07, 0.05, 0.03)×10−2 [Ns·m/rad]

0.02
0.01
0

2

4

6

8

10

Fig. 9. Transient responses of the x-component of the position of theactual object frame x

50

θxd
θx new
θx pre25

θ

proposed method. Each parameter of the system is shownin Table IV. The grasped object is a triangular prism, and
it is made of styrene form. Therefore, its weight is lightenough to ignore the effect of gravity. Each parameter of thegrasped object is shown in Table V. A stereo optical tracking
system (Micron Tracker H3-60, Claron, Inc.) is utilized tomeasure the position and attitude of the grasped object. The
system can measure the position and attitude of the grasped
object in every 50 [ms] by tracking a specific marker on
the object. The overall system which includes a prototype
robotic hand consists of three 4 DOFs fingers and the visual
sensing system is shown in Fig. 8. This visual sensing system
involves two types of the time-delays, tsample and timage
as is mentioned in section IV-A. The sampling rate of thesystem h = 20 [Hz] is not enough to use a real-time feedback
manner directly. An average of the time-delay due to image
processing and latency of data transmission is timage ≈ 50[ms]. The time-delay due to image processing is observedin real-time by the measurement of the time from capturing
an image to the beginning of data transmission. The othertime-delay due to data transmission from the vision sensing
system to the controller is measured in advance.
Two types of experiments are performed similar to thenumerical simulations. In these experiments, the robotic hand
has already grasped the object initially. One is using the
proposed method and the other is using the previous method.
Figure 9 shows the transient responses of the x-componentof the position of the actual object frame x. In this figure,0

Time[s]Actual Attitude x [deg]

fd
Kp
Ko
C1
C2
C3

x [m]

TABLE VIN OMINAL DESIRED GRASPING FORCE AND GAINS

xd
x new
x pre

0
-25
-50
0

2

4

6

8

10

Time[s]Fig. 10. Transient responses of θx of the attitude of the actual object frame
R expressed by XYZ Euler angles

xnew indicates x(t − tdelay ) in the case that the new control
inputs up (t) and uo (t) are utilized, and xpre indicates thetransient responses of x(t − tdelay ) in the case that theprevious control inputs upreal (t) and uoreal (t) are utilized.
Figure 10 shows the transient responses of θx which means
the rotational angle around the x-axis expressed by XYZ
Euler angles. In this figure, θxnew indicates θx (t − tdelay ) in
the case that the newly proposed control inputs up (t) anduo (t) are utilized, and θxpre indicates θx (t − tdelay ) in thecase that the previous control inputs upreal (t) and uoreal (t)

4802

x

x

Actual Position and Virtual Position vir [m]

0.03

xd
x
xd vir
xvir

0.02
0.01
0

0

2

4

6

8

10

Time[s]ACKNOWLEDGMENT
50

θThis work was partially supported by Grant-in-Aid for
JSPS Fellows.
R EFERENCES

θxdθx
θxd vir
θx virθ

Actual Attitude x and Virtual Attitude x vir [deg]Fig. 11. Transient responses of the x-component of the positions of theactual object frame x and the virtual object frame xvir

Firstly, the new control scheme of visual servoing for object manipulation was proposed to avoid unstable behavior
caused by two types of time-delays. In this scheme, thevirtual object frame is complementarily utilized as a nominal
controlled variable in the interval of visual image acquisition.
Next, the effectiveness of the method was verified by thenumerical simulations. Finally, its practical usefulness wasdemonstrated through the experimental results.
In our future work, the stability of the overall system must
be analyzed deeply. In addition to the detailed stability analysis, necessary conditions to realize the robustness against
time-delays must be revealed in more detail.

25
0
-25
-50
0

2

4

6

8

10

Time[s]Fig. 12. Transient responses of θx of the attitude of the actual object frame
R and θxvir of the attitude of the virtual object frame Rvir expressed by
XYZ Euler angles

are utilized. In these figures, the position and the attitude of
the actual object frame in the case of using the new control
inputs converge to each desired value. However, behavior of
the overall system becomes unstable and the object graspingis failed finally in the case of using the previous method.
Thus, it is confirmed that the new visual servoing method
accomplishes stable object grasping and manipulation even
if the sampling rate is low and there exists considerable timedelays, although the previous method cannot realize it.
Figure 11 shows the transient responses of the xcomponent of the position of the actual object frame, thevirtual object frame and the desired virtual object frame when
the control inputs up (t) and uo (t) are used. Also Fig. 12shows the transient responses of θx which means the actual
rotational angle around the x-axis expressed by XYZ Euler
angles. We see from these figures that the desired position
and attitude of the virtual object frame are actually updatedwith a step-like pattern in every interval of visual image
acquisition. Finally, both the actual and virtual object frames
mostly converge to each desired value respectively with nounstable behavior.
Thus, it is confirmed that the new visual servoing method
is practically useful through these experimental results.
VII. C ONCLUSION
This paper presented a novel visual servoing method for
object manipulation that is robust not only to temporary loss,
but also to considerable time-delays of visual information.

[1] L. E. Weiss, A. C. Sanderson and C. P. Neuman, “Dynamic sensorbased control of robots with visual feedback,” IEEE Trans. on Robot.
and Automation, Vol. 3, No. 5, pp. 404–417, 1987.
[2] J. T. Feddema and O. R. Mitchell, “Vision-guided servoing withfeature-based trajectory generation,” IEEE Trans. on Robot. and Automation, Vol. 5, No. 5, pp. 691–700, 1989.
[3] S. Hutchinson, G. D. Hager and P. I. Corke, “A tutorial on visual servo
control,” IEEE Trans. on Robot. and Automation, Vol. 12, No. 5, pp.
651–670, 1996.
[4] N. R. Gans and S. A. Hutchinson, “Stable visual servoing throughhybrid switched-system control,” IEEE Trans. on Robotbotics, Vol.
23, No. 3, pp. 530–540, 2007.
[5] P. I. Corke and M. C. Good, “Dynamics effects in visual closed-loopsystems,” IEEE Trans. on Robot. and Automation, Vol. 12, No. 5, pp.
671–683, 1996.
[6] A. J. Koivo and N. Houshangi, “Real-time vision feedback for servoing
robotic manipulator with self-tuning controller,” IEEE Trans. Systems,
Man, and Cybernetics, Vol. 21, No. 1, pp. 134–142, 1991.
[7] K. Hashimoto and H. Kimura, “Visual servoing with nonlinear observer,” Proc. IEEE Int. Conf. Robot. Automat., pp. 484–489, Nagoya,Japan, 1995.
[8] H. Fujimoto and Y. Hori, “Visual servoing based on intersample
disturbance rejection by multirate sampling control - time delay compensation and experiment verification -,” Proc. IEEE Conf. Decision
Contr., pp. 334–339, Orlando, Florida, 2001.
[9] K. Sasajima and H. Fujimoto, “6 dof multirate visual servoing for
quick moving objects,” Proc. of the 2007 American Contr. Conf., pp.
1538–1543, New York, USA, 2007.
[10] A. Kawamura, K. Tahara, R. Kurazume and T. Hasegawa, “Dynamic
grasping for an arbitrary polyhedral object by a multi-fingered handarm system,” Proc. of the 2009 IEEE/RSJ Int. Conf. on Intelligent
Robots and Systems, pp. 2264–2270, St. Louis, Missouri, 2009.
[11] A. Kawamura, K. Tahara, R. Kurazume and T. Hasegawa, “Dynamic
Object Manipulation using a Multi-Fingered Hand-Arm System: Enhancement of a Grasping Capability using Relative Attitude Constraints of Fingers,” Proc. Int. Conf. on Advanced Robotics, pp. 8–14,
Estonia, Tallinn, 2011.
[12] A. Kawamura, K. Tahara, R. Kurazume and T. Hasegawa, “Robust
manipulation for temporary lack of sensory information by a multifingered hand-arm system,” Proc. of the 2011 IEEE/RSJ Int. Conf. onIntelligent Robots and Systems, pp. – , San Francisco, CA, 2011 (toappear).
[13] T. Wimböck, C. Ott, and G. Hirzinger, “Passivity-based Object-Level
Impedance Control for a Multifingered Hand,” Proc. of the 2006
IEEE/RSJ on Intelligent Robots and Systems, pp. 4621–4627, Beijing,
China, 2006.
[14] K. Tahara, S. Arimoto and M. Yoshida, “Dynamic object manipulation
using a virtual frame by a triple soft-fingered robotic hand,” Proc.
IEEE Int. Conf. Robot. Automat., pp. 4322–4327, Anchorage, AK,
2010.

4803