Finger Motion Measurement System for
Telexistence Hand Manipulation
Yasuyuki Inoue
Institue of Gerontology
The University of Tokyo
Tokyo, Japany-inoue@tachilab.org

Fumihiro Kato
Institue of Gerontology
The University of Tokyo
Tokyo, Japan
fumihiro.kato@tachilab.org

Abstract—This research focuses on finger motion
measurement system of telexistence robot to realize precise
hand manipulation. Wearable sensor glove which measures
operator’s finger joint angles is used in most cases forreproducing his/her finger motion to robot hand. However,angle-based finger motion measurement is difficult to estimate
correct position of fingertip because of hand shape difference
between individuals. On the other hands, optical position
measurement such as motion capture can obtain exact position,
but has problem of measuring stability because of occlusion. To
overcome this issue, we propose a finger motion measurement
system which consists of sensor glove with motion capture. To
calibrate individual differences, the kinematic parameters ofoperator’s hand such as bone length are estimated at first, andthen the hand model predicts fingertip position from sensor
glove data. Once the parameters are obtained, both motion
capture and hand model provide position of fingertip in parallel,
and the measurement becomes more stably against occlusion.
The performances of the proposed system regarding parameter
estimation and precision accuracy were evaluated during finger
movement in 3D space.
Keywords—telexistence, robot hand, sensor glove, motion
captureI. INTRODUCTION
Telexistence is a concept that refers to the technology,
which enables a human to have a real-time sensation of being
at a place other than where he/she actually exist, and to interact
with the remote environment [1]. Telexistence is realized byusing human-like robot operated with master-slave systemsuch as TELESAR series [2,3,4,5,6]. Many of them are
equipped with anthropomorphic robot arm and hand for
manual teleoperation, which are controlled by the operator’s
body movement in master-slave manner. That is, operator’s
hand gesture is measured in real-time and the action isreproduced on the robot hand at the same time. To realize
precise duplication of manual operation, it is important tomeasure operator’s motion data as precisely and stably as
possible because the accuracy of reproduced robot’s finger
motion depends on accuracy of finger motion measurement
system. In most cases, operator’s hand motion is recorded byusing wearable sensor gloves and joint angle data of fingers
are used for controlling the joints of the robot hand. However,
it is difficult to estimate correct hand postures from joint
angles obtained from sensor glove unless the kinematicparameters such as bone length are exactly known, thoughthese parameters varies from person to person. Therefore,
individual difference between hands causes difficulty inprecise hand manipulation of telexistence robot.
For decades many kinds of anthropomorphic robot hands
have been developed [7,8,9,10] to replicate human equivalent
hand manipulation, and finger motion measurement system isSusumu Tachi
Institue of Gerontology
The University of Tokyo
Tokyo, Japantachi@tachilab.org

also used to obtain control signal and/or teaching data for therobot hand. These systems also suffer same problem ofmeasurement accuracy of finger motion caused by individual
differences, and various calibration methods to improve theaccuracy are proposed. Fernando et al. [11] proposed a finger
calibration method for telexistence robot which is based on
predefined hand posture matching. The adjustment coefficientof joint angle was determined by the correspondence between
actual input angle of operator’s hand and desired output angleof robot hand. The method worked decently and realized
replication of many hand shapes. However, the accuracy waslimited for calibrated hand shape and not ensured for general
postures.
In this study, we propose an novel five-finger motion
measurement system for the operator of telexistence robot,
which achieves accurate and stable position tracking of finger
motion. This system consists of two components; (1) wearable
sensor glove measuring joint angles, and (2) optical motion
capture tracking positions finger position. The key concept ofthe measurement system is that these two sensing systems areused complementarily, that is, joint angle data measured by
sensor glove are insufficient to obtain correct finger position
but has advantage of stable measurement in comparison to theoptical measurement, whereas optical motion tracking hasdisadvantage of measurement stability due to occlusion but
can obtain accurate finger position. To combine these sensor
data, this system allows to measure operator’s finger motion
more accurately and stably so that the robot hand can always
duplicate the operator’s hand manipulation as precise aspossible.
II. FINGER MOTION MESUREMENT SYSTEM
A. System Overview
Fig. 1 shows the finger motion measurement system whichis equipped with two types of motion sensing system. The
inertial measurement unit (IMU) which senses 3D rotation
angle is attached to finger segment. The optical marker whichemits infrared light is attached to fingertip. The emission lightof optical maker is received to the motion capture camera toobtain 3D position of the fingertip.
B. Sensor Glove for Attitude Measurement
An existing sensor glove system (Cobra Glove, Synertial
UK Ltd.) is employed for the measurement system to obtain
operator’s finger joint angles at the sample rate of 30 Hz
through fifteen 9-axis IMU sensors attached on the hand andfinger segments. Fig. 2 shows the layout of IMU sensors onthe hand. Obtained data are used for two ways. One is direct
output data of finger orientation, and another is input data ofXXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEEUSB ISBN 978-1-7281-4898-4, IEEE Catalog number CFP19MCR-USB

C2-1-1

way. Hence, the endpoint of the kinematic chain relative to theorigin of the hand is formulated as below;
IMU
sensor0

𝑷𝑒𝑛𝑑𝑝𝑜𝑖𝑛𝑡 = 01𝑻 ∙ 12𝑻 ∙ 23𝑻 ∙ 3𝑷𝑒𝑛𝑑𝑝𝑜𝑖𝑛𝑡()where, 𝑖−1𝑖𝑻 is 4x4 homogeneous transform matrix which
describes position and orientation of frame of i-th joint (𝑖 =1,2,3) relative to the previous joint. Assuming that rotation
components of them (attitude of finger segment) is given, Eq.
(1) derives three linear equations in terms of translation
components (length of finger segment) as shown below;

OpticalMarker

Fig. 1. Proposed finger motion measurement system. The sensing devices(IMU sensors and optical motion capture makers) are attached on a wearable
glove. IMU sensors are placed on finger segments to measure the orientation,
whereas optical markers are placed on fingertips to measure the position.

5Index3167891011Middle

132
15

16Ring

14Little

Fig. 2. Layout of fifteen IMU sensors. Each sensor fixed in finger segment
obtains local rotation angle of CM (#3), MP (#4), IP (#5), MP (#6,9,12,15),
PIP (#7,10,13,16) and DIP (#8,11,14) joint relative to the previous segment,
whereas the root sensor (#1) measures global rotation angle of hand. Notethat one sensor (#2) is not used and there is no sensor for DIP joint of little
finger due to the limitation of the system.

hand kinematic model to generate predicted finger position.
The prediction of finger position needs additional information
regarding kinematic parameters of operator’s hand, and they
are separately estimated.
C. Motion Capture for Position Tracking
An existing optical motion capturing system (OptiTrack,NaturalPoint, Inc.) is used to measure operator’s finger motion
at the sample rate of 240 Hz. The active markers are attachedon five fingertips to obtain their position with high accuracy(error < 0.1 mm, in nominal terms of the tracking system). The
captured data are used for two ways; One is direct output offinger position, and another is learning data of the kinematic
hand model to estimate operator’s parameters. The estimation
procedure is described in next section.
III. KINEMATIC HAND MODEL
A simple kinematic hand model which represents skeletal
structure of human hand is introduced to estimate operator’sfinger position in order to complement the finger motion
trajectory. For simplicity, thumb and fingers are all modeled
as same kinematic chains consisting of three spherical (3DOF)
joints because the IMU sensor used in the system can obtain
3D rotation angle of each joint around arbitrary axis. Althougheach joint of each finger actually has different DOF androtation axis, this model can represent the rotation in the same𝑝𝑥
𝑓1 (𝑙𝑥0 , 𝑙𝑦0 , 𝑙𝑧0 ⋯ 𝑙𝑥3 , 𝑙𝑦3 , 𝑙𝑧3 )[𝑝𝑦 ] = [𝑓2 (𝑙𝑥0 , 𝑙𝑦0 , 𝑙𝑧0 ⋯ 𝑙𝑥3 , 𝑙𝑦3 , 𝑙𝑧3 )]𝑝𝑧
𝑓3 (𝑙𝑥0 , 𝑙𝑦0 , 𝑙𝑧0 ⋯ 𝑙𝑥3 , 𝑙𝑦3 , 𝑙𝑧3 )()where, 𝑙𝑥𝑗 , 𝑙𝑦𝑗 , 𝑙𝑧𝑗 are link length parameters of each segment(𝑗 = 0,1,2,3). These parameters ought to be known prior to
the kinematics calculation of Eq. (1). However, theseparameters are all different between individuals. If thegeneral finger lengths are used for the parameter, it leads
position errors of fingertip due to individual difference. In the
proposed method, the numerical estimation is performed toknow the parameters. This parameter estimation deals withindividual body size difference among operators. This should
be done at least once for an operator before starting handmanipulation. That is, it is performed as initial calibration of
the kinematics model. Detailed calibration procedure is asbelow;
1. To record series of fingertip position data and finger
segment attitude data while performing finger movement.
2. To substitute the pairs of position/attitude data to Eq.
(2) and derive simultaneous equations in terms of link
length parameters.
3. To solve the simultaneous equations via least squares
and obtain the estimates of the parameters.
4. To repeat same procedure for all fingers.
Note that at least four measurement data (i.e. different finger
shapes and fingertip positions) are necessary to solve theequations because of twelve unknown parameters. As long as
the obtained parameters are valid, the kinematic model
calculates predicted positions of operator’s fingertips.
IV. INTEGRATION OF SENSOR DATA
Fig. 3 shows the process flow of sensor data. As mentioned
above, motion capture is mainly used for position
measurement of fingers whereas sensor glove is in charge oforientation measurement of fingers. Also, two sensor data areapplied for estimating kinematic parameters of hand andgenerating predicted positions of fingers. When initial
calibration (or re-calibration) is performed, the parameter
estimator updates the length parameters of each joint by usinglatest sensor data and send them to the hand model. The
predicted position data are adopted in the case motion capturesystem fails to track the optical marker. Furthermore, under
the condition of even worse tracking, the prediction data ismainly used for keeping stable position measurement.

C2-1-2

MotionCapture𝑝Position
Parameter
Estimator

𝑙𝑗𝑝

Hand
Model

Orientation

Fig. 3. Block diagram of the measurement system. The output position
source is switched according to measurement quality of motion capture.

Fig. 4. Predicted position of the person’s thumb with the true value. Thesegraphs represent time series of position data for each axis, which wasrecorded while the person performed voluntary finger movement.

V. EXPERIMENTAL VERIFICATION
A. Parameter Estimation
An experimental measurement was conducted to verify
whether the estimated parameters were valid. An adult whowore the measurement system performed some voluntary
finger movement and the link length parameters were
estimated by using the measurement data. Specifically, theman moved his thumb freely for approximately 20 seconds
while recording the finger movement. Then, recorded data
were employed for estimating the link length parameters.
Table.1 shows estimated length parameters of the thumb. As
shown in Table. 1, estimated finger length is different from themeasured length which was visually measured by using a ruler.
The error of each segment length is distributed up toapproximately ± 10mm. However, the total length of eachfinger is similar. Please note that the ruler measurement might
be also different from the true bone length of the phalanges.
B. Evaluation of Predicted Position Data
The experimental measurement described above was also
conducted to evaluate the prediction accuracy of the handmodel with valid parameters. After parameter estimation,subsequent measurement was performed for the prediction.
Specifically, the man moved his thumb freely same as theprevious measurement, and recorded IMU sensor data wereinputted to the hand model and the predicted finger position
was obtained. Besides, the motion tracking data were used as
the reference of the prediction. Fig. 4 shows predicted finger
position calculated from the hand model, and true positions ofthem recorded from the motion capture are also shown. As
shown in Fig. 4, the predicted position is quite accurate, andthe root mean square error (RMSE) of the finger position is on
average 5.22mm.

VI. CONCLUSION
In this study, a five-finger motion measurement system
was developed for the precise hand manipulation of robotic
finger. This measurement system consists of two different
types of motion sensors to compensate disadvantages of themin order to keep accurate and stable measurement of finger
motion. In the system, there is a kinematic hand model topredict the fingertip positions. An experimental measurement
was conducted to evaluate validity of the prediction method,and high prediction accuracy was confirmed. However, theprediction accuracy seems to depend on the position tracking
quality of motion capture during the parameter estimation
because the outlier affects the solution. Therefore, outlier
elimination based on the measurement reliability is necessary.
ACKNOWLEDGMENT
This research is supported by ACCEL Program fromJapan Science and Technology Agency, JST.
REFERENCES
[1]
[2]

[3]

[4]

[5]

[6]
TABLE I.

ESTIMATED PARAMETERS OF FINGER SEGMENTS

Thumb length (mm)

Estimated

Measuredmetacarpus

32.2

43

proximal phalanges

36.2

35

distal phalanges

41.1

30

total

109.5

108

[7]

[8]

S. Tachi, Telexistence (2nd Edition). World Scientific, 2015.
S. Tachi, H. Arai and T. Maeda, “Tel-existence master-slave systemfor remote manipulation”, Proceedings ofIROS ’90. IEEE
International Workshop, pp. 343–348, 1990.
N. Kawakami and D. Sekiguchi, “TelesarPHONE – Communication
Robot based on Next Generation Telexistence Technologies -”,International Symposium on Robotics, pp. 1 – 4, 2005.
K. Watanabe, I. Kawabuchi, N. Kawakami, T. Maeda and S. Tachi,“Torso: completion of egocentric telegnosis system”, ACM
SIGGRAPH 2007 emerging technologies, 2007.
S. Tachi, K. Watanabe, K. Takeshita and K. Minamizawa, “Mutual
Telexistence Surrogate System: TELESAR4 - telexistence in realenvironments using autostereoscopic immersive display -”, IEEE/RSJ
International Conference on Intelligent Robots and Systems, pp. 25 –30, 2011.
C. L. Fernando, M. Furukawa, T. Kurogi, S. Kamuro, K. Sato, K.
Minamizawa and S. Tachi, “Design of TELESAR V for TransferringBodily Consciousness in Telexistence”, 2012 IEEE/RSJ International
Conference on Intelligent Robots and Systems, pp.5112 – 5118, 2012.
L. B. Bridgwater, C. Ihrke, M. A. Diftler, M. E. Abdallah, N. A.
Radford, J. Rogers, S. Yayathi, R. S. Askew, and D. M. Linn. The
robonaut 2 hand‐designed to do work with tools. InRobotics andAutomation (ICRA), 2012 IEEE International Conference, pp. 3425–3430, 2012.
M. Reichel, Transformation of shadow dextrous hand and shadow
finger test unit from prototype to product for intelligent manipulationand grasping. Inter Conf on Intelligent Manipulation and Grasping.
123-124, 2004.

C2-1-3

[9]H. Liu, K. Wu, P. Meusel, N. Seitz, G. Hirzinger, M.H. Jin, Y.W.Liu,S. W. Fan, T. Lan, and Z. P. Chen, Multisensory five-finger dexterous
hand: the DLR/HIT hand II. Proceedings of IROS. pp. 3692-3697,
2008.
[10] T. Mouri, H. Kawasaki, K. Yoshikawa, J. Takai, S. Ito,
Anthropomorphic robot hand: Gifu hand III, Proc. Int. Conf. Control
Autom. Syst., pp. 1288–1293, 2002.
[11] C. L. Fernando, M. Furukawa, K. Minamizawa, S. Tachi, Experiencing
ones own Hand in Telexistence Manipulation with a 15 DOF
Anthropomorphic Robot Hand and a Flexible Master Glove,
Proceedings of the 23rd International Conference on Artificial Realityand Telexistence (ICAT), Tokyo, Japan, pp.20-27, 2013.

C2-1-4