Estimating Finger Postures by Attaching an Omnidirectional
Camera to the Center of a User's Palm
Yusuke Maruyama and Yasuyuki Kono
Graduate School of Science and Technology, Kwansei Gakuin UniversitySanda, Japan{htks0521, kono}@kwansei.ac.jp

ABSTRACTThis research describes the development of a system that
estimates the natural postures of a user’s fingers from the images
captured by an omnidirectional video camera attached to thecenter of the user’s palm in real time. The finger postures can be
estimated by detecting the fingertips on each image and referringto the following preset information: the positional relationshipbetween the camera and the user’s fingers/fingertips, the length
between the finger joints, and the interdependencies between thefinger joints.
1

CCS CONCEPTS• Human-centered computing → Human computer interaction ;KEYWORDS
Finger Detection, Finger Posture, Omnidirectional Camera

1

INTRODUCTION

This paper describes our method for estimating the natural
posture of a user's fingers from the images captured by anomnidirectional camera attached to the center of the user’s palm.
Research on the three-dimensional reconstruction of a finger
posture is actively being conducted for user interfaces such as ARand VR. For instance, Leap Motion [1] which estimates theposition and orientation of a user's hand with an active infrared
camera, and Myo [2] which estimates hand motion via wearable
hardware equipped with both an acceleration sensor and anelectromyogram sensor on a user's arm have been developed. Our
system estimates natural finger postures from the fish-eye image
captured by an omnidirectional camera by detecting the fingertips
in an image and referring to the following preset information: thepositional relationship between the camera and user’s
fingers/fingertips, the length between finger joints, and theinterdependencies between the finger joints. In [3], the idea ofestimating a user’s hand pose by integrating multi-viewpoint
silhouette images was proposed. The system requires plural
cameras for capturing multi-viewpoint silhouette images. In [4], asystem for detecting continuous hand openness using a Kinectlike device was proposed. However, this system cannot estimate
the posture of each finger. Our system only requires that anomnidirectional video camera be attached to the center of theuser’s palm in order to estimate the natural posture of each finger
in real time. The user attaches RICOH THETA S to the center ofeach palm as depicted in Figure 1 so as to not restrict themovement of the user’s fingers as far as possible.

2 METHOD
Our idea for finger posture reconstruction is in the following:1.

2.

Calculate the angle between the vector from the lens to eachfingertip and the camera body, i.e., θtip in Figure 2, bydetecting the fingertip from the captured fish-eye image,Calculate the finger joint angles from the θtip by applying
inverse kinematics.

Figure 1. Attaching the omnidirectional camera

Figure 2. The finger posture considered in this researchable to estimate the finger postures with an acceptable accuracy if
the fingers were successfully detected.
We are also currently working to detect the other hand andestimate the three-dimensional position and posturecorrespondences of both hands by attaching omnidirectional
cameras to both palms. We plan to develop AR applications
employing 3D hand postures and finger angles, e.g., pottery
wheel-throwing.
(a) Dependency between 𝜽𝑷𝑰𝑷 and 𝜽𝑫𝑰𝑷 (b) Dependency between 𝜽𝑴𝑷 and 𝜽𝑷𝑰𝑷

Figure 3. The dependencies of each joint angle
The following section describes our method in detail. We apply
the expression proposed by Sherrah et al. [5] to each pixel andextract the skin color region from the fish-eye image. We apply
distance transformation to the binarized skin color regions toextract relatively large skin color regions and consider them as
finger region candidates. We then apply a template-matching
method to each finger region candidate with fingertip templates inorder to detect a fingertip within the search area set by referring to
the finger joint constraints proposed by Fujiki et al. [6]. Next,
plausible candidates are selected as finger regions. The farthest
pixel on the contour of a finger region from the base of the finger
is acquired as its fingertip pixel. The θtip of each finger iscalculated from the fingertip pixel with a fish-eye lens projection
model and thus, the joint angles can be calculated based on
inverse kinematics, as shown in Figure 3, by referring to themovable range and the interdependencies of the finger joints also
proposed in [6]. We also estimate the natural finger postures fromeach calculated joint angle and the preset distance between thejoints of the fingers.

3DISCUSSION

We conducted an experiment to evaluate the effectiveness of ourmethod. We captured images of a subject's little finger to measurethe θtip value with an additional video camera from the side of thehand and calculated each joint angle based on the value of θtip. We
then compared the little finger joint angles detected by theproposed method with the measured angles. The results of theexperiments in five postures of the subject's little finger are shownin Figure 4. Given that the error was within 5% of the angle rangeof each joint angle in the cases from posture 1 to posture 3, weconsider that our method can estimate finger postures with anacceptable accuracy. Our method, however, could not detect thelittle finger because the brightness was insufficient in posture 4,
where the finger touched the lens. In addition, the finger was notin view of the camera in posture 5. In fact, we believe that theproposed method could be improved by continuously trackingfingers.

423

4

CONCLUDING REMARKS

We have developed a system to estimate the natural finger
postures from an image captured by an omnidirectional camera
attached to the center of the user’s palm in real time. We were2

1

5Figure 4. Reconstruction and the finger pictures ofPosture 1 to Posture 5

REFERENCES
[1]
[2]
[3]
[4]
[5]

[6]

Weichart, F., Bachmann, D., Rudak, B. and Fisseler, D. 2013. Analysis of theAccuracy and Robustness of the Leap Motion Controller. Sensors 2013, 13,6380-6393.
Thalmic Labs. (n.d.). Retrieved October 17, 2017, fromhttps://www.myo.com/
Etsuko Ueda, Yoshio Matsumoto, Masakazu Imai, Tsukasa Ogasawara. 2003.
A Hand-Pose Estimation for Vision-Based Human Interfaces, IEEE
Transactions on Industrial Electronics, Vol.50, No.4, pp.676-684.
Gentile, V., Sorce, S., & Gentile, A. 2014. Continuous Hand Openness
Detection Using a Kinect-Like Device. CISIS 2014, pp. 553–557.
Jamie Sherrah and Shaogang Gong. 2001. Skin Colour Analysis. Retrieved
October 17, 2017, fromhttp://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/GONG1/cvOnli
ne-skinColourAnalysis.html
Fujiki R., Arita D., Taniguchi R. 2005. Real-Time 3D Hand Shape Estimation
Based on Inverse Kinematics and Physical Constraints. In: Roli F., Vitulano S.
(eds) Image Analysis and Processing – ICIAP 2005. ICIAP 2005. Lecture
Notes in Computer Science, 3617. Springer, Berlin, Heidelberg.

3