SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 1(1); June (2020), pp. 22–26
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

A Fundamental Study on Hidden Surface Removal forInteractions between User’s Bare Hands and Virtual Objects inAugmented Reality
Makoto Sakamoto1, Takahiro Ishizu1, Kenji Sakoma1, Takahiro Shinoda1, Amane Takei1, Takao Ito2
Department of Computer Science and System Engineering, University of Miyazaki, Miyazaki-City, 889-2192, Japan
Graduate School of Engineering, Hiroshima University, Higashi-Hiroshima, 739-8527, Japan12

ARTICLE INFO
Article History
Received 23 October 2019
Accepted 01 July 2020
Keywords
Augmented reality (AR)Occlusion problem
Hidden surface removal
Hand detection

ABSTRACT

Augmented reality (AR) technology is a technique of superimposing information generated by acomputer on perceptual information that we receive from real space. Recently, much attention has
been focused on interaction techniques between users and virtual objects, such as the user directly
manipulating virtual objects with his/her bare hands. On the other hand, in AR technology, sincethe 3-dimensional (3D) model is superimposed on the image of the real space afterwards, it isalways displayed on the front side than the hand. Thus, it becomes an unnatural scene in somecases (occlusion problem). In this study, this system considers the object-context relations
between the user's hand and the virtual object by acquiring depth information of the user's finger.
In the evaluation experiment, it is confirmed that the hidden surface removal in this study not only
makes it possible to consider the object-context relations but also can distinguish between finger
boundaries and to clarify and process finger contours.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

and the user cannot see the object-context relations of thevirtual object and his/her hand, and feels that it is difficult
to manipulate the virtual objects.
In the existing study [4], the system used a transparent
3D model and the 3D model followed each fingers of theuser. In this way, they performed hidden surface removal
based on the depth information of user and 3D model.
However, since the 3D model to be followed is larger
than the finger, a wider range than the actual finger was
displayed on the front (see Fig. 1).

1. Introduction
In recent years, augmented reality (AR) has become
widespread [1],[2],[3]. Further, interaction with thevirtual objects is required in those papers.
Therefore, we think that it requires a more accurate
technology, because it is expected that interaction
technology using AR technology will be more and moredeveloped in the future.
On the other hand, since hands are our main means ofinteraction with objects in real life, it is necessary for AR
interfaces to be able to manipulate virtual objects with theuser's bare hand.
However, since the 3-dimensional (3D) model
displayed by the AR is superimposed on the image of thereal space afterwards, the 3D model is always displayedon the front side and user's hand is hidden by virtualobjects. Thus, the scene may become an unnatural scene,

Fig. 1: Hidden surface removal in existing study.

Corresponding author’s E-mail: sakamoto@cs.miyazaki-u.ac.jp, itotakao@hiroshima-u.ac.jp22

In this study, we realize hidden surface processing along
the fingertips of the user. Thus, we will be able to apply
this system to more advanced interaction operations.
2. Proposal
In order to realize hidden surface removal along thecontour of the user's hand, this system detects hand areaof the user by extracting depth information, color
information and edge detection in the user's hand.

Fig. 3: Blue Points.

This system converts RGB images obtained from a web
camera into HSV images.
Next, the HSV image pixel corresponding to the pixel
position where the blue point group is located is partiallyacquired.
In this study, in order to determine the range of “skincolor”, this system separates H, S and V for each pixel ofthe image, and creates the histograms. In the histogram,pixels with a small number are rejected. After that, acertain range is determined for the average value, and acolor falling within the range is recognized as “skincolor”.
In this study, we assume that each of obtained H, S, andV data follow a normal distribution.
3 σ rule [5] is known as a simple method for outlierdetection.
In this method, an observation is considered as anoutlier if its least squares residual exceeds three times its
standard deviation.
Thus, in this study, a pixel that falls within the range of3σ from the average value of each of the HSV values of
the colors acquired from the red dot group is taken as“skin color” using 3σ rule.

2.1. System Component
This system consists of a Web camera, the Leap Motion
Controller, PC and AR marker.
In this study, in order to get accurate position informationof user's fingers, we use the Leap Motion Controller.
This system controls the position and orientation of thevirtual objects by recognizing the AR marker with Webcamera.
However, when the user manipulates virtual objects,
the AR marker may be covered by his/her own hand.
Therefore, this system adopts a method of treating
Marker A to Marker F as one marker (see Fig. 2). Thus,
even when a part of the marker is covered, the virtualobjects can be displayed properly. We place the Leap
Motion Controller on the position of Marker B.
This system displays the virtual objects on the positionof Marker C and Marker G (see Fig. 2).

2.2. Generation of binary image
This system generates binary image to detect the user's
hand area using “skin color”. The generated image
contains noise, so remove it. (see Fig. 4)
Fig. 2: AR marker.

2.2 Determination of “skin color”
This system extracts the area of skin color for detectionof the user's hand area. Therefore, it is necessary to define
the “skin color” color.
First, this system plots blue points on the finger which
are the closest to the camera from the position
information obtained from Leap Motion Controller (seeFig. 3).

Fig. 4: Binary Image.

23

2.4. Generate image considering depth
information

2.5. Generate images which always isplay thehand in front of virtual objects

This system acquires 3D coordinates of the distal bone,
middle phalanx, basal bone and metacarpal bone of theuser's hand with Leap Motion Controller.
Based on the acquired position information of thefingers, this system generates images plotting blue point
group (following the position of the thumb) and green
point group (following the position of a finger other thanthe thumb).
Blue point group displays at the positions of each jointof the thumb when the user's thumb is front of the backobject (see Fig. 5).

This system synthesizes the original RGB image (seeFig. 7-a) only in the white area in Fig. 4 to the image in
Fig. 7-b. Thus, this system generates images that theuser's hand is always displayed on the front of the virtual
object (see Fig. 7-c).

Fig. 7: Generation of images which hand is displayed front.

2.6. Generate result image

Fig. 5: Blue point group following the thumb.

The area in Fig. 7-b corresponding the area of the blue
point group, the green point group and the red point
group (in the case where the position of the proximal
phalanges of thumb is in front of the virtual object) of Fig.
8-a are replaced with the image shown in Fig. 8-b (seeFig. 7-c).
Thus, based on the depth information, this systemgenerates images that the hand is displayed in front of thevirtual object (see Fig.8-c).

In addition, green point group displays at the positionsof each joint of the fingers except thumb when the user'sfingers except thumb are front of the back object (see Fig.
6).

Fig. 6: Blue point group and green point group following thewhole hand.

Furthermore, this system makes it possible to handle
multiple virtual objects of different depths. The objectcontext relations between the back virtual object and thehand of the user is determined depending on whether or
not blue point group and green point group are displayed.
On the other hand, the object-context relations betweenthe virtual object on the near side and the user's hand isdetermined by using the Z-buffer method.
Using the Z-buffer method, when a finger is positioned
behind the virtual object, blue point group and green
point group following the position of the finger are
hidden by the virtual object.
In this way, this system can determine the objectcontext relations between the user's hand and the virtualobjects even if there are multiple virtual objects.

Fig. 8: Generation of result images.

3. EXPERIMENTAL RESULTS
We executed this system to confirmed whether hidden
surface removal along the fingers is possible or not.
First, we checked whether this system can cope withmultiple virtual objects. The object-context relations
between the virtual object on the near side and the user’s
hand is determined by using the Z-buffer method. Inorder to confirm whether this is properly processed, weplaced the virtual object at position G in Fig. 2 and24

performed manipulation that a user grabs the virtualobject.

The distinction between the virtual object and the user's
hand was not clear when using the Z-buffer method when
considering the depth between the virtual object and theuser's hand.
Therefore, if there is only one virtual object or if thedepth coordinates of the virtual objects are all at the same
position, it is better to have blue point group follow thethumb of the user only than to have blue point group andgreen point group follow the palm of the user's hand.
On the other hand, the processing may be delayed, so
we need to solve this.

3.1. In the case of an object behind
Since only the thumb comes to front side than MarkerC, the blue point group follows only the thumb (see Fig.
5) and draw the thumb properly on the front of the virtual
object (see Fig. 9).

5. Conclusion
In this study, we paid attention to detection the contourof the hand to realize hidden surface removal along theuser's fingers. In this system, processing based on hand
depth information, color information and edge detection
is conducted to detect hand contours.
By these processes, hidden surface processing along
the fingertip was realized.
In the evaluation experiment, it was shown that it cancope with multiple virtual objects.

Fig. 9: Result at the position of C in the AR marker.

3.2. In the case of an object in front
This system conducted the hidden surface removal
considering multiple virtual objects by the Z-buffer
method. The blue point group follows only the thumb andthe green point group follows the other fingers (see Fig.
6). Besides, only the point group of the thumb was drawn
on the front of the virtual object by the Z-buffer method.
As a result, it was possible to draw only the thumb on thefront (see Fig. 10).
The blue point group and green point group in Fig. 6
are based on the depth information of the user's fingers.
Also, the anteroposterior relationship between the bluepoint group and the green point group is correctly
displayed by the Z-buffer method.
In this way, when the thumb is positioned behind theother fingers, the green point group is displayed on thefront side of the blue point group.

References
1.

2.
3.

4.

5.

Fig. 10: Result at the position of G in the AR marker.

4. Consideration
We confirmed that this system can perform correctly
hidden surface removal even when a user manipulates
multiple virtual objects by Z-buffer method from theexperiment result.

25

Takahiro Ishizu, Makoto Sakamoto, Takaaaki Toyota,Masamichi Hori, Satoshi Ikeda, Amane Takei, Proposal
for Interaction Techniques for Intuitive Virtual Objects
Manipulation in Augmented Reality, Proceedings of the2019 International Conference on Artificial Life andRobotics, pp.237-240, 2019.
Kangdon Lee, Augmented Reality in Education andTraining, TechTrends, vol.56, no.2, pp.13-21, 2012.
Carmigniani J, Furht B, Anisetti M, Ceravolo P,Damiani E, Ivkovic M, Augmented reality technologies,systems and applications, Multimedia Tools andApplications 51:341-77, 2011.
R. Katahira, M. Soga, Development and Evaluation of aSystem for AR enabling Realistic Display of Gripping
Motions using Leap Motion Controller, Procedia
Computer Science 60:1595–1603, 2015.
Hekimoglu, S., Koch, K. R., How can reliability of the test
foroutliers
bemeasured?,Allgemeine
Vermessungsnachrichten, 7, 247–253, 2000.

============================================

Prof. Amane TakeiAuthors Introduction

Makoto Sakamoto received thePh.D. degree in computer scienceand systems engineering from
Yamaguchi University. He ispresently an associate professor in
the Faculty of Engineering,
University of Miyazaki. He is atheoretical computer scientist, andhis current main research interests are automata theory,languages and computation. He is also interested indigital geometry, digital image processing, computer
vision, computer graphics, virtual reality, augmentedreality, entertainment computing, complex systems andso on.

Amane Takei is working asAssociate
ProfessorforDepartment of Electrical andsystems Engineering, University ofMiyazaki, Japan. His researchinterest includes high performancecomputing for computational
electromagnetism,
iterative
methods for the solution of sparselinearsystems,
domaindecomposition methods for large-scale problems. Prof.
Takei is a member of IEEE, an expert advisor of The
InstituteofElectronics,
Information
andCommunication Engineers (IEICE), a delegate of theKyushu branch of Institute of Electrical Engineers ofJapan (IEEJ), a director of Japan Society for Simulation
Technology (JSST).

Mr. Takahiro Ishizu

Dr. Takao Ito

Dr. Makoto Sakamoto

Takahiro Ishizu is a master student
at Department of Computer ScienceandSystemEngineering,
University of Miyazaki. His current
research interests are computer
graphics and image processing.

Dr. Takao Ito is Professor ofManagementofTechnology(MOT) in Graduate School ofEngineeringatHiroshima
University.
Heisserving
concurrently as Professor of Harbin
Institute of Technology (Weihai)
China. He has published numerous papers in refereed
journals and proceedings, particularly in the area ofmanagement science, and computer science. He has
published more than eight academic books including abook on Network Organizations and Information(Japanese Edition). His current research interests
include automata theory, artificial intelligence, systems
control, quantitative analysis of inter-firm relationships
using graph theory, and engineering approach oforganizational structures using complex systems theory.

Mr. Kenji Sakoma
Kenji Sakoma is a master student at
Department of Computer Scienceand System Engineering, Universityof Miyazaki. His current researchinterests are computer graphics andimage processing.
Mr. Takahiro Shinoda============================================

Takahiro Ishizu is a master student at
Department of Computer Scienceand System Engineering, Universityof Miyazaki. His current researchinterests are computer graphics andimage processing.

26