Journal of Robotics, Networking and Artificial Life
Vol. 6(4); March (2020), pp. 265–269
DOI: https://doi.org/10.2991/jrnal.k.200221.003; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

Intuitive Virtual Objects Manipulation in Augmented Reality:
Interaction between User’s Hand and Virtual Objects
Makoto Sakamoto1,*, Takahiro Ishizu1, Masamichi Hori1, Satoshi Ikeda1, Amane Takei1, Takao Ito2
Department of Computer Science and System Engineering, University of Miyazaki, Miyazaki 889-2192, Japan
Graduate School of Engineering, Hiroshima University, Higashi-Hiroshima 739-8527, Japan

12ARTICLE INFOABSTRACT

Article History

Augmented Reality (AR) is a technological advancement which brings to reality the intuitive interaction that a user is attracted
to virtual objects. In this paper, we focus on intuitive manipulation of virtual objects with bare hands. Specifically, we propose
a method to deform, move, and join virtual objects by intuitive gestures, and aim at intuitive interactions with virtual objects.
In order to manipulate virtual objects with bare hands, the position coordinates of the user’s fingers are acquired using theLeap Motion Controller. Additionally, we use Bullet Physics as an engine to represent virtual objects. On the other hand, in AR
technology, since the three-dimensional (3D) model is superimposed on the image of the real space afterward, it is always displayedon the front end rather than the user’s hand (occlusion problem). Thus, the user cannot perform an intuitive manipulation. We
solve the occlusion problem by performing appropriate hidden surface removal. From the results of a study using data gatheredfrom a questionnaire survey, we found support for the intuitive interaction between the user and virtual objects.

Received 23 October 2018
Accepted 22 November 2018
Keywords
Augmented reality
virtual objectsocclusion problem
hidden surface removal© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
Augmented Reality (AR) is a technique of superimposing information generated by a computer on perceptual information that
we receive from real space. The combination of AR technology
with many contents creates new type of applications and AR
is used in many fields, including education, medicine andentertainment [1–3].
More recently, techniques for realizing interaction between a user
and virtual objects in AR have garnered much attention. Therefore, it
is expected that these technologies will be used in various applicationsin the future. On the other hand, most of the existing studies focusedon interactions using rigid virtual objects. However, in this investigation, we aim to realize a flexible virtual object that can be deformed—
much like clay—and use it in a wider range of applications.
This paper introduces examples of flexible virtual objects.
Kato et al. [4] introduced flexible body virtual objects by using
Bullet Physics (or physics engine), reflecting the action of crushingon the virtual object with the user’s bare hand in the virtual space,and realized the deformation manipulation of virtual objects byinteraction with the user.
Suzuki et al. [5] depicted the moving manipulation of rigid body virtualobjects through an interaction between the user and virtual objects.
However, after deforming the virtual object, moving or rejoining thevirtual object while maintaining the original shape was not possible.
Corresponding author. Email: sakamoto@cs.miyazaki-u.ac.jp*

To the best of our knowledge, a series of movements that deformed,moved, or joined virtual objects were not realized.

2. PROPOSAL
In this paper, we aim to manipulate virtual objects more intuitively by allowing a user to perform basic manipulations,such as creating virtual objects, deforming, moving and combining them.
We make it possible to realize the interaction of the user and thevirtual objects by adopting the Leap Motion Controller specialized
for acquisition of three-dimensional (3D) coordinates of fingers
and introducing flexible body virtual objects using Bullet Physics.
We also perform a hidden surface removal to allow the user to recognize the context of the user’s hand and the virtual objects.

3. SYSTEM COMPONENTThis system consists of a Web camera for real image acquisition andmarker recognition, the Leap Motion Controller for obtaining 3D
coordinates of fingers, and PC for performing arithmetic processing and video output. In this system, we use network programming
to prevent interference between the Leap Motion Controller andthe Web camera. This system acquires the 3D coordinates of fingers
with the Leap Motion Controller on the server side. In addition,
this system acquires images with the Web camera, and performs
physical calculation, and video output on the client side.

266

M. Sakamoto et al. / Journal of Robotics, Networking and Artificial Life 6(4) 265–269

As shown in Figure 1, we place the paper on which the AR marker
is printed and the Leap Motion Controller, and observe the paperwith the Web camera. The virtual objects are displayed by recognizing the AR marker.
This system consists of virtual objects of the following types:•• Virtual object to be manipulated: a virtual object of an object tobe actually manipulated using bare hands.
•• Virtual objects attached to the fingertips: virtual objects following the tip of the user’s thumb and index finger.
•• Virtual object for movement: a virtual object providing the position information accompanying “virtual object to be manipulated”.
However, “virtual objects attached to the fingertips” and “virtualobject for movement” are hidden.

4. MANIPULATION OF VIRTUALOBJECTS BY GESTURE
In this section, we will explain the manipulation of virtual objects
using bare hand gestures.

4.1. Generation of Virtual Object
The user can newly generate a “virtual object to be manipulated”,
which size is desired by user by gesture of index fingers of bothhands (Figure 2).
The procedure will be described below.
1. This system acquires the 3D coordinates by putting the user’s
index finger of the right hand on Leap Motion Controller.
2. This system acquires the 3D coordinates by putting the user’sindex finger of left hand on the Leap Motion Controller while
keeping the state of Step 1.

3. The user moves away the own left index finger from the Leap
Motion Controller.
4. This system determines the size of the “virtual object to bemanipulated” based on the value when the index finger ofboth hands comes closest to generate the object.

4.2. Deformation and Moment of
the Virtual Object
It is difficult to distinguish whether the user wants to grasp orwants to move the virtual object. Therefore, this system performs deformation of the virtual object when the user’s palm isdownward, and movement of the virtual object when the user’spalm is upward.
This system decides that the palm is upward when the user’s littlefinger, ring finger, middle finger, index finger, thumb line up fromthe left of the Leap Motion Controller.
The system performs deformation and movement manipulations
on virtual object with only the right hand.
The user turns the palm downward and squeezes the “virtual object
to be manipulated” with the thumb and index finger, whereby theuser’s “virtual objects attached to the fingertips” and “virtual object
to be manipulated” collide with each other, and the “virtual object
to be manipulated” can be deformed.
The user can move “virtual object to for movement” which holdthe representative coordinate of “virtual object to be manipulated” by turning the user’s palm upward and moving his/her
hand near the virtual object. Since “virtual object for movement”
is attached to “virtual object to be manipulated” in advance, “virtual object to be manipulated” can be moved by moving “virtualobject for movement”.
In the case where two virtual objects are generated, this system
obtains the Euclidean distance between the center of the handand each “virtual object for movement” of each of the two virtualobjects and selects a closer “virtual object for movement” as themoving target.

4.3. Joining of the Virtual Objects
The user can join two virtual objects by bringing one “virtualobject to be manipulated” closer to the other “virtual object to bemanipulated”.
The procedure is described below.
Figure 1 | System component.

1. The user brings one “virtual object to be manipulated” closer
to the other “virtual object to be manipulated”.
2. Since this system is not performed the collision determination
between the “virtual objects to be manipulated”, the user canbring the two “virtual manipulation target objects” closer toeach other so as to overlap with each other.

Figure 2 | Gesture for object generation.

3. When the two “virtual objects to be manipulated” approacheach other to where they overlap, the two virtual objects
are joined.

M. Sakamoto et al. / Journal of Robotics, Networking and Artificial Life 6(4) 265–269

267

4.4. Movement after Joining theVirtual Objects
The user can move the joined virtual object. The procedure isdescribed as follows:
1. The user performs a gesture of “movement of a virtual object”
near the joined virtual object.
2. The each of two “virtual object for movement” move according to the user’s finger coordinates.
3. As a result, it is possible to move the “virtual object to bemanipulated” in the joined state.

5. EXECUTION RESULTThe execution result of this system is shown in Figures 3–6.
As depicted in Figure 3, we confirmed that “virtual object to bemanipulated” is newly generated by the user’s generation gesture.
Figure 4 is a deformation manipulation of the virtual object. We
confirmed that the virtual object can be deformed by crushing thevirtual object with the user’s thumb and index finger.

Figure 6 | Joint and move.

Figure 5 is a movement manipulation of the virtual object. The virtual object on the right side of the figure is moved to the left side ofthe figure. Finally, the two virtual objects are combined.
Figure 6 is a movement manipulation of the virtual object afterjoining. The two virtual objects move with the virtual objects didjoin was confirmed.

6. EVALUATION
We conducted questionnaire survey in order to evaluate whether
the user can intuitively perform a series of tasks such as creation,deformation, movement, and combination of virtual objects, andwhat problems are involved in performing the tasks.

6.1. Outline of Experiment
We asked six university students of subjects to perform basic
manipulations on virtual objects as shown in Figures 3–6.

Figure 3 | Generate.

6.2. Evaluation Method
After having pretested this system, each item on the survey wasanchored by responses of “good point”, “difficult point in manipulation”, and “improvement point” in this system.

6.3. Experiment Result
The results using data from the survey questionnaire is asfollows.
Figure 4 | Deform.

As for “good point”, there was some opinions such as “manipulation
method is intuitive and easy to understand”, and “movements of thehands were reflected in the system”.
In response to “difficult points in manipulation”, some respondents
indicated that “it is difficult to join two virtual objects”, “it is difficult to grasp the sense of distance with the virtual object”, and“when the hand is manipulated at a low position from the desk, thevirtual object does not react in user’s gesture”.

Figure 5 | Move.

As for “improvement point”, some respondents indicated “I want
feedback when crushing virtual objects”, “I want functions that canbe transformed into arbitrary figures, such as cones and rectangular parallelepipeds by gesture”, and “I want a manipulation function
to cut off virtual objects”.

268

M. Sakamoto et al. / Journal of Robotics, Networking and Artificial Life 6(4) 265–269

6.4. Discussion
From the “good point” perspective, the study confirms that theinteraction manipulation with the virtual object incorporating thegesture is intuitive. Therefore, we believe that it is possible to superimpose the physical phenomenon, such as deformation object byhand in the real space on the virtual objects.
Next, we considered the “difficult point in manipulation”. A possible reason as to why this response was obtained is that “it is difficult
to grasp the sense of distance with the virtual object.” What’s more,when the user manipulated the virtual object, an error occurred due
to a delay in the position information of the finger acquired with theLeap Motion Controller, and the user felt a sense of incompatibility.
It is possible the subjects responded that “it is difficult to join
two virtual objects” in this system because it was not possible tocouple virtual objects. Indeed, when an attempt is made to join thetwo virtual objects as they approach a certain distance, “it becomesdifficult to grasp the sense of distance with the virtual object”.
Specifically, an error occurs between the position of the finger on
the real space and the position of the finger on the virtual space.
It is possible that the reason why respondents reported that “whenthe hand is manipulated at a low position from the desk, the virtualobject does not react in user’s gesture” is that when the hand is ina low position, the hand covers the Leap Motion Controller, andthe coordinates of the fingertips cannot be acquired we considered.

using a survey questionnaire. of the data from the experiments
show support that this system can seamlessly perform a seriesof operations, such as generating virtual objects, transforming,moving, and joining them, and intuitively interact with virtualobjects.
Future research investigations should address the issues of “difficultpoint in manipulation” and add functions pointed out in “improvement point,” so that it can more intuitively interact with virtualobjects. Furthermore, we think that it is necessary to increase theprecision of interaction, such as enabling finer deformation ofvirtual objects.

CONFLICTS OF INTEREST
The authors declare they have no conflicts interest.

REFERENCES
[1]
[2]
[3]

7. CONCLUSION
In this study, we aimed more intuitive manipulation between users
and virtual objects using gestures by acquiring 3D coordinates of
the user’s fingers with the Leap Motion Controller and introducing
flexible body virtual objects using Bullet Physics.
In this exploratory experiment, we asked six subjects to performthe basic manipulation on the virtual objects and gathered data

[4]
[5]

K. Lee, Augmented reality in education and training, TechTrends.
56 (2012), 13–21.
J. Carmigniani, B. Furht, M. Anisetti, P. Ceravolo, E. Damiani,
M. Ivkovic, Augmented reality technologies, systems and applications, Multimedia Tools Appl. 51 (2011), 341–377.
A. Bellarbi, C. Domingues, S. Otmane, S. Benbelkacem,
A. Dinis, Underwater augmented reality game using theDOLPHYN, Proceedings of the 18th ACM Symposium on
Virtual Reality Software and Technology, ACM: Toronto,
Canada, 2012, pp. 187–188.
S. Kato, S. Ishigaki, S. Suzuki, M. Sato, A study on presentationof softness of virtual object in gripping motion with bare, J. Inst.
Image Inform. Tele. Eng. 71 (2017), J121–J124.
H. Suzuki, S. Suzuki, M. Sato, Visual improvement of AR system
focusing on grasping motion with bare hands, J. Inst. Image
Inform. Tele. Eng. 68 (2014), J299–J301.

AUTHORS INTRODUCTION
Dr. Makoto Sakamoto
He received the PhD degree in computerscience and systems engineering from
Yamaguchi University. He is presently
an associate professor in the Faculty ofEngineering, University of Miyazaki. He isa theoretical computer scientist, and hiscurrent main research interests are automata theory, languages and computation. Heis also interested in digital geometry, digital image processing,computer vision, computer graphics, virtual reality, augmentedreality, entertainment computing, complex systems and so on.

Mr. Takahiro Ishizu
He is a master student at Departmentof Computer Science and System
Engineering, University of Miyazaki. His
current research interests are computergraphics and image processing.

M. Sakamoto et al. / Journal of Robotics, Networking and Artificial Life 6(4) 265–269

Mr. Masamichi Hori
He received the master of engineering
at Department of Computer Scienceand System Engineering, Universityof Miyazaki. He is working for NEC
Corporation.

Dr. Satoshi Ikeda
He received PhD degree from Hiroshima
University. He is an associate professor inthe Faculty of Engineering, University ofMiyazaki. His research interest includes
graph theory, probabilistic algorithm,
fractal geometry and measure theory.

Prof. Amane Takei
He is working as an associate professor
for Department of Electrical and systems
Engineering, University of Miyazaki, Japan.
His research interest includes high performance computing for computational electromagnetism, iterative methods for thesolution of sparse linear systems, domain
decomposition methods for large-scaleproblems. He is a member of IEEE, anexpert advisor of The Institute of Electronics, Information andCommunication Engineers (IEICE), a delegate of the Kyushu
branch of Institute of Electrical Engineers of Japan (IEEJ), adirector of Japan Society for Simulation Technology (JSST).

269

Dr. Takao Ito
He is a Professor of Management ofTechnology (MOT) in Graduate Schoolof Engineering at Hiroshima University.
He is serving concurrently as Professor ofHarbin Institute of Technology (Weihai)
China. He has published numerous papers
in refereed journals and proceedings,
particularly in the area of management
science, and computer science. He has published more than eight
academic books including a book on Network Organizations
and Information (Japanese Edition). His current research interests include automata theory, artificial intelligence, systems control, quantitative analysis of inter-firm relationships using graphtheory, and engineering approach of organizational structuresusing complex systems theory.