SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 1(2); September (2020), pp. 86–90
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Interactive Projection Mapping using Human Detection byMachine Learning
Makoto Sakamoto1, Takahiro Shinoda1, Kenji Sakoma1, Takahiro Ishizu1, Amane Takei1, Takao Ito2
Department of Computer Science and System Engineering, University of Miyazaki, Miyazaki-City, 889-2192, Japan
Graduate School of Engineering, Hiroshima University, Higashi-Hiroshima, 739-8527, Japan1
2ARTICLE INFO
Article History
Received 25 October 2019
Accepted 03 July 2020
Keywords
Entertainment computing
ECProjection mapping
Kinect

ABSTRACTIn recent years, “Entertainment Computing” (EC) has attracted attention and has become one ofthe major industries in Japan. “Projection Mapping” is well known in this EC. Projection mapping
is a video technology that creates a new space by synthesizing space and video using a projector.
Among them, many people are fascinated by works that create a fantastic world by combining
dancer performance and projection mapping. However, these works require the performer toaccurately align with the coordinates of the image objects in the projection mapping, which is noteasy for everyone. In this study, we aim to entertain not only the people who see the projection
mapping but also performers. Therefore, we have prototyped an interactive projection mappingthat changes according to user movement. This time, we focused on sports and projected the ballto the user to experience baseball pitching and soccer lifting. Furthermore, we conducted aquestionnaire survey to evaluate the sense of use of this system, and the results showed that many
people can enjoy by this projection mapping.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

1. Introduction

In this study, we propose a participatory projection
mapping that changes according to the movement ofparticipants by projecting to participants.

In recent years, exciting projection mapping hasreceived more and more attention. Many people have seen
projection mapping to buildings such as theme parks.
For example, there is a closing ceremony for the “RioOlympics” held in August 2016.
Currently, projections are also being made on familiar
items such as clothes, faces, and notebooks. For example,during a performance at the world's largest advertising
festival in Cannes, the artist's white costume became ascreen, and colorful graphics were projected one after
another.
Projection mapping will delight us in many ways.
However, in the conventional projection mapping, theviewer mainly enjoys watching and enjoying the projected
image, so that the viewer's feeling of immersion in thecontent is considered insufficient [1].

2. Study method
2.1. Equipment Used
2.1.1. Kinect for Windows
Kinect for Windows is available from Microsoft. It’s aperipheral device that enables operations by body
movement, gestures, sound, etc. without using a controller(see Fig.1).
Kinect has an infrared sensor, an RGB camera that
acquires 8-bit 3-channel (RGB) image data, a depth image
sensor that acquires distance (depth) image data fromKinect, and a sound microphone that estimates the positionof a sound source.

Corresponding author’s E-mail: sakamoto@cs.miyazaki-u.ac.jp, itotakao@hiroshima-u.ac.jp

86Fig. 2: “Ball” screen display.

Also, Kinect’s feature is "posture recognition
technology", recognizes the whole human body andperforms operations by its movement. Thereby, it ispossible to estimate where the parts of the body are byusing the depth image.
Infrared sensor

RGB camera

Fig. 3: “Color” screen

Depth image sensor

display.

Microphone

Fig. 1: Kinect for Windows v1[2].
2.1.2. Projector
The projector is used for projection.
Fig. 4: “Depth” screen display.

2.2. Development Environment
・ OS: Windows10,
・ Development Environment: Visual Studio 2017,
・ Development Language: C++,・ OpenNI2,OpenNI is an interface that allows you to configure
Kinect, acquire images, measure distance, and use
a microphone.
It is also used in this study because it can be linked
with OpenCV, an image processing library.
・ NiTE,NiTE is used to get data (eg gesture recognition,motion tracking etc) from Kinect [2].
・ OpenCV,
Create an object detector using the OpenCV library.
・ OpenGL.

g. 6: “Skeleton” screen

display.

3. Outline of the system
It was programmed to display seven screens: Ball, Color,Depth, User, Skeleton, Combination (for projection), andCombination PC (for PC) (see Fig. 2, 3, 4, 5, 6, 7).

87

Fig. 5: “User” screen
display.

Fig.7: “Combination”“Combination PC”
screen display.

3.3. Soccer mode

3.1. User Detection

The first screen is in the black initial state. When the user
raises his knee, a soccer ball and a soccer stadium areprojected. In addition, a sound is now played when kicking
the ball. If the ball coordinate is lower than the ycoordinate threshold, the ball disappears and is initialized.
In addition, physics was used for the lifting operation (seeFig.9).

3.1.1. Boosting
Boosting is a learning algorithm that sequentiallygenerates weak classifiers and combines them to create astrong classifier [3]. In this study, we decided to use amethod called AdaBoost among various Boosting methods.
AdaBoost is a technique for creating a classifier with highaccuracy by learning by adaptively weighting therecognition rate of the classifier during the learning
process [4] (see Fig.8).

3.4.

Baseball mode

The first screen is in the black initial state. When the user

Fig. 8: AdaBoost.
3.2. Extract image features

Fig. 9: Soccer mode execution result.
raises his hand, a baseball ball and a baseball stadium areprojected. In addition, a sound is now played whenthrowing the ball. In order to make the ball appear to be
thrown in the back, the ball gradually gets smaller as theflight distance increases. When the ball position is greaterthan the threshold value of the x coordinate (the left edgeof the screen as seen by the user), the ball disappears andis initialized (see Fig.10).

The classifier learns by extracting the feature amount of
the image at the time of creation. There are the following
three feature extraction methods.
・ Haar-Like,・ Local Binary Pattern (LBP),・ Histogram of Oriented Gradients (HOG).
In this study, we conducted experiments using LBP.
3.2.1. LBP
LBP is one of feature quantities that can be used forimage recognition and classification [5]. It is usuallycalculated in a 3x3 pixel area and extracts local features. It
is particularly resistant to lighting changes and has theadvantage of being able to calculate at high speed. In thecalculation method, first, the luminance value of the centeris compared with the luminance values of pixels in thevicinity of the surrounding eight. Of the 8 neighborhoods,
1 is set when the luminance value is equal to or greaterthan the central luminance value, and 0 is set otherwise.
This is multiplied by a mask, the sum is obtained, and thisvalue is replaced with the luminance value of the central
pixel. A mask is a weight assigned 2n clockwise from thetop left. This operation is performed for all pixels, and theresulting image is called an LBP image. The object isrecognized using the LBP feature value thus obtained.

Fig. 10: Baseball mode execution result.
4. Experimental result
The experiment using the threshold of skeleton
coordinates and the method of this study were tested andcompared [6]. In this study, we performed grayscale on the“Color” screen display obtained from Kinect and detected88the user using object detector. Fig. 11 shows human bodyrecognition in soccer mode.

image processing with OpenCV (in Japanese)”, Kodansha
Ltd., ISBN978-4061538290, 2017.
Nobuyuki Nishiuchi, Takuma Suzuki, Kimihiro
Yamanaka "Development of Work Posture Evaluation
System Using Boosting" J Jpn Ind Manage Assoc 62,51-58,2011.
Yoav Freund, Robert E Schapire "A DecisionTheoretic Generalization of On-Line Learningand an
Application to Boosting" journal of computer and systemsciences 55, 119139, 1997.
Ojala T, Pietikäinen M & Harwood D, “Performance
evaluation of texture measures with classification based on
Kullback discrimination of distributions.”, Proc. 12thInternational Conference on Pattern Recognition (ICPR
1994), Jerusalem, Israel. Vol I, 582-585, 1994.
Makoto Sakamoto, Takahiro Shinoda, Takahiro Ishizu,
Masahiro Hori, Hazuki Watanabe, Amane Takei, Takao Ito"A Proposal of Interactive Projection Mapping Using
Kinect" IEEE Xplore: 29, November, 2018.

3.

4.

5.

Fig. 11: User detection using human detector in soccer
mode.

6.

In previous study, when the subject disappeared from theKinect field of view and entered the field of view again,
there was a problem that the skeletal information was notretraced and mapping was not successful. In this study, we
solved the problem by performing human bodyrecognition using OpenCV without using tracking ofskeleton coordinates.

============================================

Authors Introduction

Dr. Makoto Sakamoto
He received the Ph.D. degree incomputer science and systems
engineering
fromYamaguchi
University. He is presently an
associate professor in the Faculty ofEngineering,
UniversityofMiyazaki. He is a theoreticalcomputer scientist, and his current
main research interests are automata theory, languagesand computation. He is also interested in digitalgeometry, digital image processing, computer vision,
computer graphics, virtual reality, augmented reality,
entertainment computing, complex systems and so on.

5. Conclusion
In this study, we used an object detector created with
Opencv to create an interactive projection mapping whosemapping changes according to human movement. Inprevious study, when the subject disappeared from theKinect field of view and entered the field of view again,
there was a problem that the skeletal information was not
retraced and mapping was not successful. In this study, wewere able to solve these problems by using a method that
does not use arbitrary thresholds. For example, in previous
artist concerts, artists have been following the movementof projection mapping. However, with this method, it ispossible to perform projection mapping that matches themovement of the artist. This will reduce the difference
between each other's movements and make it possible toproduce more realistic performances. In the future, wewould like to realize interactive projection mapping using
more accurate human body classifiers by increasing thenumber of sample images.

.

.

References
1.

2.

W.Ogasa, H.Katayose, "TPPM (Take Part in
Projection Mapping): Multiplayer participation typeprojection mapping application using tablet terminal",
Entertainment Computing 2014, 2014-09-12, 77-79, 2014.
M.Koeda, E.Ueda, T.Nakamura, “Introduction to89

Mr. Takahiro Shinoda
He is a master student at
Department of Computer ScienceandSystemEngineering,
University of Miyazaki. His current
research interests are computer
graphics and image processing.

. Mr. Kenji Sakoma
He is a master student at
Department of Computer Scienceand
SystemEngineering,
University of Miyazaki. His current
research interests are computer
graphics and image processing.
Mr. Takahiro Ishizu
He is a master student at
Department of Computer ScienceandSystemEngineering,
University of Miyazaki. His current
research interests are computer
graphics and image processing.
4.
Prof. Amane Takei
He is working as Associate
Professor for Department ofElectrical and systems Engineering,
University of Miyazaki, Japan. His
research interest includes highperformancecomputingforcomputational electromagnetism,
iterative methods for the solution ofsparse linear systems, domain decomposition methods
for large-scale problems. Prof. Takei is a member ofIEEE, an expert advisor of The Institute of Electronics,
Information and Communication Engineers (IEICE), adelegate of the Kyushu branch of Institute of ElectricalEngineers of Japan (IEEJ), a director of Japan Society
for Simulation Technology (JSST).
Dr. Takao Ito

He is Professor of Management ofTechnology (MOT) in Graduate
School of Engineering at Hiroshima
University.
Heisserving
concurrently as Professor of Harbin
Institute of Technology (Weihai)
China. He has published numerous
papers in refereed journals and proceedings, particularlyin the area of management science, and computerscience. He has published more than eight academic
books including a book on Network Organizations andInformation (Japanese Edition). His current researchinterests include automata theory, artificial intelligence,systems control, quantitative analysis of inter-firm
relationships using graph theory, and engineering
approach of organizational structures using complex
systems theory.

============================================

90