WISS 2018

A Camera-based human contact detection & dynamic projection
systemErwin Wu∗

Mitski Piekenbrock∗ †Toshiki Sato∗

Hideki Koike∗

Abstract. In this research, we propose a novel human body contact detection and projection system.
We use Kinect V2 camera and generated human 3D model to detect the contact between user s bodies.
A special algorithm of dividing human mesh into small pieces of polygons to do collision detection isdeveloped and detected hit information will be dynamically projected according to its magnitude ofdamage. Our system can be used in various sports with body contacts and virtual martial training.

1Introduction and Related Works

In martial sports such as boxing or karate, it
is diﬃcult for the viewer to notice hit information received by the actual players. Therefore,
we proposed a contact detection system using
Kinect RGB-D camera which can visualize thehit information on user’s bodies. Our systemgenerates precise 3D body models representing
users and developed a novel algorithm to recognize the collisions. Furthermore, the magnitudeof a collision is displayed dynamically in diﬀerent colors. Our system is therefore suitable forsupporting martial sports by visualizing damage
information or make martial training more safer
by controlling the virtual 3D model remotely.
Augmented Studio[1] is a project dealing withhuman body projection. However, it can only beapplied to a person moving slowly and is not suitable for objects with fast movements as in sports.
There are some previous works that focus on
dynamic projection mapping like Ballumiere[2]and LightSpace[3]. Although Ballumiere is doing projection on dynamic sphere but not on
bodies, it proposed a novel projection system
for high-speed moving objects by applying dynamic Kalman Filter. The LightSpace is anotherproject we looked into that deals with a variety of interactions between surfaces with multiple depth camera. However, LightSpace is limited to emulating interactive display features onflat and static shapes that are designated beforehand, which means it is not suitable for quickly
changing surfaces such as human body. This reCopyright is held by the author(s).
Tokyo Institute of Technology†
RWTH Aachen University∗

Fig 1: Damages are projected to hit position in
diﬀerent colors according to their magnitudes.
search targets to detect contacts between moving people by visualizing the exact position anddamage level with color on the body.

2Implementation

We use one Kinect V2 Camera for capturing
user motion and generating 3D human modelfrom user contour. After skeleton fitting, the 3Dmodel becomes a virtual representation of specific user. Since it is diﬃcult to calculate collisions between all point clouds of user in real time
due to high computation, we divide the humanmesh into small pieces for collision detection. Atlast, we import RoomAlive to calibrate the projector using Kinect.
2.1 Human Model Generation
A perfect matching model of user would bedesirable, but such precision requires multiple
depth camera which lead to the cost of increased
delay. We designed a body matching function toobtain exact body size of user. The gray-scaledata from Kinect is used to represent the humanbody with great quality. Our algorithm aims to
get the contour of body from these data with
diﬀerent angles and calculate the parameters ofuser body, which is used to adjust the model.

WISS 2018

3Fig 2: 3D Model Generation

Damage Visualization Since the force of damage will be calculated and visualized, the system
can also project an illustration of damage heatmap of user’s body. Which also has the possibility to become a combat game with HP gauges
and make the sports more interesting and understandable for the audience.
Safe Martial Training To avoid injury, many
players will not fight seriously during martialtraining. We can develop a safe training systemwith adjustments to specific sports, for example,
by setting the 3D model nearer than reality to let
the user control them remote which means theusers are not really fighting but can recognize
the damage they receive.

4(a)(b)Fig 3: (a) Divided meshes. (b) Collision between
two hulls H1 and H2 .
2.2 Collision detection
Dividing human mesh: Detecting collision
between multiple dynamic humans precisely requires a high processing power to calculate eachpoint of human whether it is colliding or not.
This is, however, practically impossible with sucha precise human model and we therefore divide
our model into about 200 segments, which wecalled ‘hulls’, while each of these hulls contains acollider that matches its segment and is assigned
to corresponding bone so that they move and rotate with the joints accordingly. Figure 3a shows
how meshes are divided into hulls.
Detection algorithm: Our approach is thatwhen a collision happens, the approximate coordinate h′1 is calculated by first determining the→center position −p 1,2 of both hulls. From one of→→→the central positions, a ray −
r = −
p2 −−
p 1 iscasted towards the other central position. We−→
take the coordinate h1 ′ when the ray hits the→surface of the other hull which is between −p1−→−→−→−→−→′and p 2 by definition: h1 = p 2 + ( p 2 − p 1 ) · α,in which α ∈ [0, 1]. We can also get another projection point h′2 by swapping p1 and p2 . As one
can see in figure3b , the actual collision points−→
h 1,2 are slightly below the perspective projec−→tion points h ′1,2 . The diﬀerence distance δ1,2 canbe estimate according to the figure.

ApplicationConclusions & Futureworks

We proposed a novel 3D model-based humancontact detection method and its visualization
system. Currently, this project is still facingsome problems:
Latency of projection: Since high speed
dynamic projection mapping has being realized
nowadays, it is possible to improve the projection
accuracy with the help of high speed cameras.
Accuracy of hit position: Multiple Kinects
might solve the accuracy problem, however, Kinect
has a low resolution rate along with only 30 frame
rate and other fabric technologies might disturb
the sportsmen. We believe that deep learningbased pose estimation method could solve thisproblem, and simple high speed RGB camera
could work as motion capture in the future.

References[1] T. Hoang, M. Reinoso, Z. Joukhadar, F. Vetere, and D. Kelly. Augmented studio: projection
mapping on moving body for physiotherapy education. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems,pp. 1419–1430. ACM, 2017.
[2] S. Miyafuji, M. Sugasaki, and H. Koike. Ballumiere: Real-time tracking and spherical projection for high-speed moving balls. In Proceedings of the 2016 ACM on Interactive Surfaces andSpaces, pp. 33–37. ACM, 2016.
[3] A. D. Wilson and H. Benko. Combining multiple
depth cameras and projectors for interactions on,
above and between surfaces. In Proceedings ofthe 23nd annual ACM symposium on User interface software and technology, pp. 273–282. ACM,2010.