Implementation of 3D Drawing to Interactive Display System
National Institute of Technology, Nara College, Department of Electrical Engineering
Taiki Kato Shigeki Doi

Introduction

Results

In recent years, the application of new information technologies suchas AR (augmented reality), gamification and image processing in theamusement field is progressing. Our laboratory has develop "lightcrayon system" based on the image processing technology .

●This output uses HoloLens provided by Microsoft. HoloLens isgenerally referred to as AR glasses, and it is possible to arrange 3D
images superimposed in the space of everyday life. The result of theimages operator will see and operator’s appearance are shown below.

Constitution●Actually field test was carried out and evaluation aboutentertainment value was obtained. In Science event the Tenri ScienceExperience Festival, a questionnaire was conducted from 21 players.
The item “after” in the questionnaire is the result of implementing
the 3D drawing function with AR glasses for the first time. Comparedwith traditional 3Dlight crayons without AR glasses, dominant results
was obtained in terms of “Creative mind is inspired”. And, the result
was not good in term of “Interested in 3D light crayon”.
.

This system mainly consists of light pointer, PC, camera, projector, andscreen. The light pointer is a device using as a brush. By holding thislight pointer in front of the screen, you can draw lines freely. Thissystem is not merely tracking the trajectory of the optical pointer andt h e drawing with the projector.
.

Implementation of 3D drawing to interactive
display system
And also, VR(Virtual Reality),AR(Augmented Reality) andMR(Mixed Reality) technologies
have attracted attention, and manysystems have been developed. We
tried to develop a system that canbe drawn around us or in the air
without any restrictions, using MR
technology for the addition of newentertainment features. This systemi s ca lle d "3D light c rayon" .

Improve

drop

Preliminary experiment
The prototype 3D light crayon inpreliminary experiment captures the3D data of the orbit drawn in spaceby the light crayon to PC using thedepth sensor Intel RealSense andvisualize the drawn orbit with Rviz ofthe 3D display tool from theROS(Robot Operating System).

Development of the 3D light crayon
We tried to make the 3D light crayon system using Microsoft
HoloLens which is one of AR glasses as 3D output device. The system
consists of three functional steps.
①The trajectory of the light crayonis complemented in 3D. Therefore,a se nsor that m easures thedistance of the light crayon pointeris required.
.
②On the unity game development
p l at fo r m p ro v i d e d b y U n i t yTechnologies, it generates 3Dgraphics on the PC with thecaptured trajectory.
.
③This 3D image is output to theparticipant as a light crayon
trajectory in three dimensions.

Conclusions●In this study, the depth sensor and the AR glasses were used toconstruct a system that allows operators to draw pictures aroundthemselves. However, some problems were found doing actual
construction and operation. With Microsoft HoloLens, an AR glasses
used in this study, it is only possible to share the situation with theperson who is wearing the glasses and drawing the picture.
●With future development, we are considering a system that draws
pictures with the figure of hands using AI image recognitionte c h n o l o g y. T h e f l ow o f t h i s syste m i s s h ow n b e l ow.
①The camera shoot the user’s hand and to send them to the PC.
②It is to recognize the image by using AI to determine the shape andcoordinates of the user’s hand.
.
③ The user is able to draw a picture by connecting the coordinates of
the hands’ movement. Moreover the hands’ gesture will change thecolor of the lines shown on the screen.
.