A FINGER MOTION TRACKING DISPLAY SYSTEMFOR JAPANESE SIGN LANGUAGE DEMONSTRATIONS
Makoto J. Hirayama† Akira Asada† Ritsuko Kikusawa†††††
Osaka Institute of Technology
National Museum of Ethnology
ABSTRACT
A finger motion tracking display system is developed
for Japanese sign language demonstrations. The system
uses a finger tracking device by infrared stereo cameras(LEAP Motion Controller). This device and its software
detect finger shapes and positions in real time. In acurrent demonstration system, animation pictures aredisplayed depending on user's finger gestures asJapanese sign language, although the number of words
is few. The system can also be used for a user interface
of interactive museum exhibitions without touching.

In some other museums, non-contact user interfaces areused. Use of non-contact user interface devices issuitable in the current situations. For example, in one
exhibition at National Ethnology Museum, a page
flipping user interface without touching fingers[2] wasused as shown in Fig. 2.

Keywords: digital museum, motion sensor device,finger tracking, sign language, museum exhibition
1. INTRODUCTION
During COVID-19 Crisis, touch panel screens, pushbuttons, or any kinds of physically touching userinterfaces have been avoided for prevention ofinfections.
In most museums, touch screen display contents are outof order. In some museums, disposable gloves are used
to touch a display screen for interactive contents, suchas in Fig. 1. The figure is compiled for an explanation
purpose, from pictures at National Crafts Museum inKanazawa[1]. This content can rotate the pottery used
for tea ceremony in any angles on screen by dragging itby fingers.

Fig. 2: A page flipping user interface without
touching[2].
Although there are some candidates to realize simple
non-contact switches, more complex inputs are
sometimes required for some interactive applications.
To realize a flexible user interface for display screenbased contents, a finger motion tracking display system
is developed, mainly for Japanese sign language
demonstration contents in a museum exhibition.
Fig. 1: Disposable gloves to touch display screen forinteractive contents.

2. PROPOSED METHOD
As a user interface of screen-based display contents, useof a finger motion tracking device is proposed formuseum exhibitions, to enable variety of complex
inputs by gestures and sign language behavior. Users
can input gesture like commands by moving fingers infront of display screen without touching any physicaldevices. Furthermore, if finger shape and motion
recognition could be done with high accuracy in future,sign language inputs can be done to increase
accessibility to native signers.
As a finger motion tracking device, LEAP Motion
Controller[3] is used for the current implementation of
the system. LEAP Motion Controller is a commercial
product released by Ultraleap Inc. Two infrared
cameras are placed inside the device to detect fingers. It
is connected to PC via USB. It can be used fromprogramming environment such as C# in Visual Studio.
Supplied SDK of LEAP Motion Controller is
specialized to recognize fingers shapes so that
programmers can add hand gesture like user input
interfaces for original applications easily. Detecting
area coverage is 140 degrees horizontal and 120 degrees
vertical and 10cm to 60cm far from the device. In thesedetecting areas, fingers shapes and positions of left andright hands can be detected. Fig. 3 shows the LEAP
Motion Controller device.

3.2. Recognitions of Hands and Fingers
A target application of the current system is a museum
exhibition that explains Japanese sign language towhom does not know sign language. Examples ofJapanese syntax are shown as interactive contents like agame. Thus, some simple finger shapes of Japanese sign
language are recognized.
Frame class of SDK holds a current frame captured by
two infrared cameras and recognition results of hands
and fingers configurations. Fig. 4 shows objects withina frame used in the current application.
Count is the number of hands in a current frame. Palmdirection is also stored. In Fingers, information of 5fingers is stored. Bending or stretching of each finger isstored in IsExtended.

FrameHands

CountIsLeft
IsRight
PalmPosition
PalmNormalFingers

IsExtended

Fig. 4: Leap Motion Controller Objects used in thecurrent application.
4. CONTENTS OF SIGN LANGUGUAGE
DEMONSTRATION

Fig. 3: LEAP Motion Controller.
3. IMPLEMENTATION
3.1. Programming Environment
Implementations of prototype systems are done asapplication programs of Windows 10 on a notebook PC.
Unity[4] is used as a development platform of interactive
graphics contents. The version of Unity is 2019 (64bit).
Leap Development Kit 4.1.0+52211_win andUnityModules_LM_4.6.0 was used to use LEAP
Motion Controller from Unity. C# is used asprogramming language.

4.1. Demonstration System and Contents
A prototype of museum exhibition that explains
Japanese sign language to whom does not know sign
language is developed. The main purpose is todemonstrate that sign language is language which haveindividual syntax different from spoken language.
The system consists of a display screen, a finger motion
tracking device (LEAP Motion Controller), and apersonal computer. A notebook PC version is shown in
Fig. 5. The device is placed vertically on upper edge ofthe screen. In real exhibitions, a large display ishanging on a wall vertically, and the device is put on thewall below the display, facing front of a person whowatch the contents. Or, it may be placed on a table infront of the large display, facing up. In this case,person's hands are tracked from the bottom.
When the contents is started, an initial screen isdisplayed, and it is waiting for user. When a user comeand he/she moves his/her hands in front of the display, it
recognizes them as sign language words, then acharacter animation is displayed on screen depending on
hands and fingers shapes.

Fig. 7: "from/to" by an index finger movement.
Fig. 5: LEAP Motion Controller device placed on anotebook computer display.
In the current prototype, only 4 words are recognized,
which are "boy," "girl," "like," and "to/from." Fig. 6
shows the example Japanese sign language word
sequence of /boy/, /girl/, /to/, /like/, which means "Aboy likes a girl." "Boy" and "girl" are noun, "like" isverb, and "to/from" is a word which express a direction
by moving an index finger from an object to another(Fig. 7). Recognized sentences are either "A boy likes agirl." or "A girl likes a boy."
When a user does such movement in Fig. 6 in front of
the screen, the system displays characters of a boy and agirl (Fig. 8), and a heart mark is flying from the boy to
the girl. Fig. 9 shows an example of character animation.
This is a very limited and simple sign language
recognition system used only for an introduction
purpose, although it cannot be used in real situations.
However, even if it is very limited, contents with userinteractions are more impressive than one-way
explanations.

Fig. 8: Character animation by hands and fingers
recognition.
4.2. Feasibility Evaluations
With 10 students, recognition rates of 3 sign language
words, "boy, " "girl," and a direction of "from/to" weretested. "Boy" (stretching thumb and bending other 4fingers) were recognized at accuracy of 88%, "girl"(stretching little finger and bending other fingers andthumb) 94%, "from/to" motion 72%. Flipping motionsof "from/to" were sometimes framed out and overlappedby other fingers and palm, then recognition rate is low.
In verb "like," thumb and index finger were often
invisible by chin and framed out often so that therecognition was very low less than 50%, thus
improvements and position changes were needed.
In conclusion, the prototype is not feasible as realexhibition contents, so far.

Fig. 6: A sign language sentence, "A boy likes a girl."

1m maximum, and 170×170° typical field of view.
Currently, it is evaluated and tested in our lab.

BoyGirl
From/ToFig. 9: Recognition experiment results.
4.3. Future improvements
Currently, several enhancements are planned towardfeasible exhibition contents.
At first, placements of the motion tracking device will
be reconsidered. In the evaluation experiment, hands
were often out of range when a user moved his/herhands for sign language. Also, the word "like" could
not be recognized by hiding hands by chin. The
tracking area of the device is depth between 10cm to60cm preferred, up to 80cm maximum, and 140×120°
typical field of view (Fig. 10). The best position of thetracking device will be found by repetitive trials.
In our initial developments in several years ago, weused Xbox Kinect and Kinect v2 (Microsoft)[5] as body
motion capture devices. By Kinect, wider areas could
be captured but details of hands and fingers could not becaptured. Kinect was obsoleted by the maker in few
years ago. Thus, we switched the device to LEAP
Motion Controller, for detailed finger tracking.
Currently, Kinect is reborn as Azure Kinect DK[6]forenterprise market. We may think about Azure Kinect
DK or other depth cameras, in future, depending on
application contents.

Fig. 10: Recognition zone of hand gestuers (LEAP
Motion Controller).
From Ultraleap Inc., another device similar to LEAP
Motion Controller is released as "Stereo IR 170."[7] (Fig.
11). It has wider field of view. The tracking area of thedevice is depth between 10cm to 75cm preferred, up toFig. 11: Stereo IR 170.
Second, contents will be improved. More variations andattractive features will be added.
Also, stable,maintenance free, completed standalone software witheasy user interfaces will be made for museum exhibition
purpose.
5. SUMMARY
A finger motion tracking display system is proposed,
and a prototype was implemented. A prototype
demonstration content for Japanese sign language
experiences was implemented and tested. So far, theaccuracy is low and infeasible to real exhibition, butcurrently and in near future, several enhancements arebeing applied. User input interfaces without touching
are necessary and important for interactive museum
exhibitions, in the current COVID-19 situations.
6. REFERENCES[1] National Crafts Museum, The First of the National Crafts
Museum's Grand Opening Exhibitions: Japanese Crafts －Materials, Techniques and Regionalities, Kanazawa, Ishikawa,Japan, 2020.
[2] Takuji Suzuki, Non-contact page flipping device"Sawarazumekuri" - development and application on museum
exhibition, SIG on Digital Museum and Humanities, 1, pp. 3439, 2021 (In Japanese).
[3] Ultraleap Inc., Leap Motion, http://ultraleap.com, 2021.
[4] Unity, Unity, https://unity.com/developer-tools, 2021.
[5] Microsoft, Kinect for Windows SDK 2.0,
https://developer.microsoft.com/en-us/windows/kinect/, 2021.
[6]
Microsoft,
Azure
KinectDK,
https://azure.microsoft.com/en-us/services/kinect-dk, 2021.
[7] Ultraleap Inc., Leap Motion - Stereo IR 170 Evaluation Kit,
http://ultraleap.com, 2021.