tra v 3 D Signing Avatar for Sinhala Sign

Animating signs of a sign language requires on demand bone movement of a signing avatar based on single posture, multi posture signs for the words with a known sign, and fingerprinting signs for representing unknown words character by character. Ordering and sequencing of different types of signs corresponding to words/phrases of a typical spoken/written language are the most essential features of a 3D signing avatar. Typical techniques of signing avatar animation are video sequencing of signs of corresponding words of a sentence, or replaying movements of signs with motion-captured animation sequences. This paper presents a multi-facet 3D avatar and an animation framework that supports the definition and animation of sign gestures without motion capture hardware. The system is developed to accommodate any sign language, but a prototype has been built for the Sinhala sign language (SSL). The avatar system is initially built with 10 bones per arm. It is then extended to 29 bones per arm to improve flexibility in palm and fingers. The avatar animation framework is capable of signing 200 plus gestures of both single and multi posture SSL signs and 40 fingerprinting SSL signs. Speed and uniformity of the sign gesture animation is achieved by the automatic calculation of intermediate sequences of arm movements of signs within a given number of frames, or with a user defined increment value per bone. Animation system also supports the definition of fingerprinting signs, with a user defined tag set corresponding to the alphabet of any given sign language. Keywords— 3D Signing avatar, Animation framework, Sinhala Sign Language. I. In t r o d u c t io n Express one's ideas to an aurally handicapped person requiring services of a trained sign interpreter who is a scarce resource in any country. Computer scientists throughout the world have tried to solve this problem using animated human models on demand, to perform a sign language gesture animation. Instead of moving a 3D hand based on keyboard inputs, it is essential to initiate hand movement using an ordered set of commands when a sign language gesture is animated. There are two types of sign gestures in a sign language: static and varying gestures. For example, a word such as “you” (3)9) shows the right hand index finger pointing towards the observer in a static posture in Sinhala sign language (SSL). Therefore, irrespective of the initial position of the arm, the final position of the index finger is taken as the gesture. The latter considers the movement of the arm from one posture to another to represent a word in laymen language. For example, representing the word “ugly” (as^za) in Sinhala sign language requires moving the right hand little finger clockwise around the face while the other fingers are flexed [2], One gesture per Sinhala word is more common in SSL, whereas a single gesture exists for entire phrase, such as "I love you" (®® S )90 ep ^o d S ) and "how are you" (® 8® ®2s»®833®^). In a majority o f techniques available to animate a sign language’s gestures, a human model that moves the fingers in one hand or both hands simultaneously is required. This process is also known as manual signing. Moreover, Phonetic symbols that contain complex torso movement and fecial expressions combined with mouth movements are referred to as non-manual signing or fingerprinting. Graphical modelling of a human avatar requires a lot of time, creativity and specialist skills. Therefore good graphical human models, mostly commercial in nature, are usually built for computer games and the film industry. iCommunicator [1] for American Sign Language (ASL), which is a commercial signing application, utilizes such commercial graphical models coupled with expensive motion capture hardware for making video sequences to model human gestures using complex graphic cards. II. Re l a t e d W o r k A sequencing of motion-captured British sign language (BSL) words based on English sentences is the technique presented by Pezeshkpour et al. [3] to animate BSL. Gestures animated in a computer terminal using the TESSA framework supports conveying most common messages to a deaf person in British post offices with BSL. Vocabulary of the TESSA system is limited only to common words/phrases used in British post offices [4]. Adding new gestures to both these systems involves an expensive motion capture process. To reduce the motion capture process for each and every sign, the Signing Gesture Markup Language (SiGML) notation is used as presented by Elliott et al. [6], based on the HamNoSys version 2 by Prillwitz et al. [7] under the ViSiCAST project [5]. The SiGML notation supports defining signs of BSL, the Dutch and the German Sign languages. SiGML notation represents a sign gesture as a collection of movements of different parts of a human body in an XML-like notation. SiGML has a core set of commonly used hand postures while other hand poses are defined by the transition of hand location from these core sets of hand postures. Transitions can take the form of a straight line, rotation or zigzag motion [9]. A virtual avatar, vGuido, is developed with the contributions made by multiple institutions in the European Union under the e-SIGN project [8] to perform a synthesis o f signs in SiGML notation from the content of government web sites and weather forecasts in Europe. Defining a sign in SiGML notation is a complex task even with a supporting editor because it requires a complete and comprehensive knowledge of existing core gestures. Kaneko et al. has developed a technique of word-to-word translation of Japanese text to Japanese Sign Language (JSL) using TV program Making Language (TVML) [11]. TVML is a text-based markup language that generates graphically animated TV programs by simply writing a script in TVML 24“' -26th August 2015 International Conference on Advances in ICT for Emerging Regions ICTer2015 3D Signing Avatar for Sinhala Sign Language 2/ and played in a TVML player. TVML supports embedding motion data of bio vision hierarchy (BVH) format [12] into key frame positions and supports moving the scene camera to point a specific virtual character with different close up positions. Researchers have also developed a Microsoft DirectX based new TVML player to display sign gesture animations of the JSL [10]. This technique also requires expensive motion capture data of sign gestures. M. Huenerfauth presented a classifier predicate based decoding technique to represent a sign gesture in ASL on different body/hand channels for the animation, and later improved it to embed motion captured facial expression data to signing virtual avatars[13], R. Wolfe et al. developed a Finger spelling tutor for teaching finger spelling to school children [14]. Paula Signing tutor was developed by M. J. Davidson to train ASL for hearing adults [15]. Both software systems utilize 3D signing avatars to train ASL. This paper presents an improved 3D signing avatar and an animation framework which has the following characteristics: • Supports the animation of words, phrases and finger printed signs based on posture coordinates. • Provides support for defining sign gestures as a group of combined hand/body poses • Allows for the movement of avatar hands to a final position irrespective of initial position • Manages movement of different skeleton components by means of an auto calculated or user-defined increment • Provides the ability to add new gestures to the system without altering code. The asynchronous nature of the avatar animation framework supports the accommodation of new sentences to the animation playlist while an animation is being played back. Seamless switchover to the character-to-character finger printing technique using a specific sign alphabet is supported when an unknown word is found in a sentence for which an animation does not exist in the database of known gestures. III. M e t h o d o l o g y A. Human Avatar for Sign Animations An animation avatar requires the modelling of a human figure in a graphical environment as a mesh and covering it with texture. A bone rig is then attached to the avatar with different animation channels attached to the bone rig. The Makehuman software, which provides facilities to build a human avatar based on gender, age and skin colour (African, Caucasian and Asian), is used in the present research. Makehuman does not support face rigs, hence the model is exported in Makehuman exchange (MHX) format without a skeleton attached to the mesh. The human model generated by Makehuman is imported to Blender animation software and the automatic rigging facility is then invoked. The camera has to be positioned facing the human model, and proper lighting needs to be enforced on the model to minimize shadows and to display proper arm movements for all the signs. In this study, three light sources and one scene camera is focused on the human model, displaying the upper body of the avatar in the view port. Change of viewport resolution and camera zoom facility is facilitated through the avatar animation framework. B. Animation Engine and Scripting Environment The dynamic nature of the sign language animations require the real time rendering of each display frame. Therefore, the blender game engine (BGE) is selected as the animation engine. The BGE works in sensor, controller and actuator modes. When a defined sensor is triggered, the BGE initiates action/activity through an actuator. The “Always Sensor” in BGE, which runs in the idle frequency of 60Hz with a python controller, is used for the present research. C. Human Avatar Rig Structure for the Sign Animation To animate a given sign gesture, each bone/bone group of the model has to be animated independently. The arm is broken down into sub components named clavicle, deltoid, upper arm, fore arm, hand (palm) and five fingers as depicted in Fig. 1. Fingers have multiple bones of which the pose coordinates are measured, calcul