DESIGN OF OBJECT SPELLING RECOGNITION SOCIAL ASSISTIVE ROBOT FOR DEAF PEOPLE USING GESTURE RECOGNITION
By: Ega Guntara
Tags: Machine Learning, Convolutional Network

Scope of Problem

The spelling input still comes from the keyboard so that in every spelling

have to type in any letters that will be spelled by the user

have to type in any letters that will be spelled by the user Gesture Recognition is only performed on the wrist and finger scale

hands, can’t completely one body

hands, can’t completely one body The method used is for gesture recognition only

using the Convolutional Neural Network (CNN) method.

How The System Works

This system is activated via keyboard input entered by the user, so the robot can tell when he started his job, when the input is wrong then the robot will remain in an idle state until the keyboard input is correctly entered. When the system already active, the first step that must be taken is provides input from the keyboard in the form of alphabets of the object to be spelled, after input entered then the robot will change to move and the movement indicates the system is ready to process the spelling of the user, the spelling performed will be checked one by one. When there is a spelling error, the robot will immediately do it a gesture indicating the student’s misspelling and will retest with a maximum of three times, when If the error has passed the maximum limit, the robot will stay at idle condition with the aim of the teacher concerned can explain the material again regarding the introduction of the alphabet. If there is no mistake then the robot will do gesture indicating correct spelling is up to the object successfully spelled completely into a word,

The Data

the data that i use are photo of my hand while doing the alphabet movement from A-Z. Each alphabet had 1200 photo then i flip it, so eazh alphabet had 2400 photo.Image collected is converted to numpy arrays to make this suitable for tensor processing in training.

To make a good data set like the image above, it is recommended to use gloves that have a contrast color with background.

Training Process

All the classes has been trained using CNN method with the architecture as shown below:

after the dataset being processed with those architecture, the matrix is converted into a linear array so that to input it into the nodes of my neural network. After that those linear outputs are converted to number of output classes.

Next step is training the model:

after the training process is done , then the model will save the alphabet movements that we have made in the dataset.

here is my alphabet movement which will be used in the testing process.

Testing Process

the testing process is the process where we try the models we have created to recognize sign language movements if there are defects or inconsistencies in the recognition of sign language movements, there is something wrong when we do the training process.

Make the Spelling mode

spelling mode is created using the finite state machine method, therefore the spelling mode in this system can only perform spell check with variations in the same number of alphabets. On this occasion I made 3 variations on the spelling mode, that is variations with 4 alphabetical order, 5 alphabetical order and 6 alphabetical order. the finite state machine of 3 various mode :

To clarify the conditions START condition is a condition where the system has not been entered on spelling mode, IDLE condition is a condition when keyboard input has been pressed to enter spelling mode, but alphabet input has not entered and the robot condition is in a condition that represents that the robot is ready to check the spelling of the user. Next conditions T1 to T6

is a spelling condition in alphabetical order on the object, for example if the user give alphabetical input with object B-U-K-U, then T1 is the process the user must spell the letter B, while the user process T2 must spell the letter U and so on, whereas the finished process is the process by which the user spells successfully the whole alphabet that is inputted to the system and the condition of the robot will do clapping movement while saying “you, ve been finished your spell”.

Robot Prototype

the robot that i used is a humanoid robot by Robotis with type Robotis Bioloid Premium . The robot is built by integrating 3 systems, namely a computer system where the computer system on this robot uses the INTEL NUC mini pc, webcam, and the humanoid robot itself. The power source that used to turn on this system is power adapter, there are 2 power adapters used to power this system wich is 12 volt power adapter for the CM-530 to turn on the humanoid robot and 19 volt power adapter to power up the INTEL NUC. Communication between mini PC with CM-530 uses a USB to TTL cable, because of its communication used between the 2 components is serial communication. Communication between 2 This component is done with the aim that the mini pc can send commands movements on the CM-530 correspond to the workflow of the system being built.

Robot Movement

Robot movement in this system is done by sending commands movement in the form of hexa data written in python programming to CM-530. sending data is done serially and will be received by CM-530.Then will be the command to move the servo on the robot. Basically hexa data sent is replacement data from remote controller is the innate robot controller from Robotis, but its communication is originally radio in the system this time changed to serial with the aim that the robot can move automatically without having to be controlled by the remote.