###### Abstract  
This manuscript presents a full duplex communication system for the Deaf and Mute (D-M) based on Machine Learning (ML). These individuals, who generally communicate through sign language, are an integral part of our society, and their contribution is vital. They face communication difficulties mainly because others, who generally do not know sign language, are unable to communicate with them. The work presents a solution to this problem through a system enabling the non-deaf and mute (ND-M) to communicate with the D-M individuals without the need to learn sign language. The system is low-cost, reliable, easy to use, and based on a commercial-off-the-shelf (COTS) Leap Motion Device (LMD). The hand gesture data of D-M individuals is acquired using an LMD device and processed using a Convolutional Neural Network (CNN) algorithm. A supervised ML algorithm completes the processing and converts the hand gesture data into speech. A new dataset for the ML-based algorithm is created and presented in this manuscript. This dataset includes three sign language datasets, i.e., American Sign Language (ASL), Pakistani Sign Language (PSL), and Spanish Sign Language (SSL). The proposed system automatically detects the sign language and converts it into an audio message for the ND-M. Similarities between the three sign languages are also explored, and further research can be carried out in order to help create more datasets, which can be a combination of multiple sign languages. The ND-M can communicate by recording their speech, which is then converted into text and hand gesture images. The system can be upgraded in the future to support more sign language datasets. The system also provides a training mode that can help D-M individuals improve their hand gestures and also understand how accurately the system is detecting these gestures. The proposed system has been validated through a series of experiments resulting in hand gesture detection accuracy exceeding 95%. deaf-mute person; hand gesture recognition; leap motion device; machine learning; multi-language processing; sign language dataset +
Footnote †: journal: Medical Image and Image Processing  
hardware and uses different algorithms to solve some of the above-discussed problems. This is expected to be a step towards improving the integration of the D-M in society. The development presented in this manuscript uses existing sign language datasets to establish automated communication. The proposed system is designed around commercial-off-the-shelf (COTS) hardware integrated with a software application using machine learning (ML) algorithms that provide full-duplex communication. The use of ML is providing a growing number of solutions to complex problems in diverse areas related to human impairment. ML is already applied to establish effective communication systems so that the D-M and the ND-M can communicate. Researchers are using existing algorithms, as well as creating new algorithms, to enhance the existing systems. ML algorithms are not only being deployed to solve existing problems, but these algorithms are also employed to increase efficiency by reducing processing time, creating consistent datasets, and improving quality. This research is the continuation of the previously published work [1]. In this manuscript, the capability of the previous system is significantly enhanced, and new features include the new combined sign language datasets, American Sign Language (ASL), Pakistani Sign Language (PSL) and Spanish Sign Language (SSL). The online versions of these datasets are enhanced by including more images of better quality and variations of individual gestures. The updated system now includes a language detection feature and an improved user profile which contains more information such as right/left hand, speed and accuracy of user's hand gestures, default user's sign language dataset, etc. The proposed system is further validated using new experiments. A comparison of the three-sign languages is also carried out.