Soft Sign Language Interpreter on Your Skin

Sign language represents an essential means of communication for a significant fraction of the population but can only be used by trained individuals. Researchers report a wearable system that incorporates stretchable sensors and a wireless circuit for real-time sign-to-speech translation. Sign language represents an essential means of communication for a significant fraction of the population but can only be used by trained individuals. Researchers report a wearable system that incorporates stretchable sensors and a wireless circuit for real-time sign-to-speech translation. Around 466 million (one in every twenty) people worldwide suffer from disabling loss of hearing. This number may double in 30 years because of a combination of congenital origins and acquired causes such as infectious diseases, ear infections, use of medicines, and excessive noise.1World Health OrganizationDeafness and hearing loss.https://www.who.int/mediacentre/factsheets/fs300/en/Date: 2020Google Scholar People with hearing loss are often unable to effectively communicate with others, and therefore are at a high risk of experiencing loneliness, anxiety, depression, and even decreased cognitive function. Efforts to address these issues involve initiatives in national sign language education and support from sign language interpreters. These activities, however, have significant costs, estimated to approach $750 billion globally, on an annual basis. Even when supported at these levels, existing approaches are unable to adequately address communication barriers. Automated systems for sign language recognition represent an alternative, and perhaps complementary, means to overcome these barriers. Certain systems for automated sign language recognition are suitable for large-scale production, with potential for real-time sign-to-speech translation for hundreds of millions of people. Such hardware approaches rely on camera(s) or projection of structured light for vision-based recognition, and electromyography (EMG), inertial measurement units (IMUs), or haptic technology for sensor-based recognition schemes.2Cheok M.J. Omar Z. Jaward M.H. A review of hand gesture and sign language recognition techniques.Int. J. Mach. Learn. Cybern. 2019; 10: 131-153Crossref Scopus (145) Google Scholar Although advances in microelectronics and microelectromechanical systems can support miniaturized, portable platforms of these types, further improvements and/or transformations of form factors are necessary for wearable, on-demand sign-to-speech translation technologies. Daunting challenges are in overcoming the mismatch between the materials used in traditional electronic devices—rigid silicon for IMUs and integrated circuits, metal plates for EMG, thick plastic layers for packaging—and the soft, curvilinear surfaces of the human body. Research teams from the University of California, Los Angeles and Chongqing University, led by Profs. Jun Chen and Jin Yang, report a wireless wearable system that addresses these challenges.3Zhou Z. Chen K. Li X. Zhang S. Wu Y. Zhou Y. Meng K. Sun C. He Q. Fan W. et al.Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays.Nat. Electron. 2020; (Published online June 29, 2020)https://doi.org/10.1038/s41928-020-0428-6Crossref PubMed Scopus (150) Google Scholar The system incorporates an array of soft, stretchable strain sensors mounted on five fingers to convert hand gestures of sign language into analog electric signals, and a customized circuit to amplify, multiplex, digitize, and wirelessly transmit the resulting data to portable electronic devices (Figure 1A). A machine-learning algorithm and a graphical user interface enable real-time sign-to-speech translation, with recognition rates >98% and recognition times <1 s (Figure 1B). These platforms have many other attractive properties, including but not limited to, high sensitivity, fast response, and high levels of robustness. The key component is a yarn-based stretchable sensor. The sensor leverages the authors’ expertise in smart textiles4Chen G. Li Y. Bick M. Chen J. Smart textiles for electricity generation.Chem. Rev. 2020; 120: 3668-3720Crossref PubMed Scopus (224) Google Scholar and triboelectric nanogenerators.5Chen J. Wang Z.L. Reviving vibration energy harvesting and self-powered sensing by a triboelectric nanogenerator.Joule. 2017; 1: 480-521Abstract Full Text Full Text PDF Scopus (398) Google Scholar Each sensing unit consists of a stretchable microfiber as the inner core, a conductive yarn based on twisted microfibers of stainless steel and polyester, and a polydimethylsiloxane (PDMS) sleeve to cover the entire structure (Figure 1C). The conductive yarn forms a coil structure around the rubber microfiber to afford uniaxial stretchability of up to 90%. The sensor detects changes in strain based on a combined effect of triboelectrification and electrostatic induction.5Chen J. Wang Z.L. Reviving vibration energy harvesting and self-powered sensing by a triboelectric nanogenerator.Joule. 2017; 1: 480-521Abstract Full Text Full Text PDF Scopus (398) Google Scholar Different electron affinities associated with PDMS and polyester induce electron transfer between the two materials upon physical contact, thereby building an electrical potential (Figure 1D). Tensile strains can change the contact areas between the PDMS and polyester, leading to a variation in electrical potential. The stainless steel serves as an electrode to capture such changes through electrostatic induction. These sensors do not require any specialized materials; they support high levels of elastic stretchability based on structural designs with intrinsically non-stretchable materials and determine changes in strain through simple contact electrification processes. The compatibility of these underlying designs with wide-ranging classes of materials is an important advantage in cost-effective, large-scale manufacturing. The data from these devices must, however, be processed to support accuracy and automation in a process of translation. Here, the team exploits machine learning based on a multi-class support-vector machine classifier, trained using large datasets captured from an array of sensors. As distinct from recent approaches that integrate both visual and somatosensory data,6Wang M. Yan Z. Wang T. Cai P. Gao S. Zeng Y. Wan C. Wang H. Pan L. Yu J. et al.Gesture recognition using a bioinspired learning architecture that integrates visual data with somatosensory data from stretchable sensors.Nat. Electron. 2020; (Published online June 8, 2020)https://doi.org/10.1038/s41928-020-0422-zCrossref Scopus (58) Google Scholar the authors use collections of strain sensors to simultaneously monitor changes at different strategic locations on the hands. The use of arrays in this manner can support accuracies of greater than 98% in gesture recognition, without any imaging techniques or user inputs. The same soft features and fibrous architectures of these sensing systems may support applications in other types of wearable devices. For example, placement near the eyebrows and mouth may allow for digital identification of facial expressions, mounting on multiple joints could support tracking of body movements, and integration with clothing has potential for fitness and activity monitoring.7Zeng W. Shu L. Li Q. Chen S. Wang F. Tao X.M. Fiber-based wearable electronics: a review of materials, fabrication, devices, and applications.Adv. Mater. 2014; 26: 5310-5336Crossref PubMed Scopus (1265) Google Scholar These scenarios could extend uses beyond sign language, to avenues for enhancing communications with digitized information related to aspects such as facial expressions and body language. The translational potential and relatively low-cost construction (∼$50 for the prototype in the laboratory) are additional important features of the technology. Future work may involve (1) acquiring additional training data to improve the accuracy of the machine-learning algorithm, (2) establishing reference datasets for different sign languages and developing mobile applications to convert the translation into different languages, and (3) combining such translations with gestures, body movements, and facial expressions for augmented communication. Additional opportunities lie in heterogeneous integration with other fiber-based electronic,8Xu X. Xie S. Zhang Y. Peng H. The rise of fiber electronics.Angew. Chem. Int. Ed. 2019; 58: 13643-13653Crossref PubMed Scopus (30) Google Scholar optoelectronic,9Rein M. Favrod V.D. Hou C. Khudiyev T. Stolyarov A. Cox J. Chung C.C. Chhav C. Ellis M. Joannopoulos J. Fink Y. Diode fibres for fabric-based optical communications.Nature. 2018; 560: 214-218Crossref PubMed Scopus (111) Google Scholar and energy devices10Chen J. Huang Y. Zhang N. Zou H. Liu R. Tao C. Fan X. Wang Z.L. Micro-cable structured textile for simultaneously harvesting solar and mechanical energy.Nat. Energy. 2016; 1: 16138Crossref Scopus (611) Google Scholar for multifunctional wearable systems.