Neural network vowel-recognition jointly using voice features and mouth shape image

-This paper describes a neural approach intended to improve the performance of an automatic speech recognition system for unrestricted speakers by using not only voice sound features but also image features of the mouth shape. In particular, we used the natural sample voice signals and mouth shape images that were acquired in the general environment, neither in the sound isolation room nor under specific lighting conditions. The FTT power spectrum of acoustic speech was used as the voice feature. In addition, the gray level image, binary image and geometrical shape features of the mouth were used as the compensatory information, and compared which kinds of image features are effective. For unrestricted speakers, a vowel recognition rate of about 80% was obtained using only voice features, but this increased to some 92% when voice features plus binary images were used. This method can be applied not only to the improvement of voice recognition, but also to aid the communication of hearingimpaired people. Voice recognition Neural network Binary image features Image processing Lip-reading Mouth shape 1. I N T R O D U C T I O N Acoustic speech signals include most of the information needed to understand a communication, in comparison with other cues. However, recognition of the speech signal itself becomes inaccurate when noise in the environment is very strong. Humans use multiple sources of information, e.g. lip-reading, facial expression, hand gestures and body language, to recognize speech with high accuracy. Especially, the lip-reading is an indispensable approach to the education of hearing-impaired people. When hearing-impaired people recognize sign language, they observe not only the hand shape, but also the mouth shape and facial expression in order to make recognition easy. In practice, they understand about 70% to 80% of the import of a statement only by reading the shape of the mouth. This is why we selected the mouth shape as a compensatory feature in the recognition of speech. Therefore, in this paper, a neural network method of recognizing Japanese vowels based on voice features plus image features of mouth shapes is presented. This new approach, jointly using two kinds of different features in an identical neural network, is different from any former approaches. In our paper, the gray level image, binary image and geometrical features of the mouth shape were tested for comparison to check which kinds of image features are effective for speech recognition by a neural network. In recent years, a new powerful learning algorithm, the artificial neural network, has become a new pattern recognition and signal processing technique. A neural network is a highly parallel information processing system. The application of neural networks to pattern recognition problems requires far less restrictive assumptions about the structure of the input signal. In addition, the inherent parallelism of these networks allows very rapid parallel search and best-match computations. In particular, the error back-propagation neural network (BPN) was applied to the analysis and recognition of speech. 0) In general, the back-propagation network is very powerful for the discrimination of features with unknown structure, like the combined features of this paper. In our experiments, we simultaneously input voice signal features and image features of the mouth shape to a back-propagation neural network for recognition of vowels spoken by unrestricted speakers. Using the mouth shape to recognize acoustic speech signals, we can expect to achieve: (1) improvement in the performance of a speech recognition system, (2) speech recognition in a noisy environment, (3) improvement in speech-reading training systems for hearing-impaired people. (2) Some studies have been reported on the recognition of speech signals using lip-reading. A lipreading postprocessing was added to a conventional