The image data comprised approximately 15,000 video frames showing the mouth region of the speaker from the tip of the nose to the chin, including the lips, teeth and tongue. They encompassed the range of mouth shapes produced in speaking the vocabulary. After compressing the image data using PCA encoding, HMMs were constructed for every speech sound used in saying number words. Activating appropriate sequences of these HMMs generates the synthesized images. Currently, the synthesizer, which has no sound, is under test at Bath.
First results are encouraging.
We are working on enlarging the vocabulary and encoding higher-resolution color images. Once the trials have been finalized, Dr. Brooke envisions the possibility of building a cheap, fast synthesizer that would enable the hearing-impaired person to be assessed and trained using this talking head gesture method. He believes there is a possibility of using computers to generate tailor-made speech utterances designed specifically to assist assessment or recognition problems. This might involve an expert human trainer to control and supervise the process. In the longer term it may be possible to program the computer to run the assessment and teaching process depending on the subject's performance.