Visual Recognition of American Sign Language Using Hidden Markov Models by Thad

Using hidden Markov models (HMM's), an unobstrusive single view camera system is developed that can recognize hand gestures, namely, a subset of American Sign Language (ASL). Previous systems have concentrated on finger spelling or isolated word recognition, often using tethered electronic gloves for input. We achieve high recognition rates for full sentence ASL using only visual cues. A forty word lexicon consisting of personal pronouns, verbs, nouns, and adjectives is used to create 494 randomly constructed five word sentences that are signed by the subject to the computer. The data is separated into a 395 sentence training set and an independent 99 sentence test set. While signing, the 2D position, orientation, and eccentricity of bounding ellipses of the hands are tracked in real time with the assistance of solidly colored gloves. Simultaneous recognition and segmentation of the resultant stream of feature vectors occurs five times faster than real time on an HP 735. With a strong grammar, the system achieves an accuracy of 97%; with no grammar, an accuracy of 91% is reached (95% correct). Thesis Supervisor: Alex Pentland Title: Head, Perceptual Computing Section, MIT Media Lab This work was supported in part by the USAF Laboratory Graduate Fellowship program. Visual Recognition of American Sign Language Using Hidden Markov Models by Thad Eugene Starner The following people served as readers for this thesis: Reader: Alex Pentland Head, Perceptual Computing Section Program in Media Arts and Sciences Reader: Nathaniel I. Durlach Senior Research Scientist Electrical Engineering and Computer Science Reader: Pattie Maes Assistant Professor Program in Media Arts and Sciences I'd like to thank Alex Pentland, my advisor, for insisting over my objections that this project was possible within the bounds of a Master's thesis. His experienced insight into what is possible and worthwhile has guided me throughout my MIT career. I would also like to thank my readers, Nathaniel Durlach and Pattie Maes for coping with the changes over the past year. Many thanks to the USAF Lab Graduate Fellowship program for funding my graduate studies. Thanks also to John Makhoul, Rich Schwartz, Long Nguyen, Bruce Musicus, Paul Placeway, and numerous others in the Bolt, Barenek, and Newman Speech Group for giving me a good understanding of and intuition for modern HMM techniques. Along the same lines, thanks to Roz Picard and Alex Pentland for their lucid explanations of pattern recognition techniques which got me started in this field. Thanks to Judy Bornstein for sharing her experience with ASL and for proofing this document. Thanks to Mike P. Johnson, Ken Russell, and the IVE gang at the Media Lab for their contributions of code snippets, which made my work much easier. Thanks also to the Vision and Modeling Group, which has harbored me for many years as I've been "playing with the toys." It is my pleasure to work with such talented and exciting people. A big thank-you to my parents, who have supported me without fail since I first discovered that I wanted to do research at MIT. Last but not least, thanks to Tavenner Hall for keeping me sane through the last death throes of this thesis.