Multimodal Fusion

Computing applications are becoming increasingly complex and pervasive as exemplified by ubiquitous microprocessors in everyday appliances and resulting feature explosion. Recent technologies like speech or gesture recognition can make such systems more natural, easy to use, and robust. Work by [Oviatt et al. 2000] and other groups have shown this to be the case. A challenge has been to fuse these different input modalities effectively to complement their natural strengths and use redundancies to improve robustness. This project looks to examine methods of combining information from different input modalities in multimodal dialogue systems. A theoretically recogniser agnostic and domain independent multimodal framework is developed. The framework is implemented in this project with a commercial speech recogniser and a freeware gesture recogniser. Many improvements were made to the gesture recogniser mainly to handle spatial gestures. Grammar rules and a unification-style algorithm were applied to fuse the semantic frames from different modalities into candidate command frames for the application. By using a multimodal grammar and scoring, the best candidate frame is selected and sent to an application. To validate and evaluate the framework, a multimodal tourist guide for Bristol was built. As implemented, it has zoom and pan commands to navigate the map, and queries about distance and paths between sites and locations of sites. The evaluation task was comprised of four high level goals that users were expected to carry out with minimal guidance. Despite the sparse instruction, users were enthusiastic to multimodal interaction. The unification algorithm used in conjunction with N-Best lists was a viable fusion method. It improved performance over the estimated lower bound and 1-Best case and allowed the system to improve recognition results by using only valid combinations of inputs. These observations were made though with a set of three users; this small sample size limits the extent to which conclusions can be made from the evaluation. Acknowledgements I want to sincerely thank the people who helped to make this an interesting project and this year in Cambridge most memorable. First all of all, I would like to express my gratitude to my thesis supervisor Marianne Hickey for providing excellent guidance, thoughts and patience throughout the development of this dissertation. Secondly, Id to thank Phil Woodland for his time and comments. Id also like to thank Ann Copestake and Mark Gales for their time over the course of the year and for their advice, enthusiasm in teaching, and effort to excel. Id like to thank those people out there who provide free knowledge and software online. The open software movement is a noble cause, and this project would not have been possible without public domain source code to leverage. Lastly, I want to thank all the great people Ive met in Cambridge for making this year a brilliant year. To the sheep herd, I gave a rousing, baaaa of approval to our crazy nights out off the court. To my friends at Harrington House One, thanks for the tea; Ill miss asking for the usual. To the other usual suspects around, you know who you are; theyll be many more hotpots wars and punting adventures to have. And to my special partner, thanks for all the support and encouragement through the days and late nights; Ill always remember the sweet memories of strawberries and champagne.