Combining representations for improved sketch recognition

Sketching is a common means of conveying, representing, and preserving information, and it has become a subject of research as a method for human-computer interaction, specifically in the area of computer-aided design. Digitally collected sketches contain both spatial and temporal information; additionally, they may contain a conceptual structure of shapes and subshapes. These multiple aspects suggest several ways of representing sketches, each with advantages and disadvantages for recognition. Most existing sketch recognitions systems are based on a single representation and do not use all available information. We propose combining several representations and systems as a way to improve recognition accuracy. This thesis presents two methods for combining recognition systems. The first improves recognition by improving segmentation, while the second seeks to predict how well systems will recognize a given domain or symbol and combine their outputs accordingly. We show that combining several recognition systems based on different representations can improve the accuracy of existing recognition methods. Thesis Supervisor: Randall Davis Title: Professor Acknowledgments First, I must thank my advisor Randy Davis, who always asks the right question and provides the right way of looking at a problem, and who may very well be the best advisor at MIT. I didn't know what I was doing when I first arrived at MIT and chose a group, but I certainly chose well. I would like to thank my committee members Patrick Winston and Pawan Sinha for their valuable suggestions, insights, and questions, and Professor Winston in particular for sharing his renowned presentation skills. I have been very fortunate not only in my advisors, but in my group mates as well. I have benefited greatly from the advice, feedback, and community the Design Rationale/Multimodal Understanding Group has provided over the years. Thank you in particular to Jacob Eisenstein and Tom Ouyang for being such great officemates and sounding boards, and to Mike Oltmans, Christine Alvarado, and Metin Sezgin for not only being great group mates but also for providing the work upon which this thesis is based. I am particularly indebted to Aaron Adler for being a great officemate and frequent lunch mate, and for his invaluable tech support. Thank you to T! G, and to Anthony in particular, for solving every problem thrown at them, including but not limited to, excessive processor time demands, mice, leaky ceilings, odd smells and extreme temperatures. I am grateful to many outside of CSAIL as well. My time spent with the cycling team will always be one of the best experiences of my life, and the women's team in particular has provided me with friendships that will last beyond MIT. Thank you to my parents for their unwavering love and support (and for not asking too many questions like that pesky "When will you be finished? " one). Thank you to my family, both new and old, for their kindness, support and understanding. Finally, thank you to my husband Bill, who has been wonderfully supportive and encouraging. I am grateful to have had him with me for the end of this journey, and look forward to the beginnings of many more.