Tailor: creating custom user interfaces based on gesture

Randy Pausch & Ronald D, Williams University of Virginia Thornton Hall Charlottesville, VA 22903 (pausch@Virginia.edu) 804-982-2211 Physical controls for most devices are either “one size fits all” or require custom hardware for each user. Cost often prohibits custom design, and each user must adapt to the standard device interface, typically with a loss of precision and efficiency. When user abilities vary widely, such as in the disabled community, devices often become unusable. Our goal is to create a system that will track user gestures and interpret them as control signals for devices. Existing gesture recognition research converts continuous body motion into discrete symbols. Our approach is to map continuous motions into a set of analog device control signals. Our system will allow us to quickly tailor a device interface to each user’s best physical range of motion. Our first application domain is a speech synthesizer for disabled users. We expect two major areas of applicability for non-disabled users: in telemanipulator interfaces, and as a design tool for creating biomechanically efficient interfaces. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. Introduction The Augmentative Communications Group at the University of Virginia consists of researchers from the Computer Science and Electrical Engineering departments, and from the Medical and Education schools. Our current effort is to create a speech synthesizer for individuals with cerebral palsy, a disability affecting approximately 700,000 Americans [Il. A significant portion of the cerebral palsy population is communicative but non-verbal although the desire to communicate is present, speech is prohibited by damage to the part of the brain that controls the vocal tract. Most of these individuals do not have enough coordination for handwriting or typing. Al though primitive electronic communication aids exist, most are variations on picture boards, where the user points or looks at a twodimensional array of pictures to convey a thought such as “hungry” or “tired. ” The Augmentative Communications Group has developed a speech synthesizer based on two-dimensional analog input. The creation of speech involves the coordination of a large number of muscles in the vocal tract. The synthesizer approximates this by receiving the position of the base and tip of the tongue as input signals and then synthesizes the sound produced by that position of the tongue. We have implemented a prototype @ 1990 ACM 089791410-4/90/0010/0123 $1.50 123 which synthesizes monotone speech from two analogue signals. The original research strategy was to attempt to design and build custom input devices for each user of the system. Biomedical engineers constructed various onedimensional potentiometers to be used in pairs to provide the analog inputs needed to drive the synthesizer. Building custom hardware interfaces is expensive, and cerebral palsy victims often have reduced strength, making control of any physical device cumbersome. As users fatigue, their efficiency with a particular device decreases and several different devices may be needed to accommodate various stages of fatigue. Our new approach is to create an individual gesture interface for each user. Our software maps body motions, reported by magnetic trackers, into continuous control signals for the speech synthesizer. The only physical effort by the user is to move a part of his body. This software tailoring allows us to create interfaces based on each user’s individual abilities, and makes it possible for those interfaces to adapt as the user fatigues. The idea of user tailoring, or customization, has long been understood as a crucial element in the design of traditional computer interfaces. Text editors allow users to rebind keyboards so that commonly used operations are easier to reach [ZI. Mice often allow for alterations in the ratio of device motion to cursor motion, to accommodate variations in user coordination. Some hardware interfaces allow minor customization! such as tilt-lock steering wheels in automobiles. Customization is necessary when there is a high degree of user idiosyncrasy, relative to the dexterity required for the task. While the need for individual tailoring is most apparent in the disabled community, able-bodied users exhibit high idiosyncrasy with respect to tasks where extreme dexterity is required, such as telemanipulation in microsurgery. We expect our techniques to be useful for able-bodied users when they must use interfaces that require high dexterity to complete tasks. Existing gesture research is dominated by a desire to understand or interpret gestures, and is commonly referred to as gesture recognition. Our approach is to map continuous data from one or more sensors to a set of continuous device control signals, rather than transforming gestures into symbols. Our primary goal is to create custom gesture mappings for device control. Our secondary goal is to make our mappings dynamically adjust for fatigue and changes in user ability. A Joystick-Driven Speech Synthesizer Because our synthesizer is such an unusual device, we first describe how it is able to synthesize speech from two analog inputs. Our articulator driven speech synthesizer produces sounds using the positions and motions of implied articulators in a simulated vocal tract. This form of speech synthesis has been discussed previously in the literature [3-51. The problem addressed here differs from previous work because we limit the number of articulator control parameters to those that can be provided by a human user in real-time. Articulator driven. synthesis is unnecessary and constraining in the text-to-speech environment, but this approach is directly analogous to the mechanisms of speech production used for normal human conversational speech. A brief review of human physical speech production will be helpful in understanding the articulator driven synthesis approach. The physical process of speech production can be divided into three parts, First, air is forced through the vocal cords to produce either a voiced or unvoiced glottal excitation. Next, air flow is modified by a series of structures that constitute the vocal tract. Finally, the modified flow is radiated through the lips and nostrils. [61. The articulators used to produce speech are shown in Figure 1.