MATCH: An Architecture for Multimodal Dialogue Systems

Mobile interfaces need to allow the user and system to adapt their choice of communication modes according to user preferences, the task at hand, and the physical and social environment. We describe a multimodal application architecture which combines finite-state multimodal language processing, a speech-act based multimodal dialogue manager, dynamic multimodal output generation, and user-tailored text planning to enable rapid prototyping of multimodal interfaces with flexible input and adaptive output. Our testbed application MATCH ( provides a mobile multimodal speech-pen interface to restaurant and subway information for New York City. 1 Multimodal Mobile Information Access In urban environments tourists and residents alike need access to a complex and constantly changing body of information regarding restaurants, theatre schedules, transportation topology and timetables. This information is most valuable if it can be delivered effectively while mobile, since places close and plans change. Mobile information access devices (PDAs, tablet PCs, next-generation phones) offer limited screen real estate and no keyboard or mouse, making complex graphical interfaces cumbersome. Multimodal interfaces can address this problem by enabling speech and pen input and output combining speech and graphics  2002) for a detailed overview of previous work on multimodal input and output). Since mobile devices are used in different physical and social environments, for different tasks, by different users, they need to be both flexible in input and adaptive in output. Users need to be able to provide input in whichever mode or combination of modes is most appropriate, and system output should be dynamically tailored so that it is maximally effective given the situation and the user’s preferences. We present our testbed multimodal application MATCH ( and the general purpose multimodal architecture underlying it, that: is designed for highly mobile applications; enables flexible multimodal input; and provides flexible user-tailored multimodal output.  MATCH running on Fujitsu PDA Highly mobile MATCH is a working city guide and navigation system that currently enables mobile users to access restaurant and subway information for New York City (NYC). MATCH runs standalone on a Fujitsu pen computer , and can also run in client-server mode across a wireless network. Flexible multimodal input Users interact with a graphical interface displaying restaurant listings and a dynamic map showing locations and street information. They are free to provide input using speech, by drawing on the display with a stylus, or by using synchronous multimodal combinations of the two modes. For example, a user might ask to see cheap Italian restaurants in Chelsea by saying show cheap italian restaurants in chelsea, by circling an area on the map and saying show cheap italian restaurants in this neighborhood; or, in a noisy or public environment, by circling an area and writing cheap and italian . The system will then zoom to the appropriate map location and show the locations of restaurants on the map. Users can ask for information about restaurants, such as phone numbers, addresses, and reviews. For example, a user might circle three restaurants as in Figure 3 and say phone numbers for these three restaurants (or write phone). Users can also manipulate the map interface directly. For example, a user might say show upper west side or circle an area and write zoom.  Unimodal pen command Flexible multimodal output MATCH provides flexible, synchronized multimodal generation and can take initiative to engage in information-seeking subdialogues. If a user circles the three restaurants in Figure 3 and writes phone, the system responds with a graphical callout on the display, synchronized with a text-to-speech (TTS) prompt of the phone number, for each restaurant in turn .  Two area gestures Figure 4: Phone query callouts The system also provides subway directions. If the user says How do I get to this place? and circles one of the restaurants displayed on the map, the system will ask Where do you want to go from? The user can then respond with speech (e.g., 25th Street and 3rd Avenue), with pen by writing (e.g., 25th St & 3rd Ave), or multimodally ( e.g, from here with a circle gesture indicating location). The system then calculates the optimal subway route and dynamically generates a multimodal presentation of instructions. It starts by zooming in on the first station and then gradually zooms out, graphically presenting each stage of the route along with a series of synchronized TTS prompts. Figure 5 shows the final display of a subway route heading downtown on the 6 train and transferring to the L train Brooklyn bound.  Multimodal subway route User-tailored generation MATCH can also provide a user-tailored summary, comparison, or recommendation for an arbitrary set of restaurants, using a quantitative model of user preferences (Walker et al., 2002). The system will only discuss restaurants that rank highly according to the user’s dining preferences, and will only describe attributes of those restaurants the user considers important. This permits concise, targeted system responses. For example, the user could say compare these restaurants and circle a large set of restaurants . If the user considers inexpensiveness and food quality to be the most important attributes of a restaurant, the system response might be: Compare-A: Among the selected restaurants, the following offer exceptional overall value. Uguale’s price is 33 dollars. It has excellent food quality and good decor. Da Andrea’s price is 28 dollars. It has very good food quality and good decor. John’s Pizzeria’s price is 20 dollars. It has very good food quality and