1.  Field of the Invention
The field of the invention is data processing, or, more specifically, methods, apparatus, and products for providing expressive user interaction with a multimodal application. 
2.  Description of Related Art
User interaction with applications running on small devices through a keyboard or stylus has become increasingly limited and cumbersome as those devices have become increasingly smaller.  In particular, small handheld devices like mobile phones and PDAs serve many functions and contain sufficient processing power to support user interaction through multimodal access, that is, by interaction in non-voice modes as well as voice mode.  Devices which support multimodal access combine multiple user input modes or channels in the same interaction allowing a user to interact with the applications on the device simultaneously through multiple input modes or channels.  The methods of input include speech recognition, keyboard, touch screen, stylus, mouse, handwriting, and others.  Multimodal input often makes using a small device easier. 
Multimodal applications are often formed by sets of markup documents served up by web servers for display on multimodal browsers.  A ‘multimodal browser,’ as the term is used in this specification, generally means a web browser capable of receiving multimodal input and interacting with users with multimodal output, where modes of the multimodal input and output include at least a speech mode.  Multimodal browsers typically render web pages written in XHTML+Voice (‘X+V’).  X+V provides a markup language that enables users to interact with an multimodal application often running on a server through spoken dialog in addition to traditional means of input such as keyboard strokes and mouse pointer action.  Visual markup tells a multimodal browser what the user interface is look like and how it is to behave when the user types, points, or clicks.  Similarly, voice markup tells a multimodal browser what to do when the user speaks to it.  For visual markup, the multimodal browser uses a graphics engine; for voice markup, the multimodal browser uses a speech engine.  X+V adds spoken interaction to standard web content by integrating XHTML (eXtensible Hypertext Markup Language) and speech recognition vocabularies supported by VoiceXML.  For visual markup, X+V includes the XHTML standard.  For voice markup, X+V includes a subset of VoiceXML.  For synchronizing the VoiceXML elements with corresponding visual interface elements, X+V uses events.  XHTML includes voice modules that support speech synthesis, speech dialogs, command and control, and speech grammars.  Voice handlers can be attached to XHTML elements and respond to specific events.  Voice interaction features are integrated with XHTML and can consequently be used directly within XHTML content. 
In addition to X+V, multimodal applications also may be implemented with Speech Application Tags (‘SALT’).  SALT is a markup language developed by the Salt Forum.  Both X+V and SALT are markup languages for creating applications that use voice input/speech recognition and voice output/speech synthesis.  Both SALT applications and X+V applications use underlying speech recognition and synthesis technologies or ‘speech engines’ to do the work of recognizing and generating human speech.  As markup languages, both X+V and SALT provide markup-based programming environments for using speech engines in an application's user interface.  Both languages have language elements, markup tags, that specify what the speech-recognition engine should listen for and what the synthesis engine should ‘say.’  Whereas X+V combines XHTML, VoiceXML, and the XML Events standard to create multimodal applications, SALT does not provide a standard visual markup language or eventing model.  Rather, it is a low-level set of tags for specifying voice interaction that can be embedded into other environments.  In addition to X+V and SALT, multimodal applications may be implemented in Java with a Java speech framework, in C++, for example, and with other technologies and in other environments as well. 
As mentioned above, current multimodal applications provide user interaction with a multimodal device through a variety of modalities.  For example, a multimodal application may provide user interaction through a voice mode using a speaker and a microphone or through a visual mode using a graphical user interface.  Within each mode of user interaction, a multimodal application may vary the expressiveness of the interaction with the user.  For example, within the voice mode of user interaction, the multimodal application may use a strong, confident voice to prompt a user for input and use a meek, more hesitant voice when requesting the user to clarity the response.  Within the visual mode of user interaction, the multimodal application may use normal text on a graphical user interface to prompt a user for input and use a bold, red text to indicate that the user needs to provide a more detailed response.  The drawback to current multimodal applications, however, is that expressive user interaction is not typically synchronized across multiple modes of user interaction.  That is, input provided in a one mode of user interaction does not affect the expressiveness of user interaction in other modes of user interaction.  As such, readers will appreciates that room for improvement exists in providing expressive user interaction with a multimodal application.