Dynamic mixed-initiative dialog generation in speech recognition

Disclosed are a method (500), apparatus (100) and computer program product for generating a mixed-initiative dialog to obtain information for dialog slots. A composite grammar dependent upon a set of unfilled slots is constructed (501). A prompt, dependent upon the a set of unfilled slots, is presented (309) to a user. An utterance is received (301) from the user in response to said prompt. Relevant information is determined based upon the further utterance. One or more said unfilled slots are filled (302) with said relevant information.

FIELD OF THE INVENTION

The present invention relates generally to speech recognition systems, and in particular, to dialog-based speech recognition systems.

BACKGROUND

Speech-enabled applications, enabling users to interact with machines using speech as a control mode, are becoming more prevalent with advances in technology.

Natural-language speech enabled systems attempts to closely emulate human-human interaction and ideally allow users to speak in a natural manner. Such systems ask open ended questions like “How May I Help You?” to the user and allow the user to respond in the user's own desired manner, a manner over which the system has no control. In order to accommodate this user flexibility, a natural-language-based speech recognizer must have a relatively large vocabulary, and a relatively large grammar, which tend to result in poor recognition accuracy. Moreover, in order to understand the free-form response, which is typical of such systems, natural-language-based systems also require a high level of natural language understanding.

On the other hand, dialog-based speech enabled systems ask very specific questions of the user and each question requires a specific response that is restricted to a set of pre-defined inputs as decided by the system. Dialog-based systems ask the user a specific question (also referred to as a “prompt”), and based upon the user's response, the dialog-based system progresses in a particular (pre-defined) order to thereby acquire sufficient information from the user to perform the desired action. Dialog-based systems exploit the limited context which results from the dialog-based approach, in order to improve recognition accuracy. Consequently, in the dialog-based system, a speech recognizer only needs to handle small grammars when processing the response elicited by each prompt in the generated dialog. This approach also reduces the size of the vocabulary required by the recognizer. The recognition accuracy of dialog-based speech recognition systems can accordingly be increased. However dialog-based systems force the user to model his or her response in a system-defined manner. Another disadvantage of dialog-based systems is the fact that the user has to traverse the prompt/response tree in order to obtain the desired information that resides at a specified leaf of such a tree.

In dialog-based systems, the inputs to the system are typically referred to as “slots” (also referred to as “fields” or “information fields” in this description), where a pre-defined set of slots is needed by application in order to perform a corresponding task. Each member slot is associated with a specific type of information. Typical dialog-based arrangements use a “system-initiated” approach, also known as directed-dialog approach, in which the user must respond to prompts from the system precisely in the order defined by the system. In such arrangements, specific grammar is defined along with a suitable prompt to elicit information to fill a particular slot. Multiple slots typically can not be filled based upon a single user utterance. Furthermore, the user utterance can not be used to fill any other slot other than the one for which information has been solicited. This approach results in rigid system-directed interaction which makes the interaction long and monotonous for the user, often resulting in user dissatisfaction.

To overcome these problems and make dialog-based system more flexible, mixed-initiative dialog systems have been developed. In mixed-initiative systems the user need not make a response which is strictly compliant with the prompt. The user response can also be used to fill a slot other than the slot that is directly associated with the current prompt. Furthermore, more than one slot can be filled on the basis of a single user utterance. This approach places some control with the user who consequently has some flexibility of approach in filling the slots, and both the computer and the user play a role in directing the dialog.

Mixed initiative systems require composite grammars (also referred to as Mixed-Initiative or MI grammars in this description) which allow slots to be filled arbitrarily. Existing mixed-initiative systems are however inflexible, complex and not easily portable across applications.

The Voice Extensible Markup Language (VXML) specification of the World Wide Web Consortium (W3C) provides constructs for writing MI dialogs. The VXML “form-level grammar” allows more than one field to be filled using a single user utterance. It is also possible to fill up information fields other than those being asked about by the system. The VXML construct “initial” together with form-level grammar and the VXML “Form Interpretation Algorithm” (FIA) are used in MI applications using VXML. However, these VXML constructs enable only very primitive mixed-initiative dialog systems. In particular, the prompts presented by such systems typically do not correspond well with the information to be collected from the user. There is no mechanism to enable information collection for only a subset of slots among the initial set of MI slots in a dialog interaction. The support for “confirmation” and “disambiguation” is not robust. The resulting systems are inflexible and can neither be easily configured for different behaviour, nor easily ported for different applications.

Agarwal et al. (R. Agarwal, B. M. Shahshahani, “Method and Apparatus for Providing A Mixed-Initiative Dialog Between A User and A Machine”, US Patent Application US2004/0085162 A1, May 6, 2004) presents a mixed-initiative dialog system that presents a natural language speech interface to the user. The speech recognizer in Agarwal uses statistical language models. Agarwal uses Natural Language Processing (NLP) to parse a user utterance in order to obtain the information needed to fill various slots. However, as discussed, natural language speech approaches are very prone to recognition error, with consequent lack of accuracy. Furthermore, use of NLP for parsing adds further recognition errors and system complexity.

SUMMARY

It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements. Disclosed are arrangements, referred to as Dynamic Mixed-Initiative Dialog Generation in Speech Recognition (or simply as DMIDG arrangements) which seek to address the above problems by (a) automatically generating a composite grammar from the atomic grammar(s) associated with individual slots, (b) making such arrangements applicable to composite tasks, and (c) dynamically enabling multiple slots to be filled based upon a single user response, (d) dynamic generation of prompts from base prompts, (e) dynamic generation of voice-dialog code for each dialog-interaction (also referred to as a dialog cycle). The DMIDG arrangements also make available configuration parameters which facilitates portability across different applications.

Therefore, broadly stated, there are disclosed methods, apparatus and computer program products for generating a mixed-initiative dialog to obtain information for a pre-defined set of dialog slots. A sub set of these slots for collecting information from the user is selected based on user-interaction history and desired dialog flow. A composite grammar dependent upon the selected subset of slots is constructed. A prompt, dependent upon the selected sub set of slots is generated and presented to the user. An ASR recognized user-input is received from the user in response to said prompt. The recognized user-input is parsed and matched against slot grammars to identify if the user-input provides relevant information for the slot. One or more said unfilled slots are filled with said relevant information.

In a further arrangement, the DMIDG approach can be implemented to incorporate dynamic generation of VXML code at the client side, to provide for user interaction, automatic generation of composite grammar for mixed-initiative dialog from individual grammar components, and author defined composition rules. An arrangement using, for example XML to define the MI dialog flow enables provision of different dialog flavours which make such arrangements usable across different applications.

Appendix A is an XML representation of the slot information component;

Appendix B is an XML representation of the MI dialog configuration component;

Appendix C depicts an example of the grammar composition rule where the composition rules has been specified using XML;

Appendix D shows an example of dialog flow information; and

Appendix E shows another example of dialog flow information.

DETAILED DESCRIPTION

The disclosed DMIDG arrangements provide a mechanism for generating a grammar based Mixed Initiative (MI) dialog that allows dynamic selection of a subset of slots for information collection. The arrangements can be configured for different dialog flow strategies, for use across different applications. The DMIDG arrangements provide support for disambiguation, confirmation and use of reusable dialog components.

An important element in the disclosed DMIDG arrangements is Dynamic composition of composite grammar from atomic grammars (or rules for atomic grammar) using grammar composition rules. A grammar specifies permissible user utterances and valid values for a piece of information to be collected from the user. The atomic grammar(s) specify grammar for single atomic pieces of information. If the information to be collected from the user consists of multiple atomic pieces of information, a composite grammar is required.

The mechanism for composing the composite grammar involves specifying rules that define how atomic grammar(s) (or the rules for atomic grammars) can be combined to dynamically form the composite grammar for a given set of information/dialog slots. It is not necessary for grammar composition to specify rules for each possible permutation of slots. Accordingly, the rules define how atomic grammar (or the rules for atomic grammars) can be combined. The composition Rules may be generic across slots, and apply to a given slot irrespective of its arrangement in the set of input slots. The generic rules refer to the atomic grammars and specify mandatory and optional prefixes and suffixes which apply to the atomic grammar for a given slot for any combination of this slot with other slots. There is also provision to specify the rules that apply to specific permutations of slots. The rules can thus, for example, specify conjunctions to join two atomic grammars, permutation specific prefixes, suffixes and so on.

In one particular arrangement, the mechanism specifies an XML schema and uses XML to define the grammar composition rules.

Another important element in the disclosed DMIDG arrangements is dynamic prompt generation. Thus, for example, an application may need to collect different pieces of information (represented by different slots) from the user. When using dynamic dialogs, the subset of slots that needs to be filled is dynamically determined during user interaction. The prompts that need to be presented are also to be determined dynamically. It is very tedious and cumbersome to provide prompt for each possible permutation of Slots. Accordingly dynamic generation of prompts, for example for “input” as well as events like “help”, “confirmation” and so on is provided by defining rules for prompt generation. These rules have a general part that defines the prompt segment(s) which is general to all slots, and a slot specific part that defines prompt segment(s) that would be included in a prompt only if corresponding slot(s) are members of subset of slots that are part of dialog cycle. These rules can also specify inclusion of a slot value in a prompt segment.

In one particular arrangement, the mechanism specifies an XML schema and uses XML for dynamic prompt generation.

Another important element in the disclosed DMIDG arrangements is dynamic selection of slots for information collection in a particular dialog turn (also referred to as a dialog cycle). For a given set of slots, the subset of slots for which information needs to be collected may vary during interaction with user and may depend on dialog history, user-input, desired dialog flow among others. For example, slots that have already been filled may need to be disabled for information collection.

The DMIDG arrangements thus provide a mechanism for dynamically enabling a subset of slots for information collection. Information collection for other slots is disabled. According to this arrangement, voice dialog code is generated after every user-interaction cycle. User-input is processed on receiving user response based on the subset of slots for which information is solicited from the user. The slot information is updated and voice dialog code is generated for next cycle. Furthermore, a dialog flow or set of rules can be specified for generation of voice dialog. The dialog flow rules may also specify task of disambiguating collected information or confirming the information besides the task of collecting information for the slots. The voice dialog code generated, as specified above, depends both on the user-input in previous interaction(s) and on the dialog flow rules.

In one particular arrangement, the mechanism specifies an XML schema to define the dialog flow.

Grammar Terminology

Atomic grammars specify basic grammars for an atomic piece of information.

Slot grammar specifies the grammar that is applicable for the given slot. It usually consists of an atomic grammar that forms the base of the slot grammar and a set of suitable prefixes and suffixes added to it.

Base grammar is the atomic grammar that forms the base (core) of the slot grammar

Example

Consider the example of a round trip flight that asks for departure date and arrival date which are two information slots. There is a date grammar that specifies basic rules/utterances for specifying a date in general. Valid utterances include 19thMar. 1973, Mar. 19, 1973, Apr. 5, 1980, 5thof Feb. 2007 etc. The date grammar specifies date irrespective of context.

Departure date and arrival date are also dates but they also have some specific connotations/context added to them. If a user says 10thMar. 2007, it is not known if the user meant departure or arrival date. The user may specify additional prefixes/suffixes to be added to vanilla date grammar to form grammar for the respective slot information.

The grammar for departure date may specify “departing on” as a prefix to date and arrival date may specify “arriving on” or “reaching on” as prefixes to date grammar. The slot grammar for departure date would be as follows:

a) “departing on”<date> (all possible utterance of date, < > means all possible values of element to be combines with specified qualifier)

Slot grammar for arrival date would be

Date is an Atomic Grammar

a) is slot grammar for departure date and b) is slot grammar for arrival date.

The date atomic grammar forms the base of the slot grammars a) and b) and is also known as base grammar in context of slot grammars.

While specifying rules for creating composite grammar for a subset of given set of slots, many rules are generic to a slot. Generic rules for a slot in this context mean the rules that apply to a slot irrespective of its permutation with other slots (ie the rules can apply to slots irrespective of their membership in a particular permutation of slots). For example flight reservation composite information asks for class, departure city and arrival city besides dates. For departure city, “from” as prefix to <city> is a generic rule for departure city slot. It could be used for all combinations of departure city with other info slots.

Possible permutations along with the example utterance include:

From as prefix is thus a generic rule for departure city slot, similarly To as prefix is generic rule for arrival city slot as they applies to these slots irrespective of where they are positioned in composite utterance.

In summary, the disclosed DMIDG arrangements for dynamic generation of MI dialog makes use of above mechanisms.

Functional Block Diagram

FIG. 1shows a functional block diagram of a DMIDG system100. A Voice User Interface (VUI)105interfaces with the user (not shown). The VUI105presents, as depicted by an arrow102, prompts to the user, to which the user responds with voice utterances (also referred to as a voice input)101. The VUI105receives these utterances101from the user. The VUI105includes an audio input interface103, and an Automatic Speech Recognition module (ASR)104. The VUI105also includes a text-to-speech module (TTS)106, and a voice browser133that interprets the voice-dialog code and executes it.

A voice browser (133) is a web browser that presents an interactive voice user interface to the user. Just as a visual web browser works with HTML pages, a voice browser operates on pages that specify voice dialogues. The voice dialogues are implemented using voice dialogue languages like VoiceXML (VXML), the W3C's standard voice dialog markup language, SALT, and other proprietary languages.

Voice browser makes use of other elements of VUI (TTS, ASR and Audio I/p) to execute voice dialog. It uses TTS to render textual information as audio and present this information aurally to the user. The voice browser receives user input in form of text from ASR.103is an audio input device, usually a microphone that transforms user's acoustic input to equivalent electrical signal. ASR (104) receives this transformed electrical speech signal and converts it to text using speech recognition algorithms that in turn makes use of appropriate acoustic models and language models/grammar.

The VUI105provides, as depicted by an arrow119, utterance information to a Dialog Manager (DM)123. The DM123manages the flow of the complete dialog with the user. The DM123receives the utterance information119from the VUI105, and directs, as depicted by an arrow122, inputs to a VUI generator121. The VUI generator121can be implemented, for example, using programming languages such as C or Java.

In response to these inputs122, the VUI generator121generates appropriate voice-dialog code120that implements a voice dialog. The voice dialog code120can take various forms, depending upon the implementation, and can be in VXML, or in another language such as SALT, X+V suitable for implementing a voice dialog. The voice dialog code is communicated, as depicted by an arrow120, to the VUI105. The voice browser133in the VUI105executes the voice-dialog code120and drives the other VUI components (eg the ASR104and the TTS106). The voice browser133passes the text prompt120to the TTS106which makes use of this text prompt120to output the prompts delivered to the user as depicted by the arrow102.

The DM123decides which type of interaction is to occur (i.e., input, confirmation, and disambiguation as described in relation toFIG. 2), which slots are to be filled, which prompts are to be provided to the user, and which pre-defined (atomic) grammars need to be active in order to obtain required input from the user to fill the slots of the dialog in question. The DM123provides, as depicted by an arrow117, appropriate inputs to a grammar composer116in order to obtain the required composite grammar. The DM123receives, as depicted by an arrow118, corresponding information (i.e., an appropriate composite grammar) from the grammar composer116. For every MI dialog, the DM123maintains an ongoing history of which user responses (i.e., utterances in response to prompts) have been received.

The Grammar Composer116is responsible for grammar composition. The grammar composer116uses, as depicted by dashed arrows112-115, sets of pre-defined atomic grammars108-110and pre-defined composition rules111as inputs. The composition rules111refer to the atomic grammars108-110and specify mandatory and optional prefixes and suffixes which apply to any combination of the atomic grammars108-110. The composition rules may be generic and need not be defined for each possible permutation of atomic grammars. The composition rules can also define rules for specific permutations of slots. Thus, for example, the composition rules can specify conjunctions to join two atomic grammars, permutation specific prefixes and suffixes and so on. Based on a dialog state signalled by the DM123, as depicted by the arrow117, the grammar composer116generates an appropriate mixed-initiative composite grammar and communicates this, as depicted by the arrow118, to the DM123. The composite grammar specifies permissible user utterances and valid information to be collected from the user. The atomic grammars108-110specify grammars for single atomic pieces of information.

With regard to the prompt(s), help and other events which are required for a dialog, the DM123makes use, as depicted by dashed arrows130and125respectively, of a pre-defined Slot Information component132, and of a pre-defined MI dialog configuration component128. The slot information component132defines all the slots that are part of the MI dialog and may optionally specify prompt information for input, help, and repeat events associated with each slot. An example of the slot information component132is presented in Appendix A. The MI dialog configuration component128specifies prompt information for an MI dialog. The MI dialog configuration component128thus enables the DM123to generate dynamic prompts corresponding to a subset of slots for which information is being solicited. An example of the MI dialog configuration component128is presented in Appendix B.

The various system elements can be stored in a centralised or distributed manner, according to system requirements and/or convenience, in a remote server426, a storage device409, or similar devices (not shown) as depicted inFIG. 5.

Overall Process Flow

FIG. 2is a flow chart showing an example process500of how the system ofFIG. 1operates. The process500comprises three concurrent sub-processes500A,500B,500C.

In the disclosed DMIDG arrangements, the subset of slots that needs to be filled in a particular dialog cycle is dynamically determined during user interaction. The prompts that need to be presented are also determined dynamically. In the sub-process500A, and particularly a step501, the grammar composer116(seeFIG. 1) generates (i.e., constructs) the composite grammar based on the atomic grammars108-110, the composition rules111, and the dialog state. Appendix C depicts an example of the grammar composition rules111where the composition rules have been specified using XML. Other formats such as tokenized text, graph etc. equally can be used to specify the composition rules. This sub-process500A loops continuously as depicted by an arrow502, as described in more detail in regard toFIG. 3.

Concurrently, in a step503of the sub-process500B, the DM123determines the structure of the dialog to be presented to the user, generates suitable prompts, and invokes the VUI generator121to generate the Voice dialog code120with appropriate prompts, and presents it to user. This sub-process500B loops continuously, as depicted by an arrow504, as described in more detail in regard toFIG. 4. The described example uses VXML to implement Voice dialog, however other languages and data structures SALT, X+V can equally be used.

In a step501of the third concurrent sub-process500C, the DM123receives the utterance101from the user that the user utters in response to the input prompt102that is generated by the step503, and fills slots based upon the user utterance. The step501fills one or more slots dependent upon the aforementioned received utterance from the user. Thereafter, in a step502, the DM123determines, based upon the dialog flow component127and the slots information component132, whether the execution flow of the current MI dialog is complete or not. If the dialog is complete, then the process500C follows a “YES” arrow from the step502to a “STOP” step503. If the dialog is not complete, then the process500C follows a “NO” arrow from the step502to a step504. In the step504, the DM123determines, based upon the current dialog state and the dialog flow information127whether to (a) collect remaining information, or to (b) disambiguate the input, or to (c) confirm the input. The step502relates to one Prompt/Response pair for prompts that solicits input information for slots.

If the step504determines that remaining information is to be collected, then the sub-process500C follows a COLLECT arrow from the step504to a step505, in which the DM123collects remaining information. Here, the DM123identifies the slots to be filled, asks the grammar composer116to generate composite grammar corresponding to the unfilled slots, and composes the dialog for a current dialog cycle, including appropriate prompts, corresponding to the unfilled slots using the slot information132and the MI dialog information128. Thereafter, the sub-process500C is directed back to the step501.

Returning to the step504, if it is determined that the input is to be disambiguated, then the process500C is directed from the step504via a DISAMBIGUATE arrow to a step506in which the DM123disambiguates the input. As a part of disambiguation process, the DM123generates a disambiguation dialog using an appropriate disambiguation prompt and corresponding grammar, and presents it to the user. Thereafter, The DM123, based upon a user utterance received as a result of the disambiguation prompt, disambiguates the original input. Thereafter, the sub-process500C is directed from the step506back to the step502.

Returning to the step504, if the DM123determines that the input is to be confirmed, then the sub-process500C is directed from the step504via a CONFIRM arrow to a step507in which the DM123confirms the input. As a part of the confirmation process, the DM123generates the appropriate prompt playing back the values of different slots that were filled as a result of the previous user utterance and asks the user for confirmation of those values. The DM123receives a user response101in confirmation. In the case of a negative confirmation, the DM123interacts with the user by generating an appropriate error correction dialog120to rectify errors, until all the values in the original input are confirmed. The sub-process500C is then directed back to the step502.

Automatic Grammar Generation

FIG. 3is a flow chart showing operation of the grammar composer sub-process500A inFIG. 2. The “generate composite grammar step”501inFIG. 2, which is performed, as depicted by a dashed rectangle, by the grammar composer116ofFIG. 1, commences with a step201in which the grammar composer116inputs the atomic grammars108-110and the grammar composition rules component111. In an alternate arrangement, the grammar composer116can access the aforementioned components108-111on a per prompt/response pair basis.

In a following step203the grammar composer116receives, from the DM123, the state of the present dialog. The state describes the type of user interaction (input, disambiguation, confirmation) and the slots that would be the part of the next user interaction. Using this information of participating slots and interaction type, the grammar composer116, in the following step205, determines the atomic grammars that should be used for composing the composite grammar, for the next user interaction.

In the following step207, the grammar composer116creates the required composite MI grammar using the required atomic grammars108-110and the set of grammar composition rules111. The composition rules111specify grammar information for each slot that defines the slot grammar (the grammar applicable for the slot). The information includes the atomic grammar that forms the base of the slot grammar. The information of the atomic grammar is mentioned, for example, in the baseGrammar attribute of grammar tag in the set of grammar composition rules in APPENDIX C. The grammar composition rules set also contain grammar composition rules that define the rules to combine slot grammars to form a composite grammar for a sub set of slots. As noted, an example of grammar composition rules set with composition rules is shown in Appendix C. This embodiment of grammar composition rules set uses XML schema for specifying composition rules but system is not limited to the usage of XML and any other format such as tag based text, tokenized text, directed graph etc. can be used.

After composing the grammar, in a following step209, the grammar composer116returns the composite grammar to the DM123. It is noted that the step207constructs the composite grammar dependent upon the dialog state received in the step203. The dialog state depends upon previous utterance(s) by the user, as well as on the dialog flow information component127. The process501is then directed, according to an arrow210, from the step209back to the step203.

Dialog Process Management

FIG. 4is a flow chart showing operation of the second sub-process500B, and specifically the DM process503inFIG. 2. The process503, which is performed by the DM123is depicted by a dashed rectangle inFIG. 4. The process503commences with a step301in which the DM123receives, as depicted by an arrow119, an input reflecting a received user utterance from the VUI105inFIG. 1.

Thereafter, in a step302, the DM123augments the dialog history which it maintains for every dialog. The dialog history records the interaction states of various slots, including whether a slot has been filled or not, and if a slot has not been filled, then its state (ie no input received, OR the slot value need disambiguation, OR the slot value need confirmation etc).

Thereafter, in a step303, the DM123receives information130and124from the slot information components132and the dialog flow component127respectively. The dialog flow information124from the dialog flow component127is used to determine the type of next user interaction (ie input, disambiguation, or confirmation). An example of dialog flow information is given in Appendix D: “Collect First Strategy”, and another example of dialog flow information is show in Appendix E: “Confirm First Strategy”. The dialog flow information124along with the dialog history determines the slots that would be part of the next user interaction and the structure of the next dialog.

In a following step304, the DM123determines a dialog state (based on the dialog history) and information on the slots to be filled.

The process503then bifurcates into two concurrent strands referred to using reference numerals310and311.

In the strand310, in a first step305the DM123sends the dialog state (based on the dialog history) and information on the slots to be filled to the grammar composer116(see the step203inFIG. 3) thereby invoking the grammar composer116. Thereafter, in a step306, the DM123receives a composite grammar from the grammar composer116(see the step209inFIG. 3).

In the strand311, in a first step307the DM123refers to the slot information component132and the MI dialog configuration information component128and obtains the prompt and other events related information for the participating slots. The DM123uses this information, to generate prompt, help and other events information required for the dialog in a following step308.

Once both the strands310and311are completed, the process503is directed to a following step309in which the DM123sends the aforementioned information, generated in the strands310and311, to the VUI generator component121. The process503is then directed, in accordance with an arrow504, back to the step301.

Computer Hardware Platform

FIG. 5is a schematic block diagram of a general purpose computer upon which DMIDG arrangements can be practiced. The DMIDG method may be implemented using a computer system400, such as that shown inFIG. 5wherein the processes ofFIGS. 2,3and4may be implemented as software, such as one or more DMIDG application programs executable within the computer system400. In particular, the DMIDG method steps are performed by instructions in the software that are carried out within the computer system400. The instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the DMIDG methods and a second part and the corresponding code modules manage a user interface between the first part and the user.

The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system400from the computer readable medium, and then executed by the computer system400. A computer readable medium having such software or computer program recorded on it is a computer program product. The use of the computer program product in the computer system400preferably effects an advantageous DMIDG apparatus.

As seen inFIG. 5, the computer system400is formed by a computer module401, input devices such as a keyboard402, microphone425and a mouse pointer device403, and output devices including a printer415, a display device414and loudspeakers417. An external Modulator-Demodulator (Modem) transceiver device416may be used by the computer module401for communicating with a remote server426over a communications network420via a connection421. The network420may be a wide-area network (WAN), such as the Internet or a private WAN. Where the connection421is a telephone line, the modem416may be a traditional “dial-up” modem. Alternatively, where the connection421is a high capacity (eg: cable) connection, the modem416may be a broadband modem. A wireless modem may also be used for wireless connection to the network420.

The computer module401typically includes at least one processor unit405, and a memory unit406for example formed from semiconductor random access memory (RAM) and read only memory (ROM). The module401also includes an number of input/output (I/O) interfaces including an audio-video interface407that couples to the video display414, microphone425and loudspeakers417, an I/O interface413for the keyboard402and mouse403and optionally a joystick (not illustrated), and an interface408for the external modem416and printer415. In some implementations, the modem416may be incorporated within the computer module401, for example within the interface408.

The computer module401also has a local network interface411which, via a connection423, permits coupling of the computer system400to a local computer network422, known as a Local Area Network (LAN). As also illustrated, the local network422may also couple to the wide network420via a connection424, which would typically include a so-called “firewall” device or similar functionality. The interface411may be formed by an Ethernet™ circuit card, a wireless Bluetooth™ or an IEEE 802.21 wireless arrangement.

The interfaces408and413may afford both serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices409are provided and typically include a hard disk drive (HDD)410. Other devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive412is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (eg: CD-ROM, DVD), USB-RAM, and floppy disks for example may then be used as appropriate sources of data to the system400.

The components405, to413of the computer module401typically communicate via an interconnected bus404and in a manner which results in a conventional mode of operation of the computer system400known to those in the relevant art. Examples of computers on which the described arrangements can be practised include IBM-PC's and compatibles, Sun Sparcstations, Apple Mac™ or alike computer systems evolved therefrom.

Typically, the DMIDG application programs discussed above are resident on the hard disk drive410and read and controlled in execution by the processor405. Intermediate storage of such programs and any data fetched from the networks420and422may be accomplished using the semiconductor memory406, possibly in concert with the hard disk drive410. In some instances, the DMIDG application programs may be supplied to the user encoded on one or more CD-ROM (not shown) and read via the corresponding drive412, or alternatively may be read by the user from the remote server426over the networks420or422.

Still further, the software can also be loaded into the computer system400from other computer readable media. Computer readable media refers to any storage medium that participates in providing instructions and/or data to the computer system400for execution and/or processing. Examples of such media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module401. Examples of computer readable transmission media that may also participate in the provision of instructions and/or data include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

The second part of the DMIDG application programs and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs), such as the Voice User Interface105(seeFIG. 1), to be rendered or otherwise represented upon the display414. Through manipulation of the keyboard402and the mouse403, a user of the computer system400and the application may manipulate the interface to provide controlling commands and/or input to the applications associated with the GUI(s).

The DMIDG approach may alternatively be implemented in dedicated hardware such as one or more integrated circuits, including, for example, graphic processors, digital signal processors, or one or more microprocessors and associated memories.

Example

The disclosed DMIDG arrangement is now further described, using an example in which a speech application requires a user to provide their credit card information. This information consists of the credit card type, number and expiry date. Credit card expiry slot uses date atomic grammar as its base. The example involves a series of prompts by the system100(system prompts being represented as “S” in the following example), and corresponding responses by the user (user responses being represented by “U” in the following example):

S1: Please specify credit card information. Specify the credit card date in dd/mm/yyyy format, credit card number and card type.U1: Visa

S2: Specify the credit card expiry date in dd/mm/yyyy format and credit card number

U2: Help

S3: Please specify the date as March 2004, card number as sixteen digits number like 4437 2164 3289 9138.

S4: Did you say credit card with card number 5145 3478 1689 4762, expiry date June 2009 and card type Visa?

Appendices A-E relates to XML implementations of components used in the above example using an MI dialog according to the disclosed DMIDG approach. In particular, the text segment in Appendix A relates to Slot Information (see128inFIG. 1) for the example dialog. The text segment in Appendix B relates to Mixed Initiative Dialog configuration Information (see127inFIG. 1) for the example dialog. The text segment in Appendix C relates to the Grammar Composition Rules (see111inFIG. 1) for the example dialog. The text segment in Appendix D relates to an example of Mixed Initiative Dialog Flow Strategy named “Collect First Strategy” used by the dialog flow component127, and the text segment in Appendix E relates to an alternate “Confirm First Strategy” used by the dialog flow component127.

Appendix A describes the preferred embodiment of the slot information component, referred to as “SlotInformation”, (see128inFIG. 1) for the example CreditCardInfo dialog. “SlotInformation” defines all the slots/fields that are part of the MI dialog and also provides prompt and other configuration information for the slot. In the preferred embodiment, the example uses an XML structure for specifying the slot information; however other languages and structures may equally be used. Each atomic slot/field that forms an element in the example MI dialog is defined. Each slot is identified by a unique ID (i.e., the identification of the slot), and this ID is used to refer this element everywhere else in the system. Help, prompts, confirm, and other events are defined for each slot and this information is used in case the dialog falls back to the directed dialog mode soliciting input only for that particular slot.

In addition to the information associated with each slot, information for the composite Mixed Initiative Dialog (see127inFIG. 1) is presented in Appendix B as a separate construct “MIDialog”. This construct defines the prompts, help, confirm and other events that are specific for the MI dialog. The Mixed Initiative dialog configuration127is used to create the appropriate input prompts and other event messages for the MI Dialog. Accordingly, prompts are created depending on whether the particular slot has been filled or not. The above-noted example relates to a dialog having three components namely Credit card type, number and expiry date.

Appendix C presents an example of the Grammar Composition Rules Component (see111inFIG. 1), this being referred to as “grammarComposition”. The input to the component111is, in this example, an XML file, however other languages and data structures can be used.

The following text also relates to the above XML embodiment of the example of the MI dialog using the disclosed DMIDG approach. In particular, the following text segment relates to the Dialog Flow Information (see127inFIG. 1) for the example dialog. The dialog flow can proceed in one of the at least two following manners:

Collect First

In this dialog flow strategy, first the input is collected for all the fields that constitute the composite MI dialog. Once input for all the fields has been collected, a confirmation is made for input for all the fields in one interaction. If user response to confirmation is negative, the errors are rectified one by one for each field. The flow steps as specified are1. Collect all fields2. Confirm3. Rectify the erroneous components/slots.

Confirm First

In this dialog flow strategy, user input is asked for the set of fields that constitute composite MI dialog. The user response may fill only a subset of fields. Before soliciting user input for remaining fields, a confirmation is made for the fields that have already been filled by the previous user response. If there is an error it is rectified. Only when this subset of fields has been filled correctly, system solicits input information for the remaining fields. The flow is specified as sequence of following steps1. Collect MI slots2. Confirm the collected slots.3. Rectify the erroneous collected slots.4. Collect remaining sots5. Repeat steps 2, 3, and 4 till all the slots are filled and confirmed.

As noted above, Appendix D presents an XML specification of the “Collect First Strategy” used by the dialog flow component127, and Appendix E presents an XML specification of an alternate “Confirm First Strategy” used by the dialog flow component127.

The above two example strategies demonstrate how the same MI dialog can be configured to provide different flavour of dialog flows and user interaction. The above strategies are merely examples of inputs to the dialog flow component127, but are not restricted to it. The application developer who is using the MI dialog can define its own dialog flow strategy according to the application requirement using the dialog flow constructs. The example embodiment uses XML and an XML schema to define a dialog flow strategy but other schemas and languages can be used as well.

Conclusion

It will be apparent from the above that the arrangements described are applicable to the computer and data processing industries. The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

Thus, for example, the sub-process commencing with the step505inFIG. 2can, in an alternate embodiment, collect user utterances relating to all slots as an initial step, and then confirm and/or rectify information as appropriate, as depicted by the XML code for the “Collect First Strategy” in Appendix D.

The disclosed DMIDG arrangements ease development of mixed-initiative dialog systems, which can consequently be easily configured and ported for different applications. The disclosed approach allows speech application developer to dynamically enable a subset of slots among the original set of MI dialog slots for gathering user information. The subset of slots to be enabled can be chosen based on previous user response, dialog-interaction history, desired dialog flow or any other suitable parameter or a combination of such parameters. This empowers application developers to build very flexible MI dialogs. The disclosed approach provides a grammar-based dialog which typically provides improved accuracy of the speech recognition interface. The disclosed arrangements also provide a mechanism for generation of composite grammar automatically from individual atomic grammars. The automatic grammar composition mechanism of the disclosed MIDAGS approach can also be used in other scenarios (besides the MI dialog scenario) requiring a large number of composite grammars. An example scenario includes speech enabled applications where the grammar for subsequent dialog depends on the choice made at current dialog or input provided at current dialog. For example, frequent flier information application. It asks for frequent flier number and password. Using the frequent flier number, the application identifies the frequent flier category. Depending on the frequent flier category user can access different level of information. A basic category user can access information on seats availability, fare discount and meals whereas Gold user can also get information on Lounges and Priority Checkin. The composite grammar that is applicable to two users is different and composite grammar for Gold customer should have Lounge choices and Priority Checkin grammar elements in addition to grammar elements valid for basic user.

APPENDIX A

The following text segment relates to Slot Information (see132inFIG. 1) for the example dialog. The text segment defines all the slots/fields that are part of the mixed initiative dialog and provide relevant information about the slots. The example dialog described in the disclosure consists of three fields, credit card type, number and expiry date. Hence the slot information for the dialog has three slot elements characterizing three fields as depicted below.

<SlotInformation><!-Description of slot element.id=unique identity of the slot or field. This is used in rest of the application to referto this field including getting the value of the field.name (optional)= name that describes the slot.grammarid = reference to the grammar in grammarComposition file. The willrelate a component to a grammar.externalCompoent (optional) = If some external component is defined e.g. rdccomponent. The component should adhere/inherit some generic properties of thearchitecture.comp-config-path (optional)=path of an external file that defines the configurationof the slot. This is optional.It contains different prompts/help and confirm events.--><!-The first element characterizes the expiry date field of credit card informationdialog. All the attributes for the slot element has been shown here.--><slot id=“date” name=“Expiry Date” grammarid=“creditCardDate”externalComponent=“rdc.date” comp-config-path=“/relativePath/DateSlotConf.xml”><prompt no=“1”> Please tell me the expiry date. </prompt><prompt no=“2”> Please specify the expiry date. </prompt><help no=“1”> You can specify as march 2004, or march two zero zerofour, etc. </help><help no=“2”> For example, specify date as april 2005. No need to specifythe day. </help><confirm>Did you say <component id=“date” />?</confirm></slot><!--This element characterizes the credit card type field. The user does not want tospecify any name for the field and hence has not specified name attribute. There isno external dialog component for the field and hence the external-componentattribute is also not specified. This is valid as these attributes are optional.--><slot id=“cardType” grammarId=”creditCradType”comp-config-path=“/relativePath/TypeSlotConf.xml”><help> You can specify one of the Visa Card or Master Card as creditcard type. </help></slot><!--This element characterizes the credit card number field of the dialog. It has onlymandatory attributes and none of the optional attribute has been specified.--><slot id=“number” grammarId=”cardNumber”></slot></SlotInformation>

APPENDIX B

The following text segment relates to Mixed Initiative Dialog Configuration Information (see128inFIG. 1) for the example dialog.

<!-Description of MIDialog element.It encapsulates the information regarding MI dialog as a whole. It defines theprompts, help messages and confirm messages that will be spoken as a combinationof slots.The actual prompt or message that is generated depends on the fields that areintended to be collected from the user in the particular dialog turn. This is achievedusing the “slot id” tag used inside with prompt and messages. The text enclosed bythe “slot id” field will be part of TTS only is the filed identified by the “slot id” ispart of that dialog turn. The value “all” is special value for slot id. It signifies thatthe enclosing text would be used with all possible combinations of the input fields.--><MIDialog><prompt no=“1”><slot id=“all”>Please specify the credit card information.</slot>Specify the <slot id=“date”>expiry date in dd/mm/yyyy format </slot> ,<slot id=“number”> card number </slot> and <slot id=“cardType” > cardtype </slot></prompt><!--The dynamic prompt generation would be explained with reference to the aboveprompt for some example conditions.1.  In the initial dialog turn all the three fields are intended to be collected, hence allthe slot ids would be active and the prompt as specified in S1 in the exampledialog would be generated that contains all text segments specified in the aboveprompt tag.2.  In the first dialog turn user has specified a valid value for the credit card type.Hence the type field has been filled. Therefore in the subsequent dialog turnremaining two fields, expiry date and card number would be collected. Thus thetext enclosed by slot id “type” would not be part of the TTS text. The promptgenerate for second dialog turn would be as specified by S2 in example dialog.--><help no=“1”>Please Specify <help id=“date”>date as 2nd march 2004 </help> <helpid=“number”> card number as sixteen digit number like 4437 2164 32899138</help> <help id=“ cardType”> card type as one of visa, master card.</help></help><!--The processing of “slot id” field in confirm message is little different. The textsegment enclosed by the “slot id” tag will be included in the corresponding TTSonly if the field referred by the slot has been filled as part of the user utterance.The “value” tag is used to capture the value of the field specified by the slot idwhich becomes part of the confirmation message and is played back to the user.The confirm message that would be generated when all the slots have been filled isspecified by S4 in the example dialog.--><confirm>Did you say credit card with <slot id=“number”> card number <valueid=“number” /> </slot>, <slot id=“date”> expiry date <value id=”date”/> </slot>and<slot id=“ cardType”> card type <value id=” cardType”/> </slot></confirm></MIDialog>

APPENDIX C

The following text segment relates to the Grammar Composition Rules (see111inFIG. 1) for the example dialog.

<!-Atomic grammars are combined using the specified composition rules toform Composite grammar. In this example of composite grammar forcredit card information, the constituent atomic grammars are date, creditcard number and credit card type. The description of each grammar used isspecified in this file.→<grammarComposition id=”creditCardInfo”><composition><prefix>The credit card information is </prefix><prefix> Credit card </prefix><suffix> is the card detail </suffix></composition><!-Description of grammar element ‘Credit Card Date’id = unique identification of the grammar component.Name (optional) = name of grammarnoPrefixWhenSingle = (true or false) indicates if this is to be usedwithout prefixes.→<grammar id=”creditCardDate” baseGrammar=”path/date.grxml”noPrefixWhenSingle=”true”><prefix> expiry date </prefix><prefix> the expiry date is </prefix><prefix> Expiring on </prefix></grammar><!-Description of grammar element ‘Credit Card Number’→<grammar id=”creditCardNumber” baseGrammar=”path/number.grxml”noPrefixWhenSingle=”false”><prefix>number</prefix><prefix>as number</prefix><prefix>the number</prefix><suffix>is the number</suffix></grammar><!-Similar specifications for other required elements of the compositegrammar, eg. ‘Credit card type’→<!-Description of fixed-rule elementgrammars-invloved= list of grammars for which the fixed rules holdname = name of grammarnoPrefixWhenSingle = (true or false) indicates if this is to be usedwithout prefixes.→<fixed-rules><fixed-rule grammars-involved=”creditCardDate cardNumber”><rule><grammar-ref name=”cardNumber” /><conjunction> expiring on </conjunction><grammar-ref name=”creditCardDate” /></rule></fixed-rule><fixed-rule grammars-invlolved=”creditCardDate cardNumbercardType”><rule> <grammar-ref name=”cardNumberGrammar” /><conjunction optional=”true”> of type </conjunction><grammar-ref name=”cardTypeGrammar” /><conjunction > expiring on </conjunction><grammar-ref  name=”creditCardDateGrammar” /></rule></fixed-rule></fixed-rules></grammarComposition>

APPENDIX D

The following text segment relates to Dialog Flow (see127inFIG. 1) for the example dialog and defines the “Collect First Strategy”. The “Collect First Strategy” can be specified using the following XML code:

<!-Specifies a dialog flow strategy, “Collect First Strategy”MIFlow: The tag that defines a particular dialog flow strategyid= Unique id of the strategydesc (optional): description of the strategy--><MIflow id=“CollectFirst” desc=”Collect user input for all slots before furtherprocessing”><!--CollectFields: Defines the user input collection phase for various fieldscandidateFields: Defines the set of fields which are candidates for the collectionphasemethod=”All” means collect all the candidate fields first (before confirmation)--><CollectFields candidateFields=”all” method=“All” /><!-repeat: repeat the enclosed set of steps until the condition specified by the condattribute is met.cond - Specifies the repeat conditionallConfirmedTrue - Till all the collected elements are confirmed as ‘true’-- ><repeat cond=“allConfirmedTrue”><!-Confirm: Confirm from user that the collected input is correctcandidateFields: Specifies the set of input fields for which the confirmation has tobe donetype: defined the method or type of confirmation. It can be confirm each element(candidate Field) one by one only or confirm all the collected elements in One-go.RectifyErrors:If confirmation is not true, rectifies the error in input collectedmethod:defines the method of identifying the error and correcting it.‘oneByone’each input is checked for correctness one by one and in case of errorrectified one by one.--><Confirm candidateFields=”all” type=“all” /><RectifyErrors method=“oneByone” /></repeat></MIflow>

APPENDIX E

The following text segment also relates to Dialog Flow (see127inFIG. 1) for the example dialog and defines the “Confirm First Strategy”. The “Confirm First Strategy” can be specified using the following XML code: