Crowdsourced, grounded language for intent modeling in conversational interfaces

Different advantageous embodiments provide a crowdsourcing method for modeling user intent in conversational interfaces. One or more stimuli are presented to a plurality of describers. One or more sets of describer data are captured from the plurality of describers using a data collection mechanism. The one or more sets of describer data are processed to generate one or more models. Each of the one or more models is associated with a specific stimulus from the one or more stimuli.

BACKGROUND

Spoken conversational interfaces for computing devices have historically been hand-scripted. This involves anticipating a specific range of utterances that users might say, and mapping these anticipated utterances to specific states or actions in a machine. Similarly, dialog aimed at clarifying ambiguous input often needs to be hand-coded. For any new application, new hand-scripting is needed. In order to localize the functionality to any new language, new hand-scripting is also needed.

Verbally spoken, conversational data is complex, with nuances including relative terms, such as “make it bigger,” ambiguous descriptions, such as “that looks good,” and oblique expressions of the users' intent, such as “the labels are too noisy” or “can we make the chart look cleaner?” Currently, there are no general mechanisms for learning grounded natural language descriptions for a verbal interface. The verbal interfaces available are limited to specific domains with particular sets of recognized commands that have been hand-scripted, and of which a user needs specific knowledge in order to interact verbally within that domain. Also, users may lack the specific domain knowledge necessary to express their goals using terminology associated with that domain.

Existing techniques for constructing verbal interfaces present a number of drawbacks that adversely affect how much data can be explored, as well as the quality of the data and the accessibility to users. Accordingly, it is desirable to have grounding between natural language and machine state change in order to create a rich interaction for verbal interfaces. What is more, systems need to recognize a user's intent regardless of exactly how the user expresses that intent in natural language.

SUMMARY

Briefly, various aspects of the subject matter described herein are directed towards a system comprising a processor, a pre-processing component, and a conversational interface. The pre-processing component is implemented on the processor and configured to process data captured from a plurality of describers to generate a model. The conversational interface is configured to be trained using the model generated by the pre-processing component.

Another aspect is directed towards an apparatus comprising one or more stimuli, a data collection mechanism, and a model generation mechanism. The one or more stimuli are configured to be presented to a plurality of describers. The data collection mechanism is configured to capture describer data from the plurality of describers. The model generation mechanism is configured to process the describer data and generate a set of metadata documents.

Yet another aspect is directed towards a method for crowdsourcing conversational interfaces. One or more stimuli are presented to a plurality of describers. One or more sets of describer data are captured from the plurality of describers using a data collection mechanism. The one or more sets of describer data are processed to generate one or more models. Each of the one or more models is associated with a specific stimulus from the one or more stimuli.

DETAILED DESCRIPTION

Various aspects of the technology described herein are generally directed towards a system and method for training a conversational interface. As will be understood, crowdsourcing provides for obtaining a broad range of natural language input, which can be used to train a conversational interface for rich verbal interaction, capable of modeling user intent.

While the various aspects described herein are exemplified with a conversational interface environment directed towards manipulating and/or controlling computer applications, it will be readily appreciated that other environments and subjects may benefit from the technology described herein. For example, the various aspects described herein may be used to perform queries within a collection of data.

Thus, as will be understood, the technology described herein is not limited to any type of environment, domain, or topic for verbal interfaces. As such, the present invention is not limited to any particular embodiments, aspects, concepts, protocols, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, protocols, structures, functionalities or examples described herein are non-limiting, and the present invention may be used in various ways that provide benefits and advantages in verbal interfaces in general, as well as possibly other interfaces.

With reference now to the figures,FIG. 1is an illustration of a crowdsourcing conversational interfaces environment in which advantageous embodiments of the present invention may be implemented. The crowdsourcing conversational interfaces environment100may be implemented at least in part on at least one processor in a computing environment, for example. Stimuli102may be, for example, without limitation, any representation of a state or change of state, such as an action or event, which may be applied to a human sensory receptor. In this example, the stimuli102include at least one of an image104, a video clip106, an audio clip108, or a document110. The image104may be, for example, without limitation, a static image, a code-generated animated graphical image, an animated bitmap image format, and/or any other suitable type of image. A code-generated animated graphical image may include, for example, without limitation, animations using dynamic hyper-text markup language (HTML), Microsoft® Silverlight®, Adobe® Flash®, Oracle® JavaScript®, and/or any other suitable framework or language. The document110may or may not be tied to one or more of the image104, the video clip106, or the audio clip108.

For example, in one illustrative embodiment, the document110may include a set of questions that correspond to the image104. In one illustrative example, the image104may be an image of a hat, and the set of questions within the document110may include questions such as: “what is this?”, “what color is it?”, “how would you describe this?”, and the like. In another illustrative embodiment, the image104may be a Microsoft® Excel® spreadsheet, and the text within the document110may describe a task or set of tasks to be completed using the Microsoft® Excel® spreadsheet, for example. The stimuli102may be presented to a plurality of describers112in this illustrative example. As used herein, an image may be any representation of a code state that can be verbally characterized by humans, including, without limitation, a code state such as a spreadsheet table, a presentation slide, a SQL query, or a snippet of programming code. The stimulus may be realistic, such as a photograph, or abstract, such as skeletal images, and it may represent a single state or a change of state.

The grounded crowdsourcing conversational interfaces environment100may utilize the stimuli102to collect a broad range of natural language data from a plurality of describers112. The plurality of describers112, also recognized as a crowd, may be a network of people. The plurality of describers112may be an unspecified number and/or group of people with a common language. For example, in one illustrative implementation, the plurality of describers112may be an English-speaking crowd. In another illustrative implementation, the plurality of describers112may be a Spanish-speaking crowd, or any other suitable language, for example. Depending on the purposes of the task, the plurality of describers112may or may not have specific expertise in a domain. In many cases, it will be advantageous to collect descriptions from humans with varying degrees of expertise in the subject domain, to capture the full range of ways that people might express a particular intent. The plurality of describers112receives the stimuli102and produces describer data114, which is captured by a data collection mechanism116. The describer data114may be a set of natural language descriptions associated with a stimulus from the stimuli102presented to the plurality of describers112. The describer data114includes the natural language descriptions captured for each describer in the plurality of describers112.

While the illustrative example shows one set of the describer data114, different advantageous embodiments may provide multiple sets of describer data, where each set of describer data corresponds to a specific stimulus. Further, where each set of describer data is a collection of natural language descriptions that corresponds to one stimulus, such as an image, each collection of natural language descriptions includes the input from each describer in the crowd that received the image, for example. In other words, if a crowd of five hundred people received the image of a hat, the describer data captured that corresponds to the image of the hat would be a set of the natural language descriptions of the hat from each of the five hundred people in the crowd.

In one illustrative example, where an example stimulus is an image of an object, the describer data114may include natural language descriptions that describe the nature of the object, the physical attributes of the object, the aesthetics of the object, and/or any other suitable descriptions. In this example, where the example image is a hat, the natural language descriptions may include words or phrases such as: “a green hat,” “a striped green hat,” “a cap,” “a men's hat,” “a winter hat,” “a hat that keeps you warm,” “a green and yellow hat,” and so on. By capturing all these different ways of describing the same semantic intent—that hat—the technique allows linguistic flexibility in referring to this object. When this data is deployed in a conversational interface, users can now refer to “a warm green hat” even though this description has not previously been seen. This ability to model intent, regardless of how that intent might expressed, is a key advantage.

In one illustrative example, where an example stimulus represents a change of machine state, the describer data114may have been elicited in a manner to obtain descriptions of an intended action from different perspectives in order to obtain specific types of linguistic information grounded in the change of state. For example, instructions to the plurality of describers112may seek to elicit different commands given to a computer and more indirect negotiations to achieve the same change when given to a human coworker.

The grounded crowdsourcing conversational interfaces environment100includes a model generation mechanism118, a knowledge harvesting mechanism120, a knowledge repository122, and a clarification mechanism124. The model generation mechanism118uses the describer data114captured from the plurality of describers112to generate a model126used in training a conversational interface. The knowledge harvesting mechanism120interacts with the knowledge repository122to provide word associations that the model generation mechanism118may use to refine the describer data114when generating the model126. The knowledge repository may be any language repository, such as, without limitation, the Web, for example. The knowledge harvesting mechanism120may query the knowledge repository122with a word128to retrieve a set of word associations130corresponding to the word128. In an illustrative example, the word128may be “hair” and the set of word associations130returned for “hair” may include a number of words associated with “hair,” such as, without limitation, “brown,” “black,” “curly,” “straight,” “messy,” “cut,” “shaggy,” “long,” “short,” and/or any other suitable word that is associated with “hair” in natural language. This limited illustrative example demonstrates the vast range of natural language data that may be associated with just one single word, which may be one of many words included in the describer data114corresponding to a single stimulus, such as the image104, for example.

The model generation mechanism118uses the set of word associations130and/or one or more ranking algorithms132to process the describer data114. The processing of the describer data114using the set of word associations130and/or one or more ranking algorithms132may include filtering and/or clustering the describer data114to refine the data to a set of metadata documents134. The set of metadata documents134may be a collection of relevant words and/or phrases associated with the stimulus corresponding to the describer data114. For example, if the stimulus used is the image104, the set of metadata documents134is associated with the image104. Each metadata document in the set of metadata documents134may correspond to a specific describer in the plurality of describers112, in one illustrative implementation. Thus, in this example, each metadata document corresponds to a specific describer in the crowd, and a specific stimulus introduced to the crowd.

The clarification mechanism124includes a comparison component136that processes the set of metadata documents134to identify the commonalities and differences in the set of metadata documents134. The comparison component136identifies ambiguities, such as relative terms or descriptive words having more than one meaning. The clarification mechanism124uses the ambiguities identified by the comparison component136to generate a set of clarifying questions138. The set of clarifying questions138may be presented to the plurality of describers112. The plurality of describers112may respond to the set of clarifying questions138by producing clarifying data140, which is captured by the data collection mechanism116.

The model generation mechanism118uses the clarifying data140to further refine the set of metadata documents134and generate the model126. The model126is associated with a specific stimulus, such as the image104of the stimuli102, for example. Although only one model is shown in this illustrative implementation, the model generation mechanism118may generate a model for each stimulus presented to the plurality of describers112. The resulting collection of models may be used to train a conversational interface, providing the grounding in natural language needed to model user intent in a rich verbal interaction with the machine.

FIG. 1is intended as an example, and not as an architectural limitation for different embodiments. For example, in other advantageous embodiments, the crowdsourcing conversational interfaces environment100may have other stimuli presented to a plurality of describers. In yet other advantageous embodiments, the plurality of describers may include one or more sets of describers, where each set of describers includes a number of people with a common language. In this example, the model generation and subsequent training of a conversational interface may be provided for any language with a crowd available for crowdsourcing.

As used herein, when a first component is connected to a second component, the first component may be connected to the second component without any additional components. The first component also may be connected to the second component by one or more other components. For example, one electronic device may be connected to another electronic device without any additional electronic devices between the first electronic device and the second electronic device. In some cases, another electronic device may be present between the two electronic devices connected to each other.

The different advantageous embodiments recognize and take into account that current techniques for verbal interfaces have a number of drawbacks that adversely affect how much data can be explored, as well as the quality of the data and the accessibility to users. These drawbacks are due in part to the fact that current methods for verbal interfaces require hand-scripting, which are updated for each application or program desired for use with the verbal interface. Additionally, there is no grounding in natural language for the existing verbal interface technologies, which results in limited use and accessibility.

Thus, various aspects of the subject matter described herein are directed towards a system comprising a processor, a pre-processing component, and a conversational interface. The pre-processing component is implemented on the processor and configured to process data captured from a plurality of describers to generate a model. The conversational interface is configured to be trained using the model generated by the pre-processing component.

Another aspect is directed towards an apparatus comprising one or more stimuli, a data collection mechanism, and a model generation mechanism. The one or more stimuli are configured to be presented to a plurality of describers. The data collection mechanism is configured to capture describer data from the plurality of describers. The model generation mechanism is configured to process the describer data and generate a set of metadata documents.

Yet another aspect is directed towards a method for crowdsourcing conversational interfaces. One or more stimuli are presented to a plurality of describers. One or more sets of describer data are captured from the plurality of describers using a data collection mechanism. The one or more sets of describer data are processed to generate one or more models. Each of the one or more models is associated with a specific stimulus from the one or more stimuli.

With reference now toFIG. 2, an illustration of a conversational interface training environment is depicted in accordance with an advantageous embodiment. The conversational interface training environment200may be an illustrative example of one implementation of the crowdsourcing conversational interfaces environment100inFIG. 1.

Describer data202may be an illustrative example of one implementation of the describer data114inFIG. 1. The describer data202may be captured by the data collection mechanism116from the plurality of describers112inFIG. 1, for example. The describer data202may include one or more different types of descriptions captured from the plurality of describers112. For example, the describer data202may include a set of absolute descriptions204, a set of relative descriptions206, and/or any other suitable type of description. Some other types of descriptions captured may include, without limitation, multilingual descriptions, color descriptions, emotion descriptions, conceptual descriptions, and/or any other suitable type of description.

The pre-processing component208filters the describer data202to identify the set of absolute descriptions204and the set of relative descriptions206, and clusters together similar data. The pre-processing component208may be an illustrative implementation of the model generation mechanism118, the knowledge harvesting mechanism120, and the clarification mechanism124, for example. The pre-processing component208generates a model210, which is used along with a training algorithm212to train a conversational interface214.

With reference toFIG. 3, an illustration of a conversational interface environment300is depicted in accordance with an advantageous embodiment. The conversational interface environment200may be an illustrative example of a conversational interface trained in the crowdsourcing conversational interfaces environment100inFIG. 1.

Describer data302may be an illustrative example of one implementation of the describer data114inFIG. 1. The pre-processing component304filters the describer data302and clusters together similar data. The pre-processing component304may be an illustrative implementation of the model generation mechanism118, the knowledge harvesting mechanism120, and the clarification mechanism124, for example. The pre-processing component304generates a model306, which is used along with a training algorithm308to train a conversational interface310.

The conversational interface310receives verbal input312from a user, such as a person speaking conversationally to an interface, for example. The verbal input312may be a command, a query, and/or any other suitable input. The conversational interface310receives the verbal input312and generates a machine state change314. The machine state change314may be a specific machine behavior that corresponds to the verbal input312. For example, if the verbal input312is “I want to send an email to John Smith,” the machine state change314may be to open a messaging program, if it is not already open, and generate a new message template with the address of “John Smith” in the “send to” cell. In another illustrative example, where the conversational interface is a gaming interface using avatars, if the verbal input312is “put a blue hat on my avatar's head,” the machine state change314may be to render a representation of the avatar wearing a blue hat, for example.

In yet another illustrative embodiment, the verbal input312may be a query submitted to conversational interface310. The machine state change314in this example may be to perform the search to return a result, which may be presented as output316, in one illustrative example.

The different advantageous embodiments recognize and take into account that current techniques for verbal interfaces have a number of drawbacks that adversely affect how much data can be explored, as well as the quality of the data and the accessibility to users. These drawbacks are due in part to the fact that current methods for verbal interfaces require hand-scripting, which are updated for each application or program desired for use with the verbal interface. Additionally, there is no grounding in natural language for the existing verbal interface technologies, which results in limited use and accessibility.

Thus, the different advantageous embodiments provide a system and methods for a robust conversational interface grounded in natural language.

Exemplary Operating Environment

With reference toFIG. 4, an exemplary system for implementing various aspects of the invention may include a general purpose computing device in the form of a computer410. Components of the computer410may include, but are not limited to, a processing unit420, a system memory430, and a system bus421that couples various system components including the system memory to the processing unit420. The system bus421may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.

The system memory430includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)431and random access memory (RAM)432. A basic input/output system433(BIOS), containing the basic routines that help to transfer information between elements within computer410, such as during start-up, is typically stored in ROM431. RAM432typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit420. By way of example, and not limitation,FIG. 4illustrates operating system434, application programs435, other program modules436and program data437.

The computer410may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only,FIG. 4illustrates a hard disk drive441that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive451that reads from or writes to a removable, nonvolatile magnetic disk452, and an optical disk drive455that reads from or writes to a removable, nonvolatile optical disk456such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive441is typically connected to the system bus421through a non-removable memory interface such as interface440, and magnetic disk drive451and optical disk drive455are typically connected to the system bus421by a removable memory interface, such as interface450.

The drives and their associated computer storage media, described above and illustrated inFIG. 4, provide storage of computer-readable instructions, data structures, program modules and other data for the computer410. InFIG. 4, for example, hard disk drive441is illustrated as storing operating system444, application programs445, other program modules446and program data447. Note that these components can either be the same as or different from operating system434, application programs435, other program modules436, and program data437. Operating system444, application programs445, other program modules446, and program data447are given different numbers herein to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer410through input devices such as a tablet, or electronic digitizer,464, a microphone463, a keyboard462and pointing device461, commonly referred to as mouse, trackball or touch pad. Other input devices not shown inFIG. 4may include a joystick, game pad, satellite dish, scanner, touch screen, gesture recognition module, or the like. These and other input devices are often connected to the processing unit420through a user input interface460that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor491or other type of display device is also connected to the system bus421via an interface, such as a video interface490. The monitor491may also be integrated with a touch-screen panel or the like. Note that the monitor and/or touch screen panel can be physically coupled to a housing in which the computing device410is incorporated, such as in a tablet-type personal computer. In addition, computers such as the computing device410may also include other peripheral output devices such as speakers495and printer496, which may be connected through an output peripheral interface494or the like.

The computer410may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer480. The remote computer480may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer410, although only a memory storage device481has been illustrated inFIG. 4. The logical connections depicted inFIG. 4includeone or more local area networks (LAN)471and one or more wide area networks (WAN)473, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computer410is connected to the LAN471through a network interface or adapter470. When used in a WAN networking environment, the computer410typically includes a modem472or other means for establishing communications over the WAN473, such as the Internet. The modem472, which may be internal or external, may be connected to the system bus421via the user input interface460or other appropriate mechanism. A wireless networking component such as comprising an interface and antenna may be coupled through a suitable device such as an access point or peer computer to a WAN or LAN. In a networked environment, program modules depicted relative to the computer410, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,FIG. 4illustrates remote application programs485as residing on memory device481. It may be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

CONCLUSION