Abstract:
Designing a natural language understanding (NLU) model for an application from scratch can be difficult for non-experts. A system can simplify the design process by providing an interface allowing a designer to input example usage sentences and build an NLU model based on presented matches to those example sentences. In one embodiment, a method for initializing a workspace for building an NLU system includes parsing a sample sentence to select at least one candidate stub grammar from among multiple candidate stub grammars. The method can include presenting, to a user, respective representations of the candidate stub grammars selected by the parsing of the sample sentence. The method can include enabling the user to choose one of the respective representations of the candidate stub grammars. The method can include adding to the workspace a stub grammar corresponding to the representation of the candidate stub grammar chosen by the user.

Description:
RELATED APPLICATION 
       [0001]    This Application is related to “Method and Apparatus for Selecting Among Competing Models in a Tool for Building Natural Language Understanding Models” by Jeffrey N. Marcus, attorney docket number 4765.1034-000, filed on the same day as this Application to a common assignee. The foregoing application is hereby incorporated by reference in its entirety. 
       BACKGROUND 
       [0002]    When a designer wants to build a Natural Language Understanding (NLU) model for a new application (e.g., an airline reservation system), the designer starts a new project and configures that project with an ontology. The ontology is an an organizing structure of concepts that the NLU application handles. For example, a project may handle fields such as DATE, LOCATION, DATERANGE, or AMOUNT, etc. Typically, NLU models for many of these concepts are built-in within a designer portal and other more specialized NLU models (such as the airplane seating types for a particular airline) may need to be created by the designer. 
       SUMMARY 
       [0003]    Natural Language Understanding (NLU) applications often include many concepts. A non-expert may experience difficulty identifying built-ins (pre-made fields) to use in an NLU project. Online documentation can assist the designer; however, a more intuitive user interface would allow the designer to type in examples of expected sentences for the NLU application. For those examples, the designer can use an embodiment of the present invention to pre-populate the project with the matching built-ins. Additionally, the embodiment of the present invention identifies sentences that do not match any concepts well for which the designer needs to create the concepts. Since NLU application designers often start off by writing “sample dialogs” consisting of system-generated prompts followed by designer responses, discover-by-example is a natural way to configure a project. 
         [0004]    Discover-content-by-example includes a set of natural language (NL) models for interpreting concepts such as DATE, DATERANGE, AMOUNT, etc. The designer types in expected sentences from users using the NLU application to say within the dialog, perhaps with some context such as the general business type (travel, banking, etc.). The phrase is processed by all NL models included in a portal, and then each model returns a semantic confidence score, which can be arbitrarily defined. Based on a designer-defined threshold, zero or more models are identified as matching the sentence. If exactly one model is identified, it can be loaded into the project ontology. If more than one model is identified, the designer can choose one or more models from among them. If no models are identified, the system can display a message that no good match has been found, and the designer can create a new model. 
         [0005]    Further, the system can be extended to discover contents by sample system prompts (e.g., the prompts the system plays to the user to elicit his responses) instead of sample user responses. This may be even more powerful since the designer has control over prompts but has to guess at user responses. A database may be built from previously created or previously released deployments consisting of prompt text and associated responses by the user, with each user response precompiled into the concepts used to form it. When the designer types in a sample prompt from the corresponding application, the sample prompt is run against the database, and a “prompt match score” is assigned for each prompt in the database using some measure of semantic similarity. If there is a high match score, the set of concepts that are associated with that prompt are loaded into the NL Portal project. 
         [0006]    For example, a sample prompt may be “When would you like to travel?”. The database may have a prompt of “When are you travelling?” associated with the concepts DATE and TIME. This sentence is given a large semantic match score with the sentence in the database. A semantic match score is a score indicating the relatedness of two sentences. A high score indicates that two sentences are highly related and a low score indicates that the two sentences are not highly related. DATE and TIME is automatically loaded into the project ontology because of the semantic match score of the two sentences and the association with the database&#39;s prompt with the DATE and TIME concepts. 
         [0007]    In one embodiment, a method for initializing a workspace for building a Natural Language Understanding (NLU) system includes parsing a sample sentence to select at least one candidate stub grammar from among multiple candidate stub grammars. The term “stub grammar” is used herein, and in any continuing applications, to refer to a subset of a grammar, where a “grammar” is understood in the art to mean a model that correlates an input of a user to a response to the user. 
         [0008]    The method can further include presenting respective representations of the candidate stub grammars selected by the parsing of the sample sentence to a user. The method can additionally include enabling the user to choose one of the respective representation of the candidate stub grammars. The method can further include adding to the workspace a stub grammar corresponding to the representation of the candidate stub grammar chosen by the user. 
         [0009]    The respective representations can be names or identifiers of the candidate stub grammar or contents of the candidate stub grammar itself. 
         [0010]    If the prasing of the sample sentence selects one candidate stub grammar from among multiple candidate stub grammars, the method can skip presenting the a respective representation of the one candidate stub grammar to the user and also skip enabling the user to choose the one candidate stub grammar. The method can include directly adding the one candidate stub grammar to the workplace (e.g., without user input since there is only grammar to choose from). 
         [0011]    In another embodiment, the method can include presenting each respective representation of the candidate stub grammars to the user by visually presenting each of the respective representations of the candidate stub grammars relative to a corresponding part of the sample sentence. 
         [0012]    In yet a further embodiment, parsing the sample sentence includes parsing the sample sentence to generate a set of annotations including mentions. The annotations can further include mentions and intents. An intent refers to an end user&#39;s basic goal for a machine interaction. A mention refers to a piece of information the natural language application uses to complete the action indicated by the intent. 
         [0013]    In another embodiment, the presented respective representations can have a semantic match to the parsed sample sentence above a particular threshold. 
         [0014]    In another embodiment, a system for initializing a workspace for building a Natural Language Understanding (NLU) system can include a parsing module configured to parse a sample sentence to select at least one candidate stub grammar from among multiple candidate stub grammars. The system can further include a presentation module configured to present respective representations of the candidate stub grammars selected by the parsing of the sample sentence to the user. The system can further include a user interface module configured to enable the user to choose one of the respective representations of the candidate stub grammars. The system can additionally include a workspace building module configured to add a chosen grammar, chosen by the user from among the respective presented representations, to the workspace. 
         [0015]    In another embodiment, a non-transitory computer-readable medium can be configured to store instructions for initializing a workspace for building a Natural Language Understanding (NLU) system. The instructions, when loaded and executed by a processor, can cause the processor to parse a sample sentence to select at least one candidate stub grammar from among multiple candidate stub grammars. The instructions can further cause the processor to present respective representations of the at least one candidate stub grammar selected by the parsing of the sample sentence to a user. The instructions can additionally cause the processor to enable the user to choose one of the respective representation of the candidate stub grammars. The instructions can further cause the processor to add to the workspace a stub grammar corresponding to the representation of the candidate stub grammar chosen by the user. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0016]    The foregoing will be apparent from the following more particular description of example embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments of the present invention. 
           [0017]      FIG. 1  is a block diagram illustrating an example embodiment of the present invention. 
           [0018]      FIG. 2  is a block diagram illustrating an example embodiment of the present invention. 
           [0019]      FIG. 3  is a flow diagram illustrating an example process employed by the present invention. 
           [0020]      FIG. 4  is a screen view diagram illustrating an example embodiment of a user interface employed by the present invention. 
           [0021]      FIG. 5  is a network diagram illustrating a computer network or similar digital processing environment in which embodiments of the present invention may be implemented. 
           [0022]      FIG. 6  is a diagram of an example internal structure of a computer (e.g., client processor/device or server computers) in the computer system of  FIG. 5 . 
       
    
    
     DETAILED DESCRIPTION 
       [0023]    A description of example embodiments of the invention follows. 
         [0024]      FIG. 1  is a block diagram  100  illustrating an example embodiment of the present invention. A user  103  employs a user device  102  to input at least one sample sentence  104 . For example, in  FIG. 1 , the user  103  inputs the sentence “What restaurants are open after 8PM in Boston?” to the user device  102 . The user device  102  forwards the sample sentence  104  to a portal module  106 . The portal module  106  parses the sample sentence  104  and compares each part of the parsed sample sentence to a description of the candidate stub grammars (not shown) of a library (not shown). Stub grammars are a subset of a grammar, where a “grammar” is understood in the art to mean a model that correlates an input of a user to a response to the user. When a candidate stub grammar has a high semantic match to the sample sentence, in one embodiment, the portal module  106  forwards the stub grammar with the high suggested match to the user device  102 . The portal module  106  can also forward a group of suggested stub grammars  108  with high semantic matches to the sample sentence to the user device  102 . The user device  102  presents the suggested stub grammars to the user  103  and allows the user  103  to select one of the suggested stub grammars  108 . The user device forwards the selected stub grammar  110  to a project  112  for incorporation. 
         [0025]    The project  112  can be a workplace or model that is to be used for building a natural language understanding (NLU) system. If the portal module  106  finds only one stub grammar with a high semantic match to the sample sentence, the portal module  106  can forward the stub grammar  108  directly to the project  112  without asking the user  103  for input because the user  103  has no choice of which stub grammar to select. However, the portal module  106  can be configured to present the user with the choice of selecting the one stub grammar  108  or selecting no grammar at all 
         [0026]      FIG. 2  is a block diagram  200  illustrating an example embodiment of the present invention. A user device  202  forwards a sample sentence  204  to a matching module  206 . The user device  202 , in one embodiment, can be a console, tablet, personal computer, or other device. The matching module  206 , in one embodiment, can be a part of the portal module  106  of  FIG. 1 , the matching module  206  of  FIG. 2  configured to receive candidate stub grammars  216  of a library of stub grammars  214 . The matching module  206  parses the sample sentence  204  and determines which stub grammars of the forwarded stub candidate grammars  216  have a high semantic match to the sample sentence  204 . The matching model  206  forwards candidate stub grammar(s)  208  with a high semantic match to the user device  202  for presentation to the user. The presentation to the user can be presenting a representation of the candidate grammar, such as a name of the candidate grammar, to the user. The user of the user device  202  then selects a presented candidate stub grammar (e.g., selected stub grammar  210 ). The matching module then installs the selected stub grammar  210  in the project  212 . 
         [0027]      FIG. 3  is a flow diagram  300  illustrating an example process employed by the present invention. The system first inputs a sample sentence to the system, for example, from the user ( 302 ). Then, the system loads a candidate stub grammar from a library ( 304 ). The system then matches a sample sentence to the candidate stub grammar ( 306 ). The system then determines whether the matched candidate stub grammar has a high semantic match to the sample sentence ( 308 ). If the sentence does have a high semantic match to the candidate stub grammar, the system adds the stub grammar to a buffer memory ( 310 ). Then, the system determines whether more candidate stub grammars are in the library to be matched ( 312 ). If so, the system loads the next candidate stub grammar ( 314 ). Then, the system matches the sample sentence to the loaded candidate stub grammar ( 306 ). 
         [0028]    On the other hand, if the system does not determine that the candidate stub grammar has a high semantic match to the sample sentence, the system determines whether there are more candidate stub grammars to be matched ( 312 ). If so, the system loads the next candidate stub grammar ( 314 ), and if not, the system presents a representation of the candidate stub grammars in the buffer to the user for selection ( 316 ). The representation of the candidate stub grammars can be a name of the grammar, a description of the grammar, representation of fields of the stub grammar, or other representation of the grammar. 
         [0029]      FIG. 4  is a diagram  400  illustrating an example embodiment of a user interface  420  employed by the present invention. The user interface  420  can be displayed on a machine, such as a monitor, display, smart phone, tablet, mobile device, or other device. The user interface  420  displays a sample sentence  404  including parse words forwarded  6   a - c . The sample sentence  404 , in this example embodiment, is “what restaurants are open after 8 PM in Boston?” The user interface  420  further displays stub grammars  408   a - c  showing fields  410   a - f  associated with the first parsed word  406   a , second parsed word  406   b,  and third parsed word  406   c.  The stub grammars  408   a - b  can be shown as representations of the stub grammars  408   a - b , such as names of the stub grammars  408   a - b , fields of the stub grammars  408   a - b , or other representation. The fields  408   a - f  of the stub grammars  408   a - b are shown in locations corresponding to a part of the sample sentence  404  with which the stub grammar  408   a - f  is associated. 
         [0030]    For example, the user interface  420  presents a field  410   a  of the first stub grammar  408  of “destination” associated with a first parsed word  406   a  (e.g., “restaurants”) and a field  410   b  of the second stub grammar  408   b  that is blank associated with the first parsed word  406   a.  The user interface presents respective fields  410   c - d  of the first and second stub grammars  408   a - b associated with the second parsed word  406   b  of “8PM,” each field  410   c - d  having the value of “time.” The user interface  420  also presents respective fields  410   e - f  of the first and second stub grammars  408   a - b having the value of “place” associated with a third parsed word  406   c  of “Boston.” 
         [0031]    A display or other device, via the user interface  420 , presents the user with the choices of the first stub grammar  408   a  and second stub grammar  408   b.  The user can select, for instance using a mouse, touch screen, or other selection device, a stub grammar to use to begin the NLU project, workspace, or model. For example, the user can select the first stub grammar  408   a  because the fields  410   a,    410   c,  and  410   e  are close matches to the parsed words  406   a - c.    
         [0032]      FIG. 5  illustrates a computer network or similar digital processing environment in which embodiments of the present invention may be implemented. 
         [0033]    Client computer(s)/devices  50  and server computer(s)  60  provide processing, storage, and input/output devices executing application programs and the like. The client computer(s)/devices  50  can also be linked through communications network  70  to other computing devices, including other client devices/processes  50  and server computer(s)  60 . The communications network  70  can be part of a remote access network, a global network (e.g., the Internet), a worldwide collection of computers, local area or wide area networks, and gateways that currently use respective protocols (TCP/IP, Bluetooth®, etc.) to communicate with one another. Other electronic device/computer network architectures are suitable. 
         [0034]      FIG. 6  is a diagram of an example internal structure of a computer (e.g., client processor/device  50  or server computers  60 ) in the computer system of  FIG. 5 . Each computer  50 ,  60  contains a system bus  79 , where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. The system bus  79  is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, input/output ports, network ports, etc.) that enables the transfer of information between the elements. Attached to the system bus  79  is an I/O device interface  82  for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer  50 ,  60 . A network interface  86  allows the computer to connect to various other devices attached to a network (e.g., network  70  of  FIG. 5 ). Memory  90  provides volatile storage for computer software instructions  92  and data  94  used to implement an embodiment of the present invention (e.g., structure generation module, computation module, and combination module code detailed above). Disk storage  95  provides non-volatile storage for computer software instructions  92  and data  94  used to implement an embodiment of the present invention. A central processor unit  84  is also attached to the system bus  79  and provides for the execution of computer instructions. 
         [0035]    In one embodiment, the processor routines  92  and data  94  are a computer program product (generally referenced  92 ), including a non-transitory computer-readable medium (e.g., a removable storage medium such as one or more DVD-ROM&#39;s, CD-ROM&#39;s, diskettes, tapes, etc.) that provides at least a portion of the software instructions for the invention system. The computer program product  92  can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable communication and/or wireless connection. In other embodiments, the invention programs are a computer program propagated signal product embodied on a propagated signal on a propagation medium (e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network such as the Internet, or other network(s)). Such carrier medium or signals may be employed to provide at least a portion of the software instructions for the present invention routines/program  92 . 
         [0036]    In alternative embodiments, the propagated signal is an analog carrier wave or digital signal carried on the propagated medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a telecommunications network, or other network. In one embodiment, the propagated signal is a signal that is transmitted over the propagation medium over a period of time, such as the instructions for a software application sent in packets over a network over a period of milliseconds, seconds, minutes, or longer. 
         [0037]    While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.