Patent Description:
Cloud computing relates to the sharing of computing resources that are generally accessed via the Internet. In particular, a cloud computing infrastructure allows users, such as individuals and/or enterprises, to access a shared pool of computing resources, such as servers, storage devices, networks, applications, and/or other computing based services. By doing so, users are able to access computing resources on demand that are located at remote locations and these resources may be used to perform a variety computing functions (e.g., storing and/or processing large quantities of computing data). For enterprise and other organization users, cloud computing provides flexibility in accessing cloud computing resources without accruing large up-front costs, such as purchasing expensive network equipment or investing large amounts of time in establishing a private network infrastructure. Instead, by utilizing cloud computing resources, users are able redirect their resources to focus on their enterprise's core functions.

Such a cloud computing service may host a virtual agent, such as a chat agent, that is designed to automatically respond to issues with the client instance based on natural language requests from a user of the client instance. For example, a user may provide a request to a virtual agent for assistance with a password issue, wherein the virtual agent is part of a Natural Language Processing (NLP) or Natural Language Understanding (NLU) system. NLP is a general area of computer science and AI that involves some form of processing of natural language input. Examples of areas addressed by NLP include language translation, speech generation, parse tree extraction, part-of-speech identification, and others. NLU is a sub-area of NLP that specifically focuses on understanding user utterances. Examples of areas addressed by NLU include question-answering (e.g., reading comprehension questions), article summarization, and others. For example, a NLU may use algorithms to reduce human language (e.g., spoken or written) into a set of known symbols for consumption by a downstream virtual agent. NLP is generally used to interpret free text for further analysis. Current approaches to NLP are typically based on deep learning, which is a type of AI that examines and uses patterns in data to improve the understanding of a program.

Certain NLU systems are based on linguistic principles of cognitive constructive grammar. One of these principles is that the shape or form of an utterance is closely related to the meaning of the utterance. As such, it is presently recognized that it is important for NLU systems to be capable of extracting or determining multiple probable ways of understanding utterances to improve the operation of the NLU system. <CIT> is directed to systems and methods that enable an understanding of arbitrary natural language text through collaboration with humans. Text is ingested and then syntactically and semantically processed to infer an initial understanding of the text. The initial understanding is captured in a story model of semantic and frame structures. The story model is tested through computer generated questions that are posed to humans through interactive dialog sessions. The knowledge gleaned from the humans is used to update the story model as well as the computing system's current world model of understanding. <CIT> is directed to a speech recognition and natural language understanding system which performs insertion, deletion, and replacement edits of tokens at positions with low probabilities according to both a forward and a backward statistical language model to produce rewritten token sequences.

According to a first aspect of the present invention, there is provided the agent automation system of claim <NUM>. According to a second aspect of the present invention, there is provided the method of claim <NUM>. According to a third aspect of the present invention, there is provided the non-transitory computer-readable medium of claim <NUM>.

A summary of certain examples is set forth below. It should be understood that these examples are presented merely to provide the reader with a brief summary and that these examples are not intended to limit the scope of this disclosure. Indeed, this disclosure may encompass a variety of examples that may not be set forth below.

Present examples are directed to an agent automation framework that is capable of extracting meaning from user utterances, such as requests received by a virtual agent (e.g., a chat agent), and suitably responding to these user utterances. To do this, the agent automation framework includes a NLU framework and an intent-entity model having defined intents and entities that are associated with sample utterances. The NLU framework includes a meaning extraction subsystem that is designed to generate meaning representations for the sample utterances of the intent-entity model to construct an understanding model, as well as generate meaning representations for a received user utterance to construct an utterance meaning model. Additionally, the disclosed NLU framework includes a meaning search subsystem that is designed to search the meaning representations of the understanding model (also referred to as the search space) to locate matches for meaning representations of the utterance meaning model (also referred to as the search key). As such, present examples generally address the hard problem posed by NLU by transforming it into a manageable search problem.

Furthermore, it is presently recognized that the performance of the meaning search is enhanced by having multiple meaning representations of a user utterance in the utterance meaning model, as well as multiple meaning representations of the sample utterances in the understanding model, each representing a different understanding or interpretation of an underlying utterance, for comparison. Furthermore, while certain alternative meaning representations improve the likelihood of identifying matches during a meaning search, it is also presently recognized that the meaning search can also consume additional computing resources searching clearly erroneous or substantially redundant meaning representations. Therefore, it is recognized that certain, high-value alternative meaning representations of an utterance (e.g., high-probability and/or sufficiently different meaning representations) should be pursued to enhance the meaning search, while other alternative meaning representations should be discarded to limit computing resource usage and improve the efficiency of the NLU framework.

With the foregoing in mind, present examples are directed to a NLU framework that includes a meaning extraction subsystem capable of generating multiple meaning representations for utterances, including sample utterances in the intent-entity model and utterances received from a user. The disclosed meaning extraction subsystem includes a number of different components, which may be implemented as plug-ins for enhanced flexibility. These include: a part of speech (POS) plug in, a correction plug-in, a variation filter (VF) plug-in, a parser plug-in, and a final scoring and filtering (FSF) plug-in. For example, the POS plug-in may include a machine-learning (ML)-based component that receives an utterance and a POS threshold value, and generates a set of potential POS taggings for the utterance, along with corresponding confidence scores for these POS taggings, based on the POS threshold value. The correction plug-in receives the set of potential POS taggings and applies rules-based or ML-based techniques to modify or remove entries in the set of potential POS taggings. The VF plug-in may receive the set of potential POS taggings and a variation threshold value, and may remove certain entries from the set of potential POS taggings that are not sufficiently different from other entries based on the variation threshold value. The parser plug-in may include a ML-based or rules-based component that receives the set of potential POS taggings (e.g., after correction and/or variability filtering) and generates a respective meaning representation (e.g., an utterance tree) and corresponding confidence score for each entry in the set of potential POS taggings. The FSF plug-in may receive the set of meaning representations, the confidence scores from POS tagging and parsing, and a FSF threshold, and determine a final set of meaning representations for the utterance based on the FSF threshold.

As used herein, the term "computing system" or "computing device" refers to an electronic computing device such as, but not limited to, a single computer, virtual machine, virtual container, host, server, laptop, and/or mobile device, or to a plurality of electronic computing devices working together to perform the function described as being performed on or by the computing system. As used herein, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store one or more instructions or data structures. The term "non-transitory machine-readable medium" shall also be taken to include any tangible medium that is capable of storing, encoding, or carrying instructions for execution by the computing system and that cause the computing system to perform any one or more of the methodologies of the present subject matter, or that is capable of storing, encoding, or carrying data structures utilized by or associated with such instructions. The term "non-transitory machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media. Specific examples of non-transitory machine-readable media include, but are not limited to, non-volatile memory, including by way of example, semiconductor memory devices (e.g., Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and flash memory devices), magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

As used herein, the terms "application," "engine," and "plug-in" refer to one or more sets of computer software instructions (e.g., computer programs and/or scripts) executable by one or more processors of a computing system to provide particular functionality. Computer software instructions can be written in any suitable programming languages, such as C, C++, C#, Pascal, Fortran, Perl, MATLAB, SAS, SPSS, JavaScript, AJAX, and JAVA. Such computer software instructions can comprise an independent application with data input and data display modules. Alternatively, the disclosed computer software instructions can be classes that are instantiated as distributed objects. The disclosed computer software instructions can also be component software, for example JAVABEANS or ENTERPRISE JAVABEANS. Additionally, the disclosed applications or engines can be implemented in computer software, computer hardware, or a combination thereof.

As used herein, the term "framework" refers to a system of applications and/or engines, as well as any other supporting data structures, libraries, modules, and any other supporting functionality, that cooperate to perform one or more overall functions. In particular, a "natural language understanding framework" or "NLU framework" comprises a collection of computer programs designed to process and derive meaning (e.g., intents, entities, artifacts) from natural language utterances based on an understanding model. As used herein, a "behavior engine" or "BE," also known as a reasoning agent or RA/BE, refers to a rule-based agent, such as a virtual agent, designed to interact with users based on a conversation model. For example, a "virtual agent" may refer to a particular example of a BE that is designed to interact with users via natural language requests in a particular conversational or communication channel. With this in mind, the terms "virtual agent" and "BE" are used interchangeably herein. By way of specific example, a virtual agent may be or include a chat agent that interacts with users via natural language requests and responses in a chat room environment. Other examples of virtual agents may include an email agent, a forum agent, a ticketing agent, a telephone call agent, and so forth, which interact with users in the context of email, forum posts, autoreplies to service tickets, phone calls, and so forth.

As used herein, an "intent" refers to a desire or goal of a user which may relate to an underlying purpose of a communication, such as an utterance. As used herein, an "entity" refers to an object, subject, or some other parameterization of an intent. It is noted that, for present embodiments, certain entities are treated as parameters of a corresponding intent. More specifically, certain entities (e.g., time and location) may be globally recognized and extracted for all intents, while other entities are intent-specific (e.g., merchandise entities associated with purchase intents) and are generally extracted only when found within the intents that define them. As used herein, "artifact" collectively refers to both intents and entities of an utterance. As used herein, an "understanding model" is a collection of models used by the NLU framework to infer meaning of natural language utterances. An understanding model may include a vocabulary model that associates certain tokens (e.g., words or phrases) with particular word vectors, an intent-entity model, an entity model, or a combination thereof. As used herein an "intent-entity model" refers to a model that associates particular intents with particular sample utterances, wherein entities associated with the intent may be encoded as a parameter of the intent within the sample utterances of the model. As used herein, the term "agents" may refer to computer-generated personas (e.g. chat agents or other virtual agents) that interact with one another within a conversational channel. As used herein, a "corpus" refers to a captured body of source data that includes interactions between various users and virtual agents, wherein the interactions include communications or conversations within one or more suitable types of media (e.g., a help line, a chat room or message string, an email string). As used herein, an "utterance tree" refers to a data structure that stores a meaning representation of an utterance. As discussed, an utterance tree has a tree structure (e.g., a dependency parse tree structure) that represents the syntactic structure of the utterance, wherein nodes of the tree structure store vectors (e.g., word vectors, subtree vectors) that encode the semantic meaning of the utterance.

As used herein, "source data" or "conversation logs" may include any suitable captured interactions between various agents, including but not limited to, chat logs, email strings, documents, help documentation, frequently asked questions (FAQs), forum entries, items in support ticketing, recordings of help line calls, and so forth. As used herein, an "utterance" refers to a single natural language statement made by a user or agent that may include one or more intents. As such, an utterance may be part of a previously captured corpus of source data, and an utterance may also be a new statement received from a user as part of an interaction with a virtual agent. As used herein, "machine learning" or "ML" may be used to refer to any suitable statistical form of artificial intelligence capable of being trained using machine learning techniques, including supervised, unsupervised, and semi-supervised learning techniques. For example, in certain embodiments, ML-based techniques may be implemented using an artificial neural network (ANN) (e.g., a deep neural network (DNN), a recurrent neural network (RNN), a recursive neural network, a feedforward neural network). In contrast, "rules-based" methods and techniques refer to the use of rule-sets and ontologies (e.g., manually-crafted ontologies, statistically-derived ontologies) that enable precise adjudication of linguistic structure and semantic understanding to derive meaning representations from utterances. As used herein, a "vector" (e.g., a word vector, an intent vector, a subject vector, a subtree vector) refers to a linear algebra vector that is an ordered n-dimensional list (e.g., a <NUM> dimensional list) of floating point values (e.g., a 1xN or an Nx1 matrix) that provides a mathematical representation of the semantic meaning of a portion (e.g., a word or phrase, an intent, an entity, a token) of an utterance. As used herein, "domain specificity" refers to how attuned a system is to correctly extracting intents and entities expressed in actual conversations in a given domain and/or conversational channel. As used herein, an "understanding" of an utterance refers to an interpretation or a construction of the utterance by the NLU framework. As such, it may be appreciated that different understandings of an utterance may be associated with different meaning representations having different structures (e.g., different nodes, different relationships between nodes), different part-of-speech taggings, and so forth.

As mentioned, a computing platform may include a chat agent, or another similar virtual agent, that is designed to automatically respond to user requests to perform functions or address issues on the platform. When attempting to derive user intent from a natural language utterance, it is presently recognized that a single utterance can have different potential part-of-speech (POS) taggings for the tokens of the utterance, and that these different POS taggings represent different form-derivations of the utterance. As a consequence, a number of different meaning representations (e.g., utterance trees) can be generated for a single utterance, each representing a distinct form-derivation of the utterance. It is presently recognized that giving all possible form-derivations of the utterance due consideration improves intent inference abilities of a NLU framework. However, it is also presently recognized that certain form-derivations of the utterance may be erroneous or substantially redundant to other forms already being considered, which can substantially increase processing and memory resource consumption without substantially improving the intent inference abilities of the NLU framework.

Accordingly, present embodiments are directed to a NLU framework that includes a meaning extraction subsystem that applies ML-based and rules-based techniques to generate multiple meaning representations for a natural language utterance. The meaning extraction subsystem includes a structure subsystem capable of generating multiple meaning representations of sample utterances of an understanding model to expand the search space and/or capable of generating multiple meaning representations of a received user utterance to expand the search key. The disclosed structure subsystem uses ML-based techniques to generate multiple potential part-of-speech (POS) taggings for the utterance, wherein only potential POS taggings having corresponding confidence scores greater than a predefined threshold value are advanced. The disclosed structure subsystem applies rule-based and/or ML-based correction techniques to modify or eliminate erroneous potential POS taggings. The disclosed structure subsystem also applies a variability filter. The disclosed structure subsystem may apply the variability filter to eliminate potential POS taggings that are not sufficiently different from one another based on a predefined variation threshold value. After correction and/or variability filtering, the disclosed structure subsystem uses ML-based or rule-based techniques to generate a respective meaning representation (e g. , an utterance tree) for each remaining potential POS tagging, wherein only meaning representations having corresponding confidence scores greater than a predefined threshold are advanced. Finally, the disclosed structure subsystem applies a final scoring and filtering step that considers the confidence scores of the advanced meaning representations, as well as the underlying confidence scores of the corresponding POS taggings, to generate a final score, and removes meaning representations having final scores below a predefined threshold. As such, the disclosed structure subsystem effectively expands the number of form-derivations that are generated for the search key and/or search space of the meaning search, improving the intent inference capabilities of the NLU framework, while eliminating erroneous or substantially redundant form-derivations to reduce resource consumption and improve efficiency of the NLU framework.

With the preceding in mind, the following figures relate to various types of generalized system architectures or configurations that may be employed to provide services to an organization in a multi-instance framework and on which the present approaches may be employed. Correspondingly, these system and platform examples may also relate to systems and platforms on which the techniques discussed herein may be implemented or otherwise utilized. Turning now to <FIG>, a schematic diagram of an embodiment of a cloud computing system <NUM> where embodiments of the present disclosure may operate, is illustrated. The cloud computing system <NUM> may include a client network <NUM>, a network <NUM> (e.g., the Internet), and a cloud-based platform <NUM>. In some implementations, the cloud-based platform <NUM> may be a configuration management database (CMDB) platform. In one embodiment, the client network <NUM> may be a local private network, such as local area network (LAN) having a variety of network devices that include, but are not limited to, switches, servers, and routers. In another embodiment, the client network <NUM> represents an enterprise network that could include one or more LANs, virtual networks, data centers <NUM>, and/or other remote networks. As shown in <FIG>, the client network <NUM> is able to connect to one or more client devices 14A, 14B, and 14C so that the client devices are able to communicate with each other and/or with the network hosting the platform <NUM>. The client devices <NUM> may be computing systems and/or other types of computing devices generally referred to as Internet of Things (IoT) devices that access cloud computing services, for example, via a web browser application or via an edge device <NUM> that may act as a gateway between the client devices <NUM> and the platform <NUM>. <FIG> also illustrates that the client network <NUM> includes an administration or managerial device, agent, or server, such as a management, instrumentation, and discovery (MID) server <NUM> that facilitates communication of data between the network hosting the platform <NUM>, other external applications, data sources, and services, and the client network <NUM>. Although not specifically illustrated in <FIG>, the client network <NUM> may also include a connecting network device (e.g., a gateway or router) or a combination of devices that implement a customer firewall or intrusion protection system.

For the illustrated embodiment, <FIG> illustrates that client network <NUM> is coupled to a network <NUM>. The network <NUM> may include one or more computing networks, such as other LANs, wide area networks (WAN), the Internet, and/or other remote networks, to transfer data between the client devices 14A-C and the network hosting the platform <NUM>. Each of the computing networks within network <NUM> may contain wired and/or wireless programmable devices that operate in the electrical and/or optical domain. For example, network <NUM> may include wireless networks, such as cellular networks (e.g., Global System for Mobile Communications (GSM) based cellular network), IEEE <NUM> networks, and/or other suitable radio-based networks. The network <NUM> may also employ any number of network communication protocols, such as Transmission Control Protocol (TCP) and Internet Protocol (IP). Although not explicitly shown in <FIG>, network <NUM> may include a variety of network devices, such as servers, routers, network switches, and/or other network hardware devices configured to transport data over the network <NUM>.

In <FIG>, the network hosting the platform <NUM> may be a remote network (e.g., a cloud network) that is able to communicate with the client devices <NUM> via the client network <NUM> and network <NUM>. The network hosting the platform <NUM> provides additional computing resources to the client devices <NUM> and/or the client network <NUM>. For example, by utilizing the network hosting the platform <NUM>, users of the client devices <NUM> are able to build and execute applications for various enterprise, IT, and/or other organization-related functions. In one embodiment, the network hosting the platform <NUM> is implemented on the one or more data centers <NUM>, where each data center could correspond to a different geographic location. Each of the data centers <NUM> includes a plurality of virtual servers <NUM> (also referred to herein as application nodes, application servers, virtual server instances, application instances, or application server instances), where each virtual server <NUM> can be implemented on a physical computing system, such as a single electronic computing device (e.g., a single physical hardware server) or across multiple-computing devices (e.g., multiple physical hardware servers). Examples of virtual servers <NUM> include, but are not limited to a web server (e.g., a unitary Apache installation), an application server (e.g., unitary JAVA Virtual Machine), and/or a database server (e.g., a unitary relational database management system (RDBMS) catalog).

To utilize computing resources within the platform <NUM>, network operators may choose to configure the data centers <NUM> using a variety of computing infrastructures. In one embodiment, one or more of the data centers <NUM> are configured using a multi-tenant cloud architecture, such that one of the server instances <NUM> handles requests from and serves multiple customers. Data centers <NUM> with multi-tenant cloud architecture commingle and store data from multiple customers, where multiple customer instances are assigned to one of the virtual servers <NUM>. In a multi-tenant cloud architecture, the particular virtual server <NUM> distinguishes between and segregates data and other information of the various customers. For example, a multi-tenant cloud architecture could assign a particular identifier for each customer in order to identify and segregate the data from each customer. Generally, implementing a multi-tenant cloud architecture may suffer from various drawbacks, such as a failure of a particular one of the server instances <NUM> causing outages for all customers allocated to the particular server instance.

In another embodiment, one or more of the data centers <NUM> are configured using a multi-instance cloud architecture to provide every customer its own unique customer instance or instances. For example, a multi-instance cloud architecture could provide each customer instance with its own dedicated application server and dedicated database server. In other examples, the multi-instance cloud architecture could deploy a single physical or virtual server <NUM> and/or other combinations of physical and/or virtual servers <NUM>, such as one or more dedicated web servers, one or more dedicated application servers, and one or more database servers, for each customer instance. In a multi-instance cloud architecture, multiple customer instances could be installed on one or more respective hardware servers, where each customer instance is allocated certain portions of the physical server resources, such as computing memory, storage, and processing power. By doing so, each customer instance has its own unique software stack that provides the benefit of data isolation, relatively less downtime for customers to access the platform <NUM>, and customer-driven upgrade schedules. An example of implementing a customer instance within a multi-instance cloud architecture will be discussed in more detail below with reference to <FIG>.

<FIG> is a schematic diagram of an embodiment of a multi-instance cloud architecture <NUM> where embodiments of the present disclosure may operate. <FIG> illustrates that the multi-instance cloud architecture <NUM> includes the client network <NUM> and the network <NUM> that connect to two (e.g., paired) data centers 22A and 22B that may be geographically separated from one another. Using <FIG> as an example, network environment and service provider cloud infrastructure client instance <NUM> (also referred to herein as a client instance <NUM>) is associated with (e.g., supported and enabled by) dedicated virtual servers (e.g., virtual servers 24A, 24B, 24C, and 24D) and dedicated database servers (e.g., virtual database servers 44A and 44B). Stated another way, the virtual servers 24A-24D and virtual database servers 44A and 44B are not shared with other client instances and are specific to the respective client instance <NUM>. In the depicted example, to facilitate availability of the client instance <NUM>, the virtual servers 24A-24D and virtual database servers 44A and 44B are allocated to two different data centers 22A and 22B so that one of the data centers <NUM> acts as a backup data center. Other embodiments of the multi-instance cloud architecture <NUM> could include other types of dedicated virtual servers, such as a web server. For example, the client instance <NUM> could be associated with (e.g., supported and enabled by) the dedicated virtual servers 24A-24D, dedicated virtual database servers 44A and 44B, and additional dedicated virtual web servers (not shown in <FIG>).

Although <FIG> and <FIG> illustrate specific embodiments of a cloud computing system <NUM> and a multi-instance cloud architecture <NUM>, respectively, the disclosure is not limited to the specific embodiments illustrated in <FIG> and <FIG>. For instance, although <FIG> illustrates that the platform <NUM> is implemented using data centers, other embodiments of the platform <NUM> are not limited to data centers and can utilize other types of remote network infrastructures. Moreover, other embodiments of the present disclosure may combine one or more different virtual servers into a single virtual server or, conversely, perform operations attributed to a single virtual server using multiple virtual servers. For instance, using <FIG> as an example, the virtual servers 24A, 24B, 24C, 24D and virtual database servers 44A, 44B may be combined into a single virtual server. Moreover, the present approaches may be implemented in other architectures or configurations, including, but not limited to, multi-tenant architectures, generalized client/server implementations, and/or even on a single physical processor-based device configured to perform some or all of the operations discussed herein. Similarly, though virtual servers or machines may be referenced to facilitate discussion of an implementation, physical servers may instead be employed as appropriate. The use and discussion of <FIG> and <FIG> are only examples to facilitate ease of description and explanation and are not intended to limit the disclosure to the specific examples illustrated therein.

As may be appreciated, the respective architectures and frameworks discussed with respect to <FIG> and <FIG> incorporate computing systems of various types (e.g., servers, workstations, client devices, laptops, tablet computers, cellular telephones, and so forth) throughout. For the sake of completeness, a brief, high level overview of components typically found in such systems is provided. As may be appreciated, the present overview is intended to merely provide a high-level, generalized view of components typical in such computing systems and should not be viewed as limiting in terms of components discussed or omitted from discussion.

By way of background, it may be appreciated that the present approach may be implemented using one or more processor-based systems such as shown in <FIG>. Likewise, applications and/or databases utilized in the present approach may be stored, employed, and/or maintained on such processor-based systems. As may be appreciated, such systems as shown in <FIG> may be present in a distributed computing environment, a networked environment, or other multi-computer platform or architecture. Likewise, systems such as that shown in <FIG>, may be used in supporting or communicating with one or more virtual environments or computational instances on which the present approach may be implemented.

With this in mind, an example computer system may include some or all of the computer components depicted in <FIG> generally illustrates a block diagram of example components of a computing system <NUM> and their potential interconnections or communication paths, such as along one or more busses. As illustrated, the computing system <NUM> may include various hardware components such as, but not limited to, one or more processors <NUM>, one or more busses <NUM>, memory <NUM>, input devices <NUM>, a power source <NUM>, a network interface <NUM>, a user interface <NUM>, and/or other computer components useful in performing the functions described herein.

The one or more processors <NUM> may include one or more microprocessors capable of performing instructions stored in the memory <NUM>. Additionally or alternatively, the one or more processors <NUM> may include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and/or other devices designed to perform some or all of the functions discussed herein without calling instructions from the memory <NUM>.

With respect to other components, the one or more busses <NUM> include suitable electrical channels to provide data and/or power between the various components of the computing system <NUM>. The memory <NUM> may include any tangible, non-transitory, and computer-readable storage media. Although shown as a single block in <FIG>, the memory <NUM> can be implemented using multiple physical units of the same or different types in one or more physical locations. The input devices <NUM> correspond to structures to input data and/or commands to the one or more processors <NUM>. For example, the input devices <NUM> may include a mouse, touchpad, touchscreen, keyboard and the like. The power source <NUM> can be any suitable source for power of the various components of the computing device <NUM>, such as line power and/or a battery source. The network interface <NUM> includes one or more transceivers capable of communicating with other devices over one or more networks (e.g., a communication channel). The network interface <NUM> may provide a wired network interface or a wireless network interface. A user interface <NUM> may include a display that is configured to display text or images transferred to it from the one or more processors <NUM>. In addition and/or alternative to the display, the user interface <NUM> may include other devices for interfacing with a user, such as lights (e.g., LEDs), speakers, and the like.

It should be appreciated that the cloud-based platform <NUM> discussed above provides an example of an architecture that may utilize NLU technologies. In particular, the cloud-based platform <NUM> may include or store a large corpus of source data that can be mined, to facilitate the generation of a number of outputs, including an intent-entity model. For example, the cloud-based platform <NUM> may include ticketing source data having requests for changes or repairs to particular systems, dialog between the requester and a service technician or an administrator attempting to address an issue, a description of how the ticket was eventually resolved, and so forth. Then, the generated intent-entity model can serve as a basis for classifying intents in future requests, and can be used to generate and improve a conversational model to support a virtual agent that can automatically address future issues within the cloud-based platform <NUM> based on natural language requests from users. As such, in certain embodiments described herein, the disclosed agent automation framework is incorporated into the cloud-based platform <NUM>, while in other embodiments, the agent automation framework may be hosted and executed (separately from the cloud-based platform <NUM>) by a suitable system that is communicatively coupled to the cloud-based platform <NUM> to process utterances, as discussed below.

With the foregoing in mind, <FIG> illustrates an agent automation framework <NUM> (also referred to herein as an agent automation system <NUM>) associated with a client instance <NUM>. More specifically, <FIG> illustrates an example of a portion of a service provider cloud infrastructure, including the cloud-based platform <NUM> discussed above. The cloud-based platform <NUM> is connected to a client device 14D via the network <NUM> to provide a user interface to network applications executing within the client instance <NUM> (e.g., via a web browser of the client device 14D). Client instance <NUM> is supported by virtual servers similar to those explained with respect to <FIG>, and is illustrated here to show support for the disclosed functionality described herein within the client instance <NUM>. The cloud provider infrastructure is generally configured to support a plurality of end-user devices, such as client device 14D, concurrently, wherein each end-user device is in communication with the single client instance <NUM>. Also, the cloud provider infrastructure may be configured to support any number of client instances, such as client instance <NUM>, concurrently, with each of the instances in communication with one or more end-user devices. As mentioned above, an end-user may also interface with client instance <NUM> using an application that is executed within a web browser.

The embodiment of the agent automation framework <NUM> illustrated in <FIG> includes a behavior engine (BE) <NUM>, an NLU framework <NUM>, and a database <NUM>, which are communicatively coupled within the client instance <NUM>. The BE <NUM> may host or include any suitable number of virtual agents or personas that interact with the user of the client device 14D via natural language user requests <NUM> (also referred to herein as user utterances <NUM> or utterances <NUM>) and agent responses <NUM> (also referred to herein as agent utterances <NUM>). It may be noted that, in actual implementations, the agent automation framework <NUM> may include a number of other suitable components, including the meaning extraction subsystem, the meaning search subsystem, and so forth, in accordance with the present disclosure.

For the embodiment illustrated in <FIG>, the database <NUM> may be a database server instance (e.g., database server instance 44A or 44B, as discussed with respect to <FIG>), or a collection of database server instances. The illustrated database <NUM> stores an intent-entity model <NUM>, a conversation model <NUM>, a corpus of utterances <NUM>, and a collection of rules <NUM> in one or more tables (e.g., relational database tables) of the database <NUM>. The intent-entity model <NUM> stores associations or relationships between particular intents and particular entities via particular sample utterances. In certain embodiments, the intent-entity model <NUM> may be authored by a designer using a suitable authoring tool. In other embodiments, the agent automation framework <NUM> generates the intent-entity model <NUM> from the corpus of utterances <NUM> and the collection of rules <NUM> stored in one or more tables of the database <NUM>. The intent-entity model <NUM> may also be determined based on a combination of authored and ML techniques, in some embodiments. In any case, it should be understood that the disclosed intent-entity model <NUM> may associate any suitable combination of intents and/or entities with respective ones of the corpus of utterances <NUM>. For embodiments discussed below, sample utterances of the intent-entity model <NUM> are used to generate meaning representations of an understanding model to define the search space for a meaning search.

For the embodiment illustrated in <FIG>, the conversation model <NUM> stores associations between intents of the intent-entity model <NUM> and particular responses and/or actions, which generally define the behavior of the BE <NUM>. In certain embodiments, at least a portion of the associations within the conversation model are manually created or predefined by a designer of the BE <NUM> based on how the designer wants the BE <NUM> to respond to particular identified artifacts in processed utterances. It should be noted that, in different embodiments, the database <NUM> may include other database tables storing other information related to intent classification, such as a tables storing information regarding compilation model template data (e.g., class compatibility rules, class-level scoring coefficients, tree-model comparison algorithms, tree substructure vectorization algorithms), meaning representations, and so forth.

For the illustrated embodiment, the NLU framework <NUM> includes an NLU engine <NUM> and a vocabulary manager <NUM>. It may be appreciated that the NLU framework <NUM> may include any suitable number of other components. In certain embodiments, the NLU engine <NUM> is designed to perform a number of functions of the NLU framework <NUM>, including generating word vectors (e.g., intent vectors, subject or entity vectors, subtree vectors) from word or phrases of utterances, as well as determining distances (e.g., Euclidean distances) between these vectors. For example, the NLU engine <NUM> is generally capable of producing a respective intent vector for each intent of an analyzed utterance. As such, a similarity measure or distance between two different utterances can be calculated using the respective intent vectors produced by the NLU engine <NUM> for the two intents, wherein the similarity measure provides an indication of similarity in meaning between the two intents.

The vocabulary manager <NUM> (which may be part of the vocabulary subsystem discussed below) addresses out-of-vocabulary words and symbols that were not encountered by the NLU framework <NUM> during vocabulary training. For example, in certain embodiments, the vocabulary manager <NUM> can identify and replace synonyms and domain-specific meanings of words and acronyms within utterances analyzed by the agent automation framework <NUM> (e.g., based on the collection of rules <NUM>), which can improve the performance of the NLU framework <NUM> to properly identify intents and entities within context-specific utterances. Additionally, to accommodate the tendency of natural language to adopt new usages for pre-existing words, in certain embodiments, the vocabulary manager <NUM> handles repurposing of words previously associated with other intents or entities based on a change in context. For example, the vocabulary manager <NUM> could handle a situation in which, in the context of utterances from a particular client instance and/or conversation channel, the word "bike" actually refers to a motorcycle rather than a bicycle.

Once the intent-entity model <NUM> and the conversation model <NUM> have been created, the agent automation framework <NUM> is designed to receive a user utterance <NUM> (in the form of a natural language request) and to appropriately take action to address the request. For example, for the embodiment illustrated in <FIG>, the BE <NUM> is a virtual agent that receives, via the network <NUM>, the utterance <NUM> (e.g., a natural language request in a chat communication) submitted by the client device 14D disposed on the client network <NUM>. The BE <NUM> provides the utterance <NUM> to the NLU framework <NUM>, and the NLU engine <NUM>, along with the various subsystems of the NLU framework discussed below, processes the utterance <NUM> based on the intent-entity model <NUM> to derive artifacts (e.g., intents and/or entities) within the utterance. Based on the artifacts derived by the NLU engine <NUM>, as well as the associations within the conversation model <NUM>, the BE <NUM> performs one or more particular predefined actions. For the illustrated embodiment, the BE <NUM> also provides a response <NUM> (e.g., a virtual agent utterance <NUM> or confirmation) to the client device 14D via the network <NUM>, for example, indicating actions performed by the BE <NUM> in response to the received user utterance <NUM>. Additionally, in certain embodiments, the utterance <NUM> may be added to the utterances <NUM> stored in the database <NUM> for continued learning within the NLU framework <NUM>.

It may be appreciated that, in other embodiments, one or more components of the agent automation framework <NUM> and/or the NLU framework <NUM> may be otherwise arranged, situated, or hosted for improved performance. For example, in certain embodiments, one or more portions of the NLU framework <NUM> may be hosted by an instance (e.g., a shared instance, an enterprise instance) that is separate from, and communicatively coupled to, the client instance <NUM>. It is presently recognized that such embodiments can advantageously reduce the size of the client instance <NUM>, improving the efficiency of the cloud-based platform <NUM>. In particular, in certain embodiments, one or more components of the similarity scoring subsystem discussed below may be hosted by a separate instance (e.g., an enterprise instance) that is communicatively coupled to the client instance <NUM>, as well as other client instances, to enable improved meaning searching for suitable matching meaning representations within the search space to enable identification of artifact matches for the utterance <NUM>.

With the foregoing in mind, <FIG> illustrates an alternative embodiment of the agent automation framework <NUM> in which portions of the NLU framework <NUM> are instead executed by a separate, shared instance (e.g., enterprise instance <NUM>) that is hosted by the cloud-based platform <NUM>. The illustrated enterprise instance <NUM> is communicatively coupled to exchange data related to artifact mining and classification with any suitable number of client instances via a suitable protocol (e.g., via suitable Representational State Transfer (REST) requests/responses). As such, for the design illustrated in <FIG>, by hosting a portion of the NLU framework as a shared resource accessible to multiple client instances <NUM>, the size of the client instance <NUM> can be substantially reduced (e.g., compared to the embodiment of the agent automation framework <NUM> illustrated in <FIG>) and the overall efficiency of the agent automation framework <NUM> can be improved.

In particular, the NLU framework <NUM> illustrated in <FIG> is divided into three distinct components that perform distinct processes within the NLU framework <NUM>. These components include: a shared NLU trainer <NUM> hosted by the enterprise instance <NUM>, a shared NLU annotator <NUM> hosted by the enterprise instance <NUM>, and an NLU predictor <NUM> hosted by the client instance <NUM>. It may be appreciated that the organizations illustrated in <FIG> and <FIG> are merely examples, and in other embodiments, other organizations of the NLU framework <NUM> and/or the agent automation framework <NUM> may be used, in accordance with the present disclosure.

For the embodiment of the agent automation framework <NUM> illustrated in <FIG>, the shared NLU trainer <NUM> is designed to receive the corpus of utterances <NUM> from the client instance <NUM>, and to perform semantic mining (e.g., including semantic parsing, grammar engineering, and so forth) to facilitate generation of the intent-entity model <NUM>. Once the intent-entity model <NUM> has been generated, when the BE <NUM> receives the user utterance <NUM> provided by the client device 14D, the NLU predictor <NUM> passes the utterance <NUM> and the intent-entity model <NUM> to the shared NLU annotator <NUM> for parsing and annotation of the utterance <NUM>. The shared NLU annotator <NUM> performs semantic parsing, grammar engineering, and so forth, of the utterance <NUM> based on the intent-entity model <NUM> and returns meaning representations of the utterance <NUM> to the NLU predictor <NUM> of client instance <NUM>. The NLU predictor <NUM> then uses these annotated structures of the utterance <NUM>, discussed below in greater detail, to identify matching intents from the intent-entity model <NUM>, such that the BE <NUM> can perform one or more actions based on the identified intents. It may be appreciated that the shared NLU annotator <NUM> may correspond to the meaning extraction subsystem <NUM>, and the NLU predictor may correspond to the meaning search subsystem <NUM>, of the NLU framework <NUM>, as discussed below.

<FIG> is a flow diagram depicting a process <NUM> by which the behavior engine (BE) <NUM> and NLU framework <NUM> perform respective roles within an embodiment of the agent automation framework <NUM>. For the illustrated embodiment, the NLU framework <NUM> processes a received user utterance <NUM> to extract artifacts <NUM> (e.g., intents and/or entities) based on the intent-entity model <NUM>. The extracted artifacts <NUM> may be implemented as a collection of symbols that represent intents and entities of the user utterance <NUM> in a form that is consumable by the BE <NUM>. As such, these extracted artifacts <NUM> are provided to the BE <NUM>, which processes the received artifacts <NUM> based on the conversation model <NUM> to determine suitable actions <NUM> (e.g., changing a password, creating a record, purchasing an item, closing an account) and/or virtual agent utterances <NUM> in response to the received user utterance <NUM>. As indicated by the arrow <NUM>, the process <NUM> can continuously repeat as the agent automation framework <NUM> receives and addresses additional user utterances <NUM> from the same user and/or other users in a conversational format.

As illustrated in <FIG>, it may be appreciated that, in certain situations, no further action or communications may occur once the suitable actions <NUM> have been performed. Additionally, it should be noted that, while the user utterance <NUM> and the agent utterance <NUM> are discussed herein as being conveyed using a written conversational medium or channel (e.g., chat, email, ticketing system, text messages, forum posts), in other embodiments, voice-to-text and/or text-to-voice modules or plugins could be included to translate spoken user utterance <NUM> into text and/or translate text-based agent utterance <NUM> into speech to enable a voice interactive system, in accordance with the present disclosure. Furthermore, in certain embodiments, both the user utterance <NUM> and the virtual agent utterance <NUM> may be stored in the database <NUM> (e.g., in the corpus of utterances <NUM>) to enable continued learning of new structure and vocabulary within the agent automation framework <NUM>.

As mentioned, the NLU framework <NUM> includes two primary subsystems that cooperate to convert the hard problem of NLU into a manageable search problem - namely: a meaning extraction subsystem and a meaning search subsystem. For example, <FIG> is a block diagram illustrating roles of the meaning extraction subsystem <NUM> and the meaning search subsystem <NUM> of the NLU framework <NUM> within an embodiment of the agent automation framework <NUM>. For the illustrated embodiment, a right-hand portion <NUM> of <FIG> illustrates the meaning extraction subsystem <NUM> of the NLU framework <NUM> receiving the intent-entity model <NUM>, which includes sample utterances <NUM> for each of the various artifacts of the model. The meaning extraction subsystem <NUM> generates an understanding model <NUM> that includes meaning representations <NUM> (e.g., parse tree structures) of the sample utterances <NUM> of the intent-entity model <NUM>. In other words, the understanding model <NUM> is a translated or augmented version of the intent-entity model <NUM> that includes meaning representations <NUM> to enable searching (e.g., comparison and matching) by the meaning search subsystem <NUM>, as discussed in more detail below. As such, it may be appreciated that the right-hand portion <NUM> of <FIG> is generally performed in advance of receiving the user utterance <NUM>, such as on a routine, scheduled basis or in response to updates to the intent-entity model <NUM>.

For the embodiment illustrated in <FIG>, a left-hand portion <NUM> illustrates the meaning extraction subsystem <NUM> also receiving and processing the user utterance <NUM> to generate an utterance meaning model <NUM> having at least one meaning representation <NUM>. As discussed in greater detail below, these meaning representations <NUM> and <NUM> are data structures having a form that captures the grammatical, syntactic structure of an utterance, wherein subtrees of the data structures include subtree vectors that encode the semantic meanings of portions of the utterance. As such, for a given utterance, a corresponding meaning representation captures both syntactic and semantic meaning in a common meaning representation format that enables searching, comparison, and matching by the meaning search subsystem <NUM>, as discussed in greater detail below. Accordingly, the meaning representations <NUM> of the utterance meaning model <NUM> can be generally thought of like a search key, while the meaning representations <NUM> of the understanding model <NUM> define a search space in which the search key can be sought. Thus, the meaning search subsystem <NUM> searches the meaning representations <NUM> of the understanding model <NUM> to locate one or more artifacts that match the meaning representation <NUM> of the utterance meaning model <NUM> as discussed below, thereby generating the extracted artifacts <NUM>.

The meaning extraction subsystem of <FIG> itself uses a number of subsystems of the NLU framework <NUM> that cooperate to generate the meaning representations <NUM> and <NUM>. For example, <FIG> is a block diagram illustrating an embodiment of the meaning extraction subsystem <NUM> of the NLU framework <NUM> of the agent automation framework <NUM>. The illustrated embodiment of the meaning extraction subsystem <NUM> uses a rules-based methods interleaved with ML-based methods to generate an annotated utterance tree <NUM> for an utterance <NUM>, which may be either a user utterance <NUM> or one of the sample utterances <NUM> of the intent-entity model <NUM>, as discussed above with respect to <FIG>. More specifically, <FIG> illustrates how embodiments of the meaning extraction subsystem <NUM> can utilize a number of best-of- breed models, including combinations of rule-based and ML-based (e.g., statistical) models and programs, that can be plugged into the overall NLU framework <NUM>. For example, because of the pluggable design of the illustrated meaning extraction subsystem <NUM>, the vocabulary subsystem <NUM> can include any suitable word vector distribution model that defines word vectors for various words or phrases. That is, since it is recognized that different word distribution models can excel over others in a given conversational channel, language, context, and so forth, the disclosed pluggable design enables the meaning extraction subsystem <NUM> to be customized to particular environments and applications. For the embodiment illustrated in <FIG>, the meaning extraction subsystem <NUM> uses three plug-in-supported subsystems of the NLU framework <NUM>, namely a vocabulary subsystem <NUM>, a structure subsystem <NUM>, and a prosody subsystem <NUM>, and the various outputs of these subsystems are combined according to the stored rules <NUM> to generate the utterance tree <NUM> from the utterance <NUM>.

For the embodiment of the meaning extraction subsystem <NUM> illustrated in <FIG>, the vocabulary subsystem <NUM> generally handles the vocabulary of the meaning extraction subsystem <NUM>. As such, the illustrated meaning extraction subsystem <NUM> includes a number of vocabulary plug-ins <NUM> that enable analysis and extraction of the vocabulary of utterances. For the illustrated embodiment, the vocabulary plug-ins <NUM> include a learned multimodal word vector distribution model <NUM>, a learned unimodal word vector distribution model <NUM>, and any other suitable word vector distribution models <NUM>. In this context, "unimodal" refers to word vector distribution models having a single respective vector for each word, while "multimodal" refers to word vector distribution models supporting multiple vectors for particular words (e.g., homonyms, polysemes) that can have different meanings in different contexts (e.g., a "bank" may refer to a place to store money, money itself, a maneuver of an aircraft, or a location near a river). The models <NUM>, <NUM>, and <NUM> provide pluggable collections of word vectors that can be selected based on suitable parameters, such as language, conversation style, conversational channel, and so forth.

For example, the learned multimodal distribution model <NUM> and the learned unimodal distribution model <NUM> can provide word distributions (e.g., defined vector spaces of word vectors) that are generated using unsupervised learning or other general clustering algorithms, as discussed below with respect to <FIG>. That is, appreciating that words commonly used in close proximity within utterances often have related meanings, the learned multimodal distribution model <NUM> and learned unimodal distribution model <NUM> can be generated by performing statistical analysis of utterances (e.g., from the corpus of utterances <NUM>), and then defining vectors for words based on how the word is commonly used with respect to other words within these utterances. As such, these vocabulary plug-ins <NUM> enable the vocabulary subsystem <NUM> to recognize and address synonyms, misspelled words, encoded symbols (e.g., web addresses, network paths, emoticons, and emojis), out-of-vocabulary terms, and so forth, when processing the user utterance <NUM> and sample utterances <NUM>. In certain embodiments, the vocabulary subsystem <NUM> can combine or select from word vectors output by the various vocabulary plug-ins <NUM> based the stored rules <NUM> to generate word vectors for nodes of the utterance tree <NUM>, as discussed below. Moreover, the word vector distribution models <NUM>, <NUM>, and/or <NUM> can be continually updated based on unsupervised learning performed on received user utterances <NUM>, as discussed below with respect to <FIG>.

For the embodiment illustrated in <FIG>, the structure subsystem <NUM> of the meaning extraction subsystem <NUM> analyzes a linguistic shape of the utterance <NUM> using a combination of rule-based and ML-based structure parsing plug-ins <NUM>. In other words, the illustrated structure plug-ins <NUM> enable analysis and extraction of the syntactic and grammatical structure of the utterances <NUM> and <NUM>. For the illustrated embodiment, the structure plug-ins <NUM> include rule-based parsers <NUM>, ML-based parsers <NUM> (e.g., DNN-based parsers, RNN-based parsers, and so forth), and other suitable parser models <NUM>. For example, one or more of these structure plug-ins <NUM> enables class annotations or tagging (e.g., as a verb, a subject or entity, a direct object, a modifier, and so forth) for each word or phrase of the utterance. In certain embodiments, the structure subsystem <NUM> can combine or select from parse structures output by the various structure plug-ins <NUM> based on one or more rules <NUM> stored in the database <NUM>, which are used to define the structure or shape of the utterance trees <NUM>, as discussed below.

For the embodiment illustrated in <FIG>, the prosody subsystem <NUM> of the meaning extraction subsystem <NUM> analyzes the prosody of the utterance <NUM> using a combination of rule-based and ML-based prosody plug-ins <NUM>. The illustrated prosody plug-ins <NUM> include rule-based prosody systems <NUM>, ML-based prosody systems <NUM>, and other suitable prosody systems <NUM>. Using these plug-ins, the prosody subsystem <NUM> analyzes the utterance <NUM> for prosodic cues, including written prosodic cues such as rhythm (e.g., chat rhythm, such as utterance bursts, segmentations indicated by punctuation or pauses), emphasis (e.g., capitalization, bolding, underlining, asterisks), focus or attention (e.g., repetition of particular terms or styles), and so forth, which can be used to determine, for example, boundaries between intents, degrees of urgency or relative importance with respect to different intents, and so forth. As such, in certain embodiments, the prosody subsystem <NUM> can combine or select from prosody parsed structures output by the various prosody plug-ins <NUM> based on the rules <NUM> stored in the database <NUM> to generate the utterance tree <NUM>, as discussed below.

As such, for the embodiment of the meaning extraction subsystem <NUM> illustrated in <FIG>, the vocabulary subsystem <NUM>, the structure subsystem <NUM>, and the prosody subsystem <NUM> of the NLU framework <NUM> cooperate to generate the utterance tree <NUM> from the utterance <NUM> based on one or more rules <NUM>. It may be appreciated that, in certain embodiments, a portion of the output of one subsystem (e.g., the prosody subsystem <NUM>) may be provided as input to another subsystem (e.g., the structure subsystem <NUM>) when generating the utterance tree <NUM> from the utterance <NUM>. The resulting utterance tree <NUM> data structure generated by the meaning extraction subsystem <NUM> includes a number of nodes, each associated with a respective word vector provided by the vocabulary subsystem <NUM>. Furthermore, these nodes are arranged and coupled together to form a tree structure based on the output of the structure subsystem <NUM> and the prosody subsystem <NUM>, according to the stored rules <NUM>.

For example, <FIG> is a diagram illustrating an example of an utterance tree <NUM> generated for an utterance <NUM>, in accordance with an embodiment of the present approach. As mentioned, the utterance tree <NUM> is a data structure that is generated by the meaning extraction subsystem <NUM> based on the utterance <NUM>. In certain embodiments, the meaning representations <NUM> of the utterance meaning model <NUM> and the meaning representations <NUM> of the understanding model <NUM> are (or are derived from) utterance trees, while in other embodiments, other parse structures can be used. For the example illustrated in <FIG>, the utterance tree <NUM> is based on an example utterance, "I want to go to the store by the mall today to buy a blue, collared shirt and black pants and also to return some defective batteries. " The illustrated utterance tree <NUM> includes a set of nodes <NUM> (e.g., nodes 202A, 202B, 202C, 202D, 202E, 202F, <NUM>, <NUM>, 202I, 202J, <NUM>, <NUM>, <NUM>, 202N, and 202P) arranged in a tree structure, each node representing a particular word or phrase of the utterance <NUM>. It may be noted that each of the nodes <NUM> may also be described as representing a particular subtree of the utterance tree <NUM>, wherein a subtree can include one or more nodes <NUM>.

As mentioned, the form or shape of the utterance tree <NUM> illustrated in <FIG> is determined by the prosody subsystem <NUM> and the structure subsystem <NUM> and represents the syntactic, grammatical meaning of the example utterance. More specifically, the prosody subsystem <NUM> breaks the utterance into intent segments, while the structure subsystem <NUM> constructs the utterance tree <NUM> from these intent segments. Each of the nodes <NUM> store or reference a respective word vector that is determined by the vocabulary subsystem <NUM> to indicate the semantic meaning of the particular word or phase of the utterance. As mentioned, each word vector is an ordered n-dimensional list (e.g., a <NUM> dimensional list) of floating point values (e.g., a 1xN or an Nx1 matrix) that provides a mathematical representation of the semantic meaning of a portion of an utterance.

Moreover, each of the nodes <NUM> is annotated by the structure subsystem <NUM> with additional information about the word or phrase represented by the node. For example, in <FIG>, each of the nodes <NUM> has a class or part-of-speech (POS) annotation. In particular, for the example utterance tree illustrated in <FIG>, certain subtrees or nodes (e.g., nodes 202A, 202B, 202C, and 202D) are annotated to be verb nodes, and certain subtrees or nodes (e.g., nodes 202E, 202F, <NUM>, <NUM>, 202I, and 202J) are annotated to be subject or object nodes, and certain subtrees or nodes (e.g., nodes <NUM>, <NUM>, <NUM>, 202N, and 202P) are annotated to be modifier nodes (e.g., subject modifier nodes, object modifier nodes, verb modifier nodes) by the structure subsystem <NUM>. These class annotations are used by the meaning search subsystem <NUM> when comparing meaning representations that are generated from utterance trees, like the example utterance tree <NUM> illustrated in <FIG>. As such, it may be appreciated that the utterance tree <NUM>, from which the meaning representations are generated, serves as a basis (e.g., an initial basis) for intent and/or entity extraction.

Referring briefly to <FIG>, as noted, performance of the meaning search by the NLU framework <NUM> can be improved by enabling the NLU framework to derive and compare multiple, alternative forms of received user utterances <NUM> to multiple, alternative forms of the sample utterances <NUM> of the intent-entity model <NUM>. For example, in certain embodiments, during the meaning search performed by the meaning search subsystem <NUM>, a search space may be defined that includes meaning representations <NUM> of the understanding model <NUM>, and potentially meaning representations of additional understanding models. Additionally, in certain embodiments, this search space may be iteratively pruned based on the CCG form of the user utterance <NUM> being search, wherein the CCG form is captured in the structure of the meaning representations <NUM> of the utterance meaning model <NUM>. In particular, the multiple, alternative meaning representations derived from a user utterance <NUM> can be used to determine what other comparable meaning representations <NUM> exist in the search space, wherein non-comparable meaning representations can be eliminated from consideration to improve search latency and over-all computing resource usage. As such, having multiple, alternative meaning representations <NUM> of the user utterance <NUM> with different CCG forms in the utterance meaning model <NUM> can result in dramatically different pruning of the search space during the meaning search, enabling a more comprehensive meaning search of the search space.

Furthermore, when the meaning search subsystem <NUM> is comparing meaning representations <NUM> of the utterance meaning model <NUM> to meaning representations <NUM> of the understanding model <NUM>, having multiple, alternative forms of the meaning representations <NUM> and <NUM> can also be advantageous. For example, in certain embodiments, a CCG form class database (e.g., part of the database <NUM>) may store a number of different CCG forms (e.g., a verb-led CCG form, a noun-led CCG form). Each CCG form is associated with a collection of mathematical functions that enable the meaning search subsystem <NUM> to calculate a similarity score between meaning representations <NUM> of the utterance meaning model <NUM> having the CCG form and meaning representations <NUM> of the understanding model <NUM> having the same CCG form (or a different, comparable CCG form). Additionally, in certain embodiments, the meaning search subsystem <NUM> may iteratively compare an expanding number of nodes of two comparable meaning representations using these mathematical functions, and the order in which the nodes are considered is also dependent on the form or shape of the meaning representations. As such, having multiple, alternative meaning representations <NUM> with different CCG forms in the utterance meaning model <NUM> enables different comparisons to different meaning representations <NUM> of the understanding model <NUM> using different mathematical functions, which also enables a more comprehensive meaning search.

With the foregoing in mind, <FIG> is a flow diagram illustrating an embodiment of a process <NUM> by which the meaning extraction subsystem <NUM> of the NLU framework <NUM> generates re-expressions of an original utterance <NUM>, and then generates a set <NUM> of meaning representations based on these re-expressions and the original utterance <NUM>. It may be appreciated that, in certain cases, the original utterance <NUM> may be a received user utterance <NUM>, and the resulting set <NUM> of meaning representations may become meaning representations <NUM> of the utterance meaning model <NUM>. In other cases, the original utterance <NUM> may be one of the sample utterances <NUM> of the intent-entity model <NUM>. For the embodiment illustrated in <FIG>, a portion of the process <NUM> is performed by the vocabulary subsystem <NUM> of the meaning extraction subsystem <NUM> of the NLU framework <NUM>, while another portion is performed by the structure subsystem <NUM> of the meaning extraction subsystem <NUM>. In other embodiments, the steps of the process <NUM> may be performed by other suitable components of the NLU framework <NUM>. Additionally, the process <NUM> may be stored in a suitable memory (e.g., memory <NUM>) and executed by a suitable processor (e.g., processor(s) <NUM>) associated with the client instance <NUM> or the enterprise instance <NUM>, as discussed above with respect to <FIG>, <FIG>, and <FIG>.

For the embodiment illustrated in <FIG>, the process <NUM> begins with the vocabulary subsystem <NUM> of the NLU framework <NUM> cleansing (block <NUM>) the original utterance <NUM>. For example, the vocabulary subsystem <NUM> may access and apply rules <NUM> stored in the database <NUM> to modify certain tokens (e.g., words, phrases, punctuation, emojis) of the utterance. For example, in certain embodiments, cleansing may involve applying a rule that removes non-textual elements (e.g., emoticons, emojis, punctuation) from the original utterance <NUM>. In certain embodiments, cleansing may involve correcting misspellings or typographical errors in the utterance. Additionally, in certain embodiments, cleansing may involve substituting certain tokens with other tokens. For example, the vocabulary subsystem <NUM> may apply a rule that that all entities with references to time or color with a generic or global entity (e.g., "TIME", "COLOR").

For the embodiment illustrated in <FIG>, the process <NUM> continues with the vocabulary subsystem <NUM> performing vocabulary injection (block <NUM>) on the original utterance <NUM>. Vocabulary injection generally involves introducing multiple re-renderings of the original utterance <NUM>. For instance, phraseology and/or terminology may be replaced with more discourse-appropriate phrases and/or terms as dictated by a vocabulary model <NUM> of the understanding model <NUM>. In certain embodiments, multiple phrases and/or terms may be replaced, and the various permutations of such replacements are used to generate a set <NUM> of utterances based on the original utterance <NUM>. For example, in certain embodiments, the vocabulary subsystem <NUM> may access the vocabulary model <NUM> of the understanding model <NUM> to identify alternative vocabulary that can be used to generate re-expressions of the utterances having different tokens. By way of specific example, in an embodiment, the vocabulary subsystem <NUM> may determine that a synonym for "developer" is "employee," and may generate a new utterance in which the term "developer" is substituted by the term "employee.

For the embodiment illustrated in <FIG>, after cleansing and vocabulary injection, the set <NUM> of utterances is provided to the structure subsystem <NUM> for POS tagging and parsing. It may be appreciated that the set <NUM> of utterances may include the original utterance <NUM> or a cleansed version of the original utterance, and may include any suitable number of alternative re-expressions utterances generated through the vocabulary injection of block <NUM>. It may be noted that, in certain circumstances, the vocabulary injection of block <NUM> may not generate re-expressions of the original utterance <NUM>, and as such, the set <NUM> of utterances may only include the original utterance <NUM>, or a cleansed version thereof. In other embodiments, the original utterance <NUM> may be provided directly to the structure subsystem <NUM> without the cleansing of block <NUM> or the vocabulary injection of block <NUM>.

Upon receipt of the set <NUM> of utterances, the structure subsystem <NUM> uses a set of plug-ins <NUM> to generate (block <NUM>) the set <NUM> of one or more meaning representations that are representative of the original utterance <NUM>. In other embodiments, the set of plug-ins <NUM> may instead be implemented as non-pluggable applications or modules of the meaning extraction subsystem <NUM> or the NLU framework <NUM>. However, it is presently recognized that the disclosed pluggable design of the illustrated structure subsystem <NUM> enables the NLU framework <NUM> to have greater flexibility. For example, support for additional languages can be added to the NLU framework <NUM> by switching the set of plug-ins <NUM> to another set designed (e.g., programmed, trained) for a different language or a different domain.

For the embodiment illustrated in <FIG>, the set of plug-ins <NUM> of the structure subsystem <NUM> include: a part of speech (POS) plug-in <NUM>, correction plug-ins <NUM>, a variation filter (VF) plug-in <NUM>, a parser plug-in <NUM>, and a final scoring and filtering (FSF) plug-in <NUM>. The functions of the plug-ins <NUM> are discussed in greater detail with respect to <FIG>. In general, the POS plug-in <NUM> includes a ML-based component (e.g., a feedforward artificial neural network) that is trained to perform POS tagging of each token of an utterance with an associated part of speech (e.g., verb, noun, adjective, pronoun, adverb). The POS plug-in <NUM> is designed to output multiple potential POS taggings of an utterance, as well as corresponding confidence scores for each potential POS tagging of the utterance. The correction plug-in(s) <NUM> include a POS correction plug-in that applies ML-based techniques or applies rules (e.g., stored in the database <NUM>) to modify or remove potential POS taggings generated by the POS plug-in <NUM> that are known to be erroneous. The VF plug-in <NUM> applies a mathematical comparison of potential POS taggings generated by the POS plug-in <NUM>, and removes POS taggings that are not sufficiently different from one another. The parser plug-in <NUM> may include a rules-based or ML-based component (e.g., a feedforward artificial neural network) that is designed and/or trained to generate a respective meaning representation for each of the remaining candidate POS taggings, as well as corresponding confidence scores for the parsing operation. The correction plug-in(s) <NUM> also include a parser correction plug-in that applies ML-based techniques or applies rules (e.g., stored in the database <NUM>) to modify or remove potential meaning representations generated by the parser plug-in <NUM> that are known to be erroneous. The FSF plug-in <NUM> determines a final confidence score for each generated meaning representation, and then outputs a final set <NUM> of meaning representations having a corresponding final confidence score that is greater than a predefined threshold.

<FIG> is a flow diagram illustrating an embodiment of a process <NUM> by which the structure subsystem <NUM> of the NLU framework <NUM> can generate multiple, alternative meaning representations <NUM> for an utterance <NUM>. As such, the process <NUM> of <FIG> corresponds to block <NUM> of <FIG>. With reference to <FIG>, it may be appreciated that the utterance <NUM> may be one of the set <NUM> of utterances, such as a received user utterance <NUM>, one of the sample utterances <NUM> of the intent-entity model <NUM>, an utterance generated from cleansing (block <NUM>), or an utterance generated from vocabulary injection (block <NUM>), while the resulting meaning representations <NUM> may be part or all of the set <NUM> of meaning representations of the original utterance <NUM>. For the embodiment illustrated in <FIG>, the process <NUM> may be stored in a suitable memory (e.g., memory <NUM>) and executed by a suitable processor (e.g., processor(s) <NUM>) associated with the client instance <NUM> or the enterprise instance <NUM>, as discussed above with respect to <FIG>, <FIG>, and <FIG>.

For the embodiment illustrated in <FIG>, the process <NUM> is divided into two stages: a POS tagging stage <NUM> and a parse stage <NUM>. The POS tagging stage <NUM> begins with the structure subsystem <NUM> performing (block <NUM>) POS tagging of the utterance <NUM> to generate a set <NUM> of potential POS taggings, as well as corresponding confidence scores <NUM> for each potential POS tagging. In addition to the utterance <NUM>, the POS plug-in <NUM> also receives a POS threshold value. For example, the POS plug-in <NUM> may be provided with a POS threshold value <NUM> that is representative of a particular confidence level (e.g., <NUM>%), and as such, the set <NUM> of potential POS taggings will only include taggings having a corresponding confidence score that is greater than or equal to the POS threshold value <NUM>. In other embodiments, the POS plug-in <NUM> may additionally or alternatively receive a POS tagging limit value (e.g., <NUM>), and as such, the number of potential POS taggings generated by the POS plug-in <NUM> will be limited to the POS tagging limit value.

For the embodiment illustrated in <FIG>, the process <NUM> continues with the structure subsystem <NUM> using the correction plug-in <NUM> to perform auto-correction (block <NUM>) of the set <NUM> of potential POS taggings to generate a corrected set <NUM> of potential POS taggings. For example, as mentioned, the POS plug-in <NUM> may include a ML-based component. As such, the POS plug-in <NUM> may be subject to inadvertently learning incorrect POS tagging during training, for example, as a result of defective training data. Therefore, the correction plug-in <NUM> is designed to modify or remove potential POS taggings from the set <NUM> to block defective POS taggings from being carried forward to the remaining steps of the process <NUM>. In certain embodiments, the correction plug-in <NUM> may consult an external lexical database (e.g., stored in the database <NUM>, stored as part of the understanding model <NUM>) to ensure that token-surface forms (e.g., word-surface forms) match the selected POS tag for that token. It may be appreciated that this type of correction can be especially effective when the POS plug-in <NUM> is trained based on sub-word learning models (e.g., character-embedding-based models, morphemic models, etc.). In other embodiments, the correction plug-in <NUM> may include a ML-based component (e.g., an artificial neural network) that is trained to associate certain tokens with certain POS tags. For such embodiments, the correction plug-in <NUM> may determine a tagging score for each tag of a potential POS tagging in the set <NUM>, and eliminate all potential POS taggings having a tagging score that is below a predetermined tagging threshold value (e.g., stored in the database <NUM>).

For example, in certain embodiments, the correction plug-in <NUM> may implement a rules-based correction technique. For such embodiments, the correction plug-in <NUM> may access and apply rules <NUM> defined by a developer and stored in the database <NUM> to correct the set <NUM> of potential POS taggings. For example, the ML-based component of the POS plug-in <NUM> may mistakenly learn during training that the token "stop" denotes punctuation and not a verb or noun. As such, the developer may define one or more rules <NUM> to be applied by the correction plug-in <NUM> to either remove potential POS taggings from the set <NUM> that include this incorrect POS tagging, or to modify potential POS taggings from the set <NUM> to replace the erroneous POS tagging with a correct POS tagging. In certain embodiments, the correction plug-in <NUM> may instead use a combination of these techniques to eliminate or modify entries in the set <NUM> of potential POS taggings.

For the embodiment illustrated in <FIG>, the POS tagging stage <NUM> of the process <NUM> concludes with the structure subsystem <NUM> using the VF plug-in <NUM> to perform variability filtering (block <NUM>) of the corrected set <NUM> of potential POS taggings to generate a final nominee set <NUM> of potential POS taggings. In addition to the corrected set <NUM> of potential POS taggings, the VF plug-in <NUM> also receives a variation threshold value <NUM>, which defines how different the potential POS taggings in the corrected set <NUM> should be from one another. The purpose of variability filtering is to improve the efficiency of the process <NUM>, as well as the subsequent meaning search, by eliminating potential POS taggings that are considered too similar to other potential POS taggings in the corrected set <NUM> and, therefore, do not represent a sufficiently distinct path to warrant the computing resource expenditure to pursue.

Variability filtering can be accomplished in a number of different ways. In general, a tagging distance or variability function is used to determine how different two potential POS taggings in the corrected set <NUM> are from one another. Variability filtering blocks or prevents a meaning search from consuming additional computing resources to explore "shallow" differences between potential POS taggings that lead to meaning representations with only minimal and/or cosmetic differences. For example, a variability score may be calculated for any two entries in the corrected set <NUM> of potential POS taggings, and when the variability score is below the variation threshold <NUM>, then one of the potential POS taggings is considered sufficiently redundant and is not included in the final nominee set <NUM> of potential POS taggings. As such, when the number of potential POS taggings is relatively small (e.g., <NUM> or less), then each potential POS tagging in the corrected set <NUM> may be compared in this manner. However, for embodiments in which a larger number of potential POS taggings are present in the corrected set <NUM> of POS taggings, then clustering techniques may be applied. For example, a tagging distance function may be used to determine a tagging distance between the potential POS taggings such that they can be clustered into groups (e.g., by maximizing distance between clusters, by forming a predefined number of clusters) based on these tagging distances. Subsequently, a representative potential POS tagging is selected from each group (e.g., the centroid potential POS tagging of each cluster) and advances to the next step in the process <NUM>. It may be appreciated that other variability filtering techniques may be used, in accordance with the present disclosure.

In one example with a relatively small corrected set <NUM> of POS taggings, the VF plug-in <NUM> may use a diversity-based inclusion technique. For this example, the utterance <NUM> may be, "Change my password. " In block <NUM>, the POS plug-in <NUM> may determine the set <NUM> of potential POS taggings indicating that the tokens of the utterance <NUM> are, in order: (<NUM>) a command form of a verb, an adjective, and a noun; or (<NUM>) a verb, an adjective, and a noun; or (<NUM>) a noun, an adjective, and a noun; or (<NUM>) a verb, a pronoun, and a noun. After correction in block <NUM>, the VF plug-in <NUM> receives the corrected set <NUM> of potential POS taggings and applies a variability function to determine how different the potential POS taggings are from one another. For this example, the VF plug-in <NUM> begins by considering the first entry in the corrected set <NUM> of potential POS taggings. The VF plug-in <NUM> determines that, since there is no basis for comparison, the first entry is sufficiently different and should be included in the final nominee set <NUM> of potential POS taggings. Next, the VF plug-in <NUM> may consider the second entry in the corrected set <NUM> of potential POS taggings by comparing it to the first entry using the variability function. An example variability function may be a weighted average. For this example, when the first and second entries are compared, the first tag (e.g., command form of verb) of the first entry and the first tag (e.g., verb) of the second entry are compared. Difference values for different tag comparisons may be stored as part of the rules <NUM> in the database <NUM>. For example, the difference value assigned to a verb-verb comparison, a noun-noun comparison, an adjective-adjective, etc., may be zero; the difference value assigned to a command form verb-verb comparison may be slightly greater than zero (e.g., <NUM>); the difference value assigned to a verb-noun comparison, a verb-adjective, a noun-adjective, etc., may be one, and so forth, within the database <NUM>. In certain embodiments, the database <NUM> may further store weighting values for different POS tags, such that certain POS tags (e.g., verbs) have a greater contribution to the output of the variability function than other POS tags (e.g., nouns, pronouns). For this example, the weights of the POS tags are equivalent. As such, the variability function may calculate a variability score between the first and second entries (e.g., (<NUM> for the difference between first tags + <NUM> for the difference between the second tags + <NUM> for the difference between the third tags)/(<NUM> tags compared) = <NUM>), and then compare this variability score to the variation threshold value <NUM> (e.g., <NUM>). Since the variability score is below the variation threshold value <NUM>, the second entry is not included in the final nominee set <NUM> of POS taggings. This process continues with the third entry in the corrected set <NUM> of potential POS taggings being compared to the first entry (e.g., (<NUM> for the difference between first tags + <NUM> for the difference between the second tags + <NUM> for the difference between the third tags)/(<NUM> tags compared) = <NUM>, which is at the variation threshold value of <NUM>), and the third entry is included in the final nominee set <NUM> of potential POS taggings. Subsequently, the fourth entry in the corrected set <NUM> of potential POS taggings is compared to the first entry (e.g., (<NUM> for the difference between first tags + <NUM> for the difference between the second tags + <NUM> for the difference between the third tags)/(<NUM> tags compared) = <NUM>, which is greater than the variation threshold value of <NUM>), and also compared to the third entry (e.g., (<NUM> for the difference between first tags + <NUM> for the difference between the second tags + <NUM> for the difference between the third tags)/(<NUM> tags compared) = <NUM>, which is greater than the variation threshold value of <NUM>), and is also included in the final nominee set <NUM> of potential POS taggings that are carried forward in the process <NUM>.

For the embodiment illustrated in <FIG>, the parse stage <NUM> begins with the structure subsystem <NUM> performing (block <NUM>) parse inference using the parser plug-in <NUM> to generate a set <NUM> of potential meaning representations from the final nominee set <NUM> of potential POS taggings, as well as a corresponding confidence score <NUM> for the parsing of each potential meaning representation in the set <NUM>. In addition to the final nominee set <NUM> of potential POS taggings, the POS plug-in <NUM> also receives a parse threshold value <NUM> that may be stored in the database <NUM>. For example, the parser plug-in <NUM> may be provided with a parse threshold value <NUM> that is representative of a particular confidence level (e.g., <NUM>%), and as such, the set <NUM> of potential meaning representations will only include meaning representations having a corresponding confidence score that is greater than or equal to the parse threshold value <NUM>. It may be noted that, in certain cases, the parser plug-in <NUM> may not be able to generate a meaning representation for certain potential POS taggings.

In certain embodiments, it may be appreciated that the process <NUM> may include a second auto-correction step (block <NUM>) to modify or remove entries in the set <NUM> of potential meaning representations before final scoring and filtering is performed. For example, as mentioned, the parser plug-in <NUM> may include a ML-based component. As such, the parser plug-in <NUM> may be subject to inadvertently learning incorrect parse tree structure generation during training, for example, as a result of defective training data. Therefore, in certain embodiments, the structure subsystem <NUM> may include a parser correction plug-in designed to modify or remove potential meaning representations from the set <NUM> to block defective meaning representations from being carried forward to the remaining steps of the process <NUM>. In certain embodiments, this parser correction plug-in may include a ML-based component (e.g., an artificial neural network) that is trained to associate certain POS taggings with certain parse tree structures. In other embodiments, the parser correction plug-in may implement a rules-based correction technique, or a combination of rules-based and ML-based techniques, as discussed for the correction plug-in <NUM>.

For the embodiment illustrated in <FIG>, the parse stage <NUM> of the process <NUM> concludes with the structure subsystem <NUM> using the FSF plug-in <NUM> to perform final scoring and filtering (block <NUM>) of the set <NUM> of potential meaning representations generated by the parser plug-in <NUM> in block <NUM>. For example, as illustrated in <FIG>, in certain embodiments, the FSF plug-in <NUM> receives the set <NUM> of potential meaning representations generated by the parser plug-in <NUM>, the corresponding confidence scores <NUM> for each of the potential meaning representations, and the corresponding confidence scores <NUM> for the potential POS taggings used to generate the set <NUM> of potential meaning representations. In addition, the FSF plug-in <NUM> also receives a FSF threshold value <NUM>, which may be stored in the database <NUM>. The FSF plug-in <NUM> defines a final scoring function that calculates a final (e.g., overall, cumulative) score for each entry in the set <NUM> of potential meaning representations based on these confidence values, and only advances meaning representations having a final score that is at or above the FSF threshold value <NUM>.

In different embodiments, the final scoring function of the FSF plug-in <NUM> may combine the confidence scores <NUM> from POS tagging and the confidence score <NUM> from parsing in different manners. For example, in certain embodiments, the final scoring function may multiply and/or sum the confidence score <NUM> and the confidence score <NUM>. In certain embodiments, this multiplication and/or summation process may be weighted to increase the impact of the confidence scores <NUM> from POS tagging or the confidence scores <NUM> from parsing to the final scores. By way of specific example, in an embodiment, the corresponding final score of a meaning representation in the set <NUM> of potential meaning representations may be the five times the confidence score <NUM> from POS tagging plus the confidence score <NUM> from parsing. In other embodiments, the confidence scores <NUM> and <NUM> may be combined as a weighted average, wherein the relative weights for each confidence score may be predefined within the database <NUM>.

<FIG> is a flow diagram illustrating an embodiment of a process <NUM> whereby the original utterance <NUM> is re-expressed by the vocabulary subsystem <NUM> and the structure subsystem <NUM> into the set <NUM> of final nominee POS taggings, as discussed above with respect to <FIG> and <FIG>. As set forth above, the original utterance <NUM>, which may be a received user utterance <NUM> or one of the sample utterances <NUM> of the intent-entity model <NUM>, is provided to the vocabulary subsystem <NUM> for vocabulary injection <NUM>. In certain embodiments, the original utterance <NUM> may first undergo a cleansing step prior to vocabulary injection <NUM>. As a result of vocabulary injection <NUM>, the set <NUM> of utterances is generated that includes the original utterance <NUM> and additional utterances (e.g., utterance <NUM>) that are re-expressions of the original utterance <NUM> having alternative phraseology and/or terminology.

Continuing through the embodiment illustrated in <FIG>, each utterance of the set <NUM> of utterances is provided to the POS plug-in <NUM> for POS tagging <NUM> to generate sets <NUM> of potential POS taggings for each utterance. The POS plug-in <NUM> generates a plurality of POS taggings for each utterance of the set <NUM> of utterances. Subsequently, each of the potential POS taggings of the set <NUM> of potential POS taggings is provided to the POS correction plug-in 270A for auto- correction <NUM>. For this example, during auto-correction <NUM>, the POS correction plug-in 270A removes certain potential POS taggings from the set <NUM> of potential POS taggings that are known to be erroneous to generate the corrected sets <NUM> of potential POS taggings.

Continuing through the embodiment illustrated in <FIG>, after auto-correction <NUM>, the sets <NUM> of potential POS taggings are provided to the VF plug-in <NUM> for variability filtering <NUM>. As indicated by the bracket <NUM>, the VF plug-in <NUM> compares the potential POS taggings across all corrected sets <NUM> of potential POS taggings. This design is based on the present recognition that a potential POS tagging generated for a first utterance (e.g., the original utterance <NUM>) may be substantially similar to the potential POS tagging generated for a second utterance <NUM> of the set <NUM> of utterances. As such, it may be appreciated that, in certain embodiments, the VF plug-in <NUM> applies a variability or tagging distance function, as discussed above, to remove potential POS taggings across all corrected sets <NUM> of potential POS taggings, generating the set <NUM> of final nominee POS taggings.

<FIG> is a flow diagram illustrating an embodiment of a process <NUM> whereby the set <NUM> of final nominee POS taggings generated by the process <NUM> of <FIG> is transformed into the set <NUM> of meaning representations of the original utterance <NUM> by the structure subsystem <NUM>, as discussed above with respect to <FIG>. Each entry in the set <NUM> of final nominee POS taggings is provided to the parser plug-in <NUM> for parse inference <NUM>, and the parser plug-in <NUM> generates a respective potential meaning representation <NUM> for each entry in the set <NUM>. Subsequently, each of the meaning representations <NUM> is provided to the parse correction plug-in 270B for auto-correction <NUM>. For this example, during parse auto-correction <NUM>, the parse correction plug-in 270B modifies or removes certain potential meaning representations <NUM> that are known to be erroneous to generate the corrected potential meaning representations <NUM>. For example, during the parse auto-correction <NUM>, the parse correction plug-in 270B may detect incorrect node-node parse links (e.g., a direct object node erroneously linking to a subject node), incorrect node POS/link type combinations (e.g., a direct object link type pointing to a node that is POS-tagged to be a verb), and so forth, in the potential meaning representations <NUM> and remove or modify these incorrect structures, as reflected by the corrected potential meaning representations <NUM>.

As discussed, each potential meaning representation <NUM> is provided to the FSF plug-in <NUM>, along with the corresponding confidence scores determined by the parser plug-in <NUM> during the parse inference <NUM>, as well as the corresponding confidence scores determined by the POS plug-in <NUM> during the POS tagging <NUM>, for each potential meaning representation <NUM>. The FSF plug-in <NUM> combines these confidence scores, as discussed above, to determine a respective final score for each potential meaning representations <NUM>. Then, the FSF plug-in <NUM> performs a final score filter operation <NUM>, in which each potential meaning representation <NUM> having a final score below the FSF threshold <NUM> is not included in the final set <NUM> of meaning representations. As such, the final output is a set <NUM> of meaning representations that are representative of the original utterance <NUM>. As noted above, whether the original utterance <NUM> is a received user utterance <NUM> or a sample utterance <NUM> of the intent-entity model <NUM>, since the final set <NUM> of meaning representations include multiple re-expressions and forms of the original utterance <NUM>, the chances of locating a match during the meaning search (intent inference) is improved. Additionally, since potentially erroneous and substantially redundant forms of the original utterance <NUM> are removed from consideration when generating the final set <NUM> of meaning representations, these forms are not incorporated into the meaning search, improving the efficiency of the meaning search and the operation of the NLU framework.

Technical effects of the present disclosure include providing an agent automation framework that is capable of extracting meaning from user utterances, such as requests received by a virtual agent (e.g., a chat agent), and suitably responding to these user utterances. Additionally, present embodiment provide an NLU framework having a structure subsystem capable of detecting multiple alternative meaning representations (e.g., utterance trees) for a given natural language utterance. The disclosed structure subsystem generates these multiple meaning representations by applying a series of refinements on the results of ML-based and rules-based techniques that detect different facets of an utterance's form or shape. The disclosed structure subsystem enables improved intent inference of natural language requests by considering as multiple probable form-derivations that should be given due consideration during the meaning search, while effectively pruning erroneous or effectively redundant form-derivations from consideration. As such, the disclosed structure subsystem improves the performance, the domain specificity, and/or the efficiency of the NLU framework.

Claim 1:
An agent automation system, comprising:
a memory (<NUM>) configured to store a natural language understanding, NLU, framework (<NUM>), wherein the NLU framework (<NUM>) includes a part-of-speech, POS, component, a correction component, a variability filter component, a parser component, and a final scoring and filtering component; and
a processor (<NUM>) configured to execute instructions of the NLU framework (<NUM>) to cause the agent automation system to perform actions comprising:
using the POS component to perform part-of-speech, POS, tagging (<NUM>) of a set of utterances to generate a set of potential POS taggings (<NUM>) from the set of utterances, wherein the POS tagging generates a plurality of potential POS taggings for each utterance of the set of utterances, and wherein for each utterance, each of the plurality of potential POS taggings is generated by performing POS tagging of each token of the utterance;
using the correction component to apply rules-based or machine learning based correction techniques to modify or remove erroneous potential POS taggings in the set of potential POS taggings;
using the variability filter component to remove one or more POS taggings (<NUM>) from the set of potential POS taggings (<NUM>) that are similar to other POS taggings in the set of potential POS taggings (<NUM>);
using the parser component to generate a set of potential meaning representations (<NUM>) from the set of potential POS taggings (<NUM>); and
using the final scoring and filtering component to calculate a respective final score for each potential meaning representation in the set of potential meaning representations (<NUM>) and to remove potential meaning representations from the set of potential meaning representations (<NUM>) based on their respective final score to generate a final set of meaning representations.