Patent Description:
<CIT> describes processing natural language user queries for commanding a user interface used to perform functions. Individual user queries are classified in accordance with the types of functions and a plurality of user queries may be related to define a particular command. To assist with classification, a query type for each user query is determined where the query type is one of a functional query requesting a particular new command to perform a particular type of function, an entity query relating to an entity associated with the particular new command having the particular type of function and a clarification query responding to a clarification question posed to clarify a prior user query having the particular type of function. Functional queries may be processed using a plurality of natural language processing techniques and scores from each technique combined to determine which type of function is commanded.

<NPL>, describe Ubuntu Dialogue Corpus, a dataset containing almost <NUM> million multi-turn dialogues, with a total of over <NUM> million utterances and <NUM> million words. This provides a unique resource for research into building dialogue managers based on neural language models that can make use of large amounts of unlabeled data. The dataset has both the multi-turn property of conversations in the Dialog State Tracking Challenge datasets, and the unstructured nature of interactions from microblog services such as Twitter. They also describe two neural learning architectures suitable for analyzing this dataset, and provide benchmark performance on the task of selecting the best next response.

This summary is not intended to exclusively identify key features or essential features of the claimed subject matter, nor is it intended as an aid in determining the scope of the claimed subject matter.

The embodiments of the disclosure include methods and systems for using, creating and maintaining goal-oriented, dialog systems (i.e., "transactional bots" or "bots") that provide interfaces to application functionality such as, for example, interfaces to application functionality for ordering a taxi, controlling devices in the home, banking, or shopping. The methods and systems of the embodiments provide a bot that may learn in supervised learning and reinforcement learning from conversational examples provided by domain experts and from interaction with users. The embodiments provide conversational bots that may be created to interact using both text and/or application programming interface (API) calls. Use of the embodiments allows a developer to configure a bot that interfaces with an application back-end and allows the behavior of the bot to be configured by use of masking actions. Use of the embodiments also allows flexible design of a specification for the bot that specifies how developer code may be organized, for example, as masking operations on the possible actions the bot may execute. Additionally, the methods and systems may automatically infer the best state representation during a dialog so a state variable need not be predefined.

In an embodiment the methods and systems may be implemented as a bot using a Long Short-Term memory (LSTM) recurrent neural network model. The LSTM recurrent neural network may be optimized with supervised learning and/or using reinforcement learning via a specialized policy gradient method. In the implementation bot behavior may be specified with example dialogs. Bot behavior also may be improved autonomously from interactions with users without human intervention using a controller that makes use of business rules to gate exploration.

The claimed embodiments comprise a system according to claim <NUM>.

In an implementation, the system and recurrent neural network may perform operations in a loop type behavior while the interactive dialog occurs by varying the configuration of the set of features provided to the recurrent neural network. For example, the system may include features generated by the system, features returned from an API call (if the previous action was an API call), or an indication of a previous action taken by the system (if there was a previous action), when providing the set of features to the recurrent neural network.

Another example implementation the system may include one or more processors that control a supervised learning process. The system may include memory that includes programs or code, that when executed, causes the one or more processors to control the system to receive one or more sample dialogs created by a developer, determine if any action in the one or more sample dialogs is masked out, and, in response to a determination that no action in the one or more sample dialogs is masked out, incorporate the one or more sample dialogs into a training set. The system may also create a log including a set of features and entity extractions for the training. The system may then apply training to a recurrent neural network using the training set and the log, score the one or more sample dialogs using the recurrent neural network to generate a score result based on the training, and, determine if a target action in the one or more sample dialogs was not assigned a highest probability in the score result. The determination of whether a target action was not assigned a highest probability may then be used to further refine the training. For example, if it is determined that a target action in the one or more sample dialogs was not assigned a highest probability in the score result, the system may provide an indication of a dialog of the one or more sample dialogs in which the target action was not assigned a highest probability to the developer. If it is determined that each target action in the one or more sample dialogs was assigned a highest probability in the score result, the system may provide an indication that all of the one or more sample dialogs were reconstructed successfully to the developer.

In a further example implementation the system may include one or more processors that control a reinforcement learning process. The system may include memory that includes programs or code, that when executed, causes the one or more processors to control the system to receive a definition for a reward signal and a return for one or more dialogs, perform the one or more dialogs using a recurrent neural network, create a log of rewards, features, available actions, and selected actions from the performed one or more dialogs, provide the return for the performed one or more dialogs to the neural network for use in improvement of the neural network, and provide the log for output at a user interface.

The system and method will now be described by use of example embodiments. The example embodiments are presented in this disclosure for illustrative purposes, and not intended to be restrictive or limiting on the scope of the disclosure or the claims presented herein.

The disclosed embodiments provide a technical advantage as compared to currently used methods for using and creating transactional bots (bots) for dialog systems. The embodiments provide a system that allows a neural network to be optimized both with supervised learning and using reinforcement learning via a specialized policy gradient method. In an implementation bot behavior may be specified with example dialogs. Bot behavior also may be improved autonomously from interactions with users without human intervention using a controller that reinforces business rules to gate exploration by the neural network. Additionally, use of the embodiments with a recurrent neural network allows inference of a latent representation of the state of the system.

Use of the embodiments provides an advantage over currently used rule-based methods. Rule languages used to define behavior in rule based systems are often not easy to use by non-experts. It is often difficult for a program manager, designer, or marketing executive to build or maintain a bot without support from a software developer expert in rules languages. Also, as the number of rules for defining a system grows, the rules begin to interact and, as a result, rule changes have unforeseen consequences, so fixing one bug can introduce many other bugs. This makes maintenance of non-trivial rule based systems slow and extremely difficult. Additionally, rule-based methods do not learn automatically from experience. Even if a bot conducts a million dialogs a day, no automatic improvement is possible. Improvements to bot behavior can only be made through time-consuming human analysis. Use of the embodiments of the disclosure allows bot behavior to be specified using example dialogs without the need to know a specialized rules language. For example, designers, program managers, or marketing executives may configure the system without help from specialized software developers who know the rules language. Use of the embodiments also allows bot behavior to be improved autonomously through interactions with users without human intervention. The embodiments also provide the advantage that exploration performed by a neural network can be gated or restricted in scope by optional code that enforces rules for bot behavior.

The capability of the embodiments for bot behavior to be improved autonomously through interactions with users of the embodiments also provides an advantage over currently used supervised learning (SL) methods. In SL, a domain expert provides example dialogs that the bot should imitate, and a machine learning algorithm ingests these and builds a model which attempts to generalize to new dialogs. SL methods do not have the ability to make improvements automatically through experience or through reinforcement learning. In order to make improvements in bot behavior, a domain expert must manually examine and label dialogs.

The capability of the embodiments of the disclosure to infer a latent representation of state without the need to manually define the state also provides an advantage over currently used supervised learning SL methods. Current SL methods require the careful design of a "state" variable, which the SL algorithm uses as the basis for choosing actions. Design of the state variable is problematic. Including too little information in the state variable definition prevents the SL algorithm from successfully learning to reconstruct the example dialogs. Including too much information in the state variable definition causes over-fitting or narrowing of behavior, which means the bot will not generalize to new situations. In the embodiments, use of a recurrent neural network allows inference of a latent representation of state. This inference of state substantially reduces the effort developers need to put into hand-engineering the state required by current SL methods.

Also, current SL methods do not allow actions to be masked by a developer. The capability of the embodiments to mask actions allows support of business logic, such as only allowing an action like transferring funds after an action confirming the transfer with the user has succeeded.

The embodiments of the disclosure also provide advantages over currently used reinforcement learning (RL) methods in which an RL algorithm explores different actions in different states, and over time makes progress toward finding the mapping from state to action that maximizes a reward signal. RL learns automatically, without input from system designers. RL systems are difficult to create and, typically, for the first <NUM> or <NUM> of dialogs, RL systems usually perform very badly because an RL system explores all possible actions, including spurious actions. In the embodiments of the disclosure the capability to gate exploration using optional code that enforces business rules may be used to prevent exploration of spurious actions and improve system behavior as compared to the current RL systems. Also, in an RL system, as with an SL system, careful definition of the state variable is needed. The embodiments of the disclosure remove the need to define the state variable that is present in current RL systems.

Referring now to <FIG>, therein is a simplified block diagram illustrating example functions according to an embodiment of the disclosure. <FIG> illustrates functional blocks of an example network <NUM> including an example system <NUM>. In the example implementations system <NUM> may implement functionality for a bot and include entity extractor <NUM>, recurrent neural network <NUM>, and controller <NUM>. System <NUM> may also include a developer interface <NUM>.

System <NUM> interacts with device <NUM> and application server <NUM>. Device <NUM> may include an application that allows a user to interact with and receive services from application server <NUM> through system <NUM>. System <NUM> may communicate with a user of device <NUM> via one or more channels. The one or more channels may be any type of communication channels carrying communications for applications including short messaging services (SMS), email services, messaging platforms such as conferencing platforms, social network platforms, text messaging platforms, or any other type of application using text communication. The applications may also include any type of application using voice or spoken communications, or web browsers. In various implementations text may be typed in by the user, or transcribed text produced by a speech recognizer, or, also may be translated from another language using an automatic translation service. The user could engage in a one to one dialog with the bot, or the bot could be participating in a dialog with a plurality of users. The bot could be addressed directly, or the bot could monitor a conversation and respond when it determines a response would be relevant.

In the claimed embodiments of the invention, the device <NUM> is a home appliance. In unclaimed embodiments, the applications may include, for example, reservation applications, retail/purchasing applications, information retrieval applications, or any other type of application that may interact with a user through a dialog. The functions of entity extractor <NUM>, recurrent neural network <NUM> and controller <NUM> of system <NUM> may be implemented using one or more servers. In alternative embodiments of <FIG>, the functions described for a particular functional block in relation to <FIG> may be performed in another of the functional blocks of <FIG> or divided up between the functional blocks in a different way.

Referring now to <FIG>, therein is a flow diagram illustrating example signal flows for data exchange between functions of an operational loop <NUM> in an implementation according to <FIG>. The functions of <FIG> may be described with reference to <FIG>. Block text input/text output <NUM> may represent functions performed at device <NUM>, blocks entity input <NUM>, action mask <NUM>, entity output <NUM>, and API call <NUM>, may represent function performed by controller <NUM>, and block entity extraction <NUM> may represent functions performed at entity extractor <NUM>. Long short-term memory recurrent neural network (LSTM) may represent functions performed at recurrent neural network <NUM>. Softmax <NUM> represents a conversion of LSTM <NUM> output values into action probabilities for communication to controller <NUM>.

The process begins when device <NUM> provides a text input signal <NUM> to entity extractor <NUM>. At <NUM>, entity extraction is performed and entities are identified in the text. For example, in the query "Will it rain in Sacramento on Friday?", entity extraction at <NUM> may identify "Sacramento" as a city, "Friday" as a day, and "rain" as a weather event. The entity extraction of <NUM> may also use custom models specific to the user's domain, or may use generic models suitable for many domains. In the claimed embodiments, generic models extract entities such as dates, times, and locations, etc. the entity extraction at <NUM> may optionally resolve entities to a machine interpretable form. For example, "January" might be resolved to "month=<NUM>". In unclaimed embodiments, generic models may also extract monetary amounts.

Next, entity extractor <NUM> sends signal <NUM> to pass marked up text at <NUM> to the controller <NUM>. Entity extractor <NUM> may also pass the input text and entities to recurrent neural network <NUM> through signal <NUM> at block <NUM>. Controller <NUM> is configured to perform a defined set of operations and the developer may be further guided in what to write through the user-interface design. The controller <NUM> may resolve the text of entities to ontology within the domain of a particular application. For example, it might resolve the user's input of "town car" to a canonical indication like "CAR_TYPE=UBER_LIVERY". The controller <NUM> code may also maintain its own internal state, for example, tracking entities that have been input or output over the course of the dialog. Controller <NUM> implemented in a variety of ways, including as a web service (e.g. in C# or node. js), or run locally.

The controller <NUM> returns a "mask" over actions, i.e., an indication of which actions in the text are allowed and disallowed at action mask <NUM> through signal <NUM> to recurrent neural network <NUM>. The code also returns other features at block <NUM> which can be used by recurrent neural network <NUM>. The other features input to neural network by signals <NUM> and <NUM> depend on whether any previous operations were performed in the loop <NUM> and what those operations were. When controller <NUM> returns an indication of which actions are available, the list of actions returned may include wildcards. For example, the indication of available actions may indicate that any text output action is allowed with a wildcard indication for text output, but only certain API calls are allowed.

Masking in the masking over available actions takes into account which entity types are available to controller <NUM>. For example, if the entity for "DEPARTURE_CITY" hasn't been received, actions that ask the user to confirm "DEPARTURE_CITY" (like "Leaving from Boston, is that right?") may be masked out. If controller <NUM> returns additional features to neural network <NUM>, these can be implemented as a programmatic dictionary, like { "estimated_wait_minutes": <NUM>, "user_logged_in": false }.

At block <NUM>, the set of features is then passed to recurrent neural network <NUM> along with the mask over available actions, if provided at <NUM>. The set of features includes the text of the user's input at <NUM>, an indication of which entities or types of entities were detected at <NUM>, features returned from the developer code at <NUM>, features returned from an API call (if the previous action was an API call), and an indication of the previous system action taken (if a previous system action was taken). The features can also include other items from previous time steps, however, the stateful nature of the recurrent neural network should prevent the need to do this.

At <NUM>, the recurrent neural network <NUM> performs the function of a LSTM recurrent neural network and generates a distribution over actions to take. The distribution may be based on the received set of features and a received mask over available actions. The output actions may be represented as a "flat" list, or as a generative process. For example, a generative process might use a second neural network which can generate text, along with the probability of that text.

The output actions may include references to entity types, such as "CAR_TYPE", without specific values populated (like "town car"). Using entity types in actions rather than entity values may substantially reduce the number of actions, and allow the system to generalize over entity values it has never seen before. This may be an advantage when an entity may take on many values, or when the possible values may change over time, such as "RESTAURANT_NAME" or "EMAIL_SUBJECT". These references may be populated, before they are output.

If the output actions are represented as a flat list, the mask may be implemented by multiplying the raw (unmasked) action probabilities by an array of <NUM> and <NUM>, with <NUM> for allowed actions and <NUM> for disallowed actions. This array may then be renormalized so the non-zero actions sum to <NUM>. The gradients of weights with respect to these masked outputs may be well-defined, so gradient descent methods may be applied.

Recurrent neural network <NUM> then samples an action from the (masked) distribution of actions. The action may be sampled from the distribution in the traditional sense. This type of sampling may provide an advantage when exploring for reinforcement learning in the neural network, at the expense of some performance. In another implementation, the action with the maximum probability may be selected. This type sampling may provide an advantage for maximizing performance, at the expense of not exploring.

At block <NUM>, the controller <NUM> is sent an indication of which action has been selected. Controller <NUM> may update its internal state at this point. Controller <NUM> may also send an indication of the selected action to recurrent neural network through signal <NUM> for use as a feature. If the chosen action contains references to entity types, they are populated by the developer code at <NUM>. For example, DESTINATION_CITY in "Going to DESTINATION_CITY, is that right?" may be changed to "Seattle". The selected action, with entity substitution, is then executed.

At <NUM> the action type is determined. If the action is a textual action, it is output to the user as text output at <NUM> through signal <NUM>. If the action is an API call, the API call is invoked at <NUM> through signal <NUM>. The API call at <NUM> may optionally return features related to the API call to neural network <NUM> through signal <NUM>. If the action is a special "LISTEN" action, control returns to <NUM> and the system waits for user input. Other special actions may be defined, such as "QUIT" (which causes the system to abandon the conversation), "ESCALATE" (which causes a human agent to step in on behalf of the agent). The special actions may be defined to include any other appropriate action.

In other implementations, the flow described above may be modified to allow for asynchronous input, for example, if the user types input while the system is waiting on a response from an API call, the system may queue up user input and execute the loop when user input is received. In embodiments, API calls may interact with external services, or may interact with the user. For example, an API call could show a map, picture, or contact card to the user.

Following is a simple example of the type of dialog sequence that is possible with this loop:.

Referring now to <FIG>, is a flow diagram illustrating runtime operations performed by example system <NUM>. <FIG> illustrates operations performed by system <NUM> of <FIG> during execution of runtime loop <NUM>. <FIG> is similar to <FIG> but shows operations in a linear manner.

The process begins at <NUM> where user input is received. At <NUM>, entity extraction is performed on the user input by entity extractor <NUM>. At <NUM>, marked up text is passed to the controller <NUM>. The input text and entities may also be passed to the recurrent neural network (LSTM). At <NUM>, controller <NUM> determines the mask over actions, and at <NUM> the set of features and the mask are passed to the recurrent neural network <NUM>. The set of features may include features as described for <FIG>. At <NUM> the recurrent neural network <NUM> provides controller <NUM> with the selected action based on the set of features and the mask. At <NUM> controller110 updates its internal state. Controller <NUM> may also send an indication of the selected action to recurrent neural network <NUM> for use as part of a feature set, or, if the selected action contains references to entity types, populate the references.

At <NUM>, controller <NUM> determines if the selected action includes entity references. If the selected action includes entity references, controller <NUM> populates the references at <NUM> as was described in relation to <FIG> and moves to operation <NUM>. If the selected action does not include entity references the process moves from <NUM> to <NUM>.

At <NUM> it is determined if the selected action is a textual action. If the selected action is a textual action, at <NUM>, controller <NUM> initiates the performance of text output at device <NUM> and returns to operation <NUM> at <NUM>. If the selected action is not a textual action the process moves from <NUM> to <NUM>.

At <NUM> it is determined if the selected action is an API call. If the selected action is an API call, at <NUM>, controller <NUM> performs the appropriate API call. Controller <NUM> may also send an indication of the API call to recurrent neural network as part of the feature set. Next at <NUM> the process returns to operation <NUM>. If the selected action is not an API call the process moves to <NUM>.

At <NUM> it is determined if the selected action is a "listen" action. If the selected action is a listen action, the controller <NUM> initiates listening for user input at <NUM>. When user input is received the process returns from <NUM> to <NUM>.

If the selected action is not a listen action the process moves to <NUM> and determines a quit action was received. The process then ends at <NUM>.

In alternative implementations, any number of actions may be defined for use in the process, and controller <NUM> may direct the process appropriately depending on which of the actions is selected.

Referring now to <FIG>, therein is a flow diagram illustrating supervised learning operations performed by an example system according to the embodiments. The process of <FIG> may be performed for supervised learning operations in the system <NUM> of <FIG>. The process of <FIG> may be used improved performance of system <NUM> by providing example dialogs, and training the recurrent neural network <NUM> to mimic the example dialogs.

The process begins at <NUM> where one or more sample dialogs are created. Because the embodiments do not use rule-based methods, the dialogs maybe created by persons who are non-experts in rules languages. For example, a domain expert such as a designer, program manager, marketing executive, or developer may create the one or more new sample dialogs. The sample dialogs may include an indication of where API calls are made and may be created in entirety by the domain expert through interaction with the current bot. The one or more sample dialogs also may be drawn from an existing corpus of interactions with real users or may be synthesized through interaction with a simulated user. The one or more sample dialogs may also be created by interacting with crowd workers, or taking existing dialogs and scrambling them.

At <NUM> a supervised learning training flag is passed to controller <NUM>.

The run-time loop, which was described in relation to <FIG>, is run on the one or sample dialogs in a modified loop form in performing the supervised learning (SL) of <FIG>. In the modified loop, actions are not sampled, but rather, a check is done to see if each action in the sample dialog is masked out or not. At <NUM> the controller determines if any action in the one or more sample dialogs is masked out. At <NUM> if an action in the one or more sample dialogs is masked out the process moves to <NUM>. If any action in the sample dialog is masked out, this means the dialog could not be produced with the existing developer code, i.e., the dialog is inconsistent with the developer code. At <NUM> the inconsistency is reported, for example, to the developer. The developer may then edit the dialogs or controller code at <NUM>. The process may then return from <NUM> to <NUM> to begin again.

If at <NUM> no action in the one or more sample dialogs is masked out the process moves to <NUM>. When no actions in the sample dialogs are masked out, this indicates the dialog may be produced by the existing developer code. In this case, the dialog is incorporated into the training set. In this modified loop, at <NUM>, a log is created for the features reported by the developer code and entity extraction model. The log created at <NUM> may be used by the supervised learning algorithm in the SL training.

At <NUM> the SL training is then applied using recurrent neural network <NUM> and the entire corpus of training dialogs. The inputs are as described for the runtime loop of <FIG>, and the target outputs are the actions which appear in the training dialogs.

The recurrent neural network <NUM> may use gradient descent to train the model. If the model is a distribution over a flat list of actions, categorical cross-entropy between the model output and the one-hot vector encoding of the target action may be used by the recurrent neural network as the loss function. After SL training is applied, at <NUM>, all of the dialogs in the training corpus are then scored using the new sequence model. At <NUM>, a check is then performed to see whether any of the target actions in the training dialogs were not assigned the highest probability by the model.

Next, at <NUM> if any action in a training dialog was not assigned the highest probability by the new sequence model, the disagreement indicates that the new sequence model has failed to re-construct the training dialogs and the process moves to <NUM>. At <NUM>, the dialog turns of any disagreement are provided to the developer. The developer may then resolve the disagreement at <NUM> by changing, deleting, or adding an example dialog in the training corpus, modifying the developer code, modifying the SL learning algorithm or parameters, or modifying the entity extraction model. If however if any action in a training dialog was not assigned the highest probability by the new sequence model, the process moves from <NUM> to <NUM>. At <NUM>, the developer may be provided with an indication that the new sequence model has successfully re-constructed the training dialogs. The SL learning cycle may then be repeated. At any time, the developer may "deploy" the trained bot, so it is available for interaction with users in the runtime loop.

In addition to SL training, use of the embodiments also allows the use of reinforcement learning (RL) training to improve performance of a bot. Referring now to <FIG>, therein is a flow diagram illustrating reinforcement learning operations performed by an example system. <FIG> shows operations to optimize performance of system <NUM> of <FIG> in the runtime loop of <FIG> through interaction with users, and adjustment of the model using reinforcement learning.

The process of <FIG>, begins at <NUM> where the developer defines a reward signal. The reward signal may be a real-valued number that indicates how "good" a system action is in a particular context. The rewards signal may be inferred from usage. For example, the signal may indicate whether the user achieved a desired task. The rewards signal may also be provided by the user providing an answer to a question at the end of a dialog such as "Did I do OK?" The reward signal may also be exogenous to the dialog, for example, whether the user later made a purchase sometime after the dialog completed. Alternately, the rewards signal may be provided by a labeler who examines dialogs and scores their quality. The reward signal may indicate a property of the whole dialog, a portion of a dialog, or a specific turn of dialog. For example, a reward at the end of the dialog might indicate a task completion for the whole dialog and a reward at a specific turn in the middle of a dialog might indicate the quality for that specific action.

At <NUM> the developer defines a return for the dialog. The overall return for the whole dialog may be a discounted sum of the rewards at each turn. The discount factor may be defined by the developer, or may be set to a standard value. For example the discount factor may be set to a value such as <NUM>.

At <NUM>, system <NUM> conducts a batch of dialogs. The batch may include one or more dialogs with one or more users. The users may be real users, crowd workers, or user simulations. At <NUM>, the rewards, features, available actions, and actions selected from the batch are logged. At <NUM>, recurrent neural network <NUM> makes improvements based on the rewards received during the batch. The improvement at <NUM> may be made using a policy gradient. Next, at <NUM> the recurrent neural network <NUM> is updated and deployed. At <NUM>, it is determined if the training if finished. If the training is finished the process ends and performance is reported to the developer at <NUM>. If the training is not finished the process moves back to <NUM> and continues conducting the training dialogs through the process of <FIG> until training is finished.

In an alternate embodiment, the batch of dialogs used for improvement may include dialogs collected in previous batches. In a further embodiment, some iterations of <FIG> may skip collecting new dialogs, and instead rely on dialogs collected from past batches.

Referring to <FIG>, is a flow diagram illustrating improvement operations for RL operations performed by an example system at operation <NUM> of <FIG>. At <NUM> the probabilities of masked actions are set to zero. At <NUM>, a constant is added to all of the probabilities. The constant is added to prevent the logarithm and its derivative of the masked actions from being zero and undefined later in the process. The constant may be added to all action probabilities prior to taking the logarithm. This causes weights to have a zero derivative, i.e., no effect, with respect to masked actions, and other actions to be unaffected because the derivative of an additive constant is zero.

At <NUM>, the gradients of the action probabilities for each turn with respect to weights are determined. At <NUM>, adjustment of the weights is performed in view of gradients, the return of dialog, and the estimated average of the current model. The gradients for each dialog may be multiplied by a "step". The step may correspond to the quantity (R_n-B) where R_n is the observed return for dialog n, and B is the estimated average return of the current neural network. B may be computed by averaging R_n in the current batch, choosing a constant, or using some form of importance sampling, such as weighted importance sampling. When importance sampling is used, either the dialogs in the current batch may be used, the most recent K dialogs, or all dialogs observed to date.

When the weights have been adjusted the updated recurrent neural network is deployed as neural network <NUM> at operation <NUM> of <FIG>.

Performance of reinforcement learning may be periodically reported to the domain expert. Also, constraints may be added which ensure that the training dialogs in the corpus are always re-constructed. For example, if the updated neural network <NUM> fails to re-generate a training dialog, then SL gradient descent can be applied on that dialog until it is re-generated.

Referring now to <FIG>, therein is a simplified block diagram of an example computing device <NUM>. System <NUM> of <FIG> may be implemented on a device such as computing device <NUM>. Computing device <NUM> may include a server <NUM> having processing unit <NUM>, a memory <NUM>, network interfaces <NUM>, and developer interfaces <NUM>. Memory <NUM> may be implemented as any type of computer readable storage media, including non-volatile and volatile memory. Memory <NUM> is shown as including extraction programs <NUM>, controller code <NUM>, and neural network code <NUM>. Server <NUM> and processing unit <NUM> may comprise one or more processors, or other control circuitry, or any combination of processors and control circuitry. Server <NUM> and processing unit <NUM> provide overall control of computing device <NUM> to implement the functions of an entity extractor, recurrent neural network, and controller according to the disclosed embodiments. Computing device <NUM> may also be configured to provide developer interface <NUM> which may be configured, for example, as developer interface <NUM> of <FIG>.

Developer interface <NUM> may be configured to allow a developer overall control of management and training of computing device <NUM>. Developer interface <NUM> may be a user interface, such as a web interface, or any other application which guides the developer. In one implementation, developer interface <NUM> allows the developer to enter a new dialog. As the dialog is entered, the developer interface <NUM> may indicate what the next system response under the current model would be, or may indicate a ranking of multiple system responses ordered by their scores. Differences between the model output and the desired output help the developer to understand the strengths and weaknesses of the current model. Another section of the developer interface <NUM> may allow the developer to browse through the dialogs which have been entered so far, highlighting dialogs which disagree with the current model.

Developer interface <NUM> may also handle entity extraction. In one implementation, the developer interface <NUM> may provide a pointer to an external entity extraction service. In another implementation, entity extraction labeling and refinement may be performed, for example by labeling entities using the same developer interface <NUM> used for entering sample dialogs.

The developer interface <NUM> may also allow a developer to interface computing device <NUM> with custom computer code. In one implementation, the custom code can be entered directly into the developer interface <NUM> on server <NUM>. In another implementation, the custom code may run on a separate server, for example, on a server hosted by the developer. This latter implementation involves the developer hosting their own webserver, but provides additional control, and allows the developer to avoid disclosing the implementation of their code at the developer interface. In either alternative, example starter code may show how to structure the code and what functions to implement.

The developer interface <NUM> may also allow the developer to manage the RL training. The developer may specify properties of reward signals, indicate whether the RL should be active or not, view graphs of performance over time, or mange other functions of the RL training.

Additionally, the developer interface <NUM> may allow the developer to set configuration options. These configuration options may specify the set of API calls that are available. (The API calls may also be provided or discovered programmatically). The configuration options may also include details about the HTTP endpoint on which the bot is available, authentication and subscription options, and general administration configuration options, such as which users have access to edit the bot.

System <NUM> is shown as an implementation that includes server <NUM> as a single server for performing operations of the embodiments according to programs and code in memory <NUM>. However, server <NUM> and memory <NUM> may be understood as representative of server functions or memory provided by one or more servers or computing devices, or storage devices, that may be co-located or geographically dispersed and that may provide the functions of the entity extraction <NUM>, the neural network <NUM>, and the controller <NUM> for other implementations of system <NUM>. For example, the controller code may be implemented on a separate server separate from the server on which the extractor and neural network code are implemented. The term server as used in this disclosure is used generally to include any computing devices or communications equipment.

The example embodiments disclosed herein may be described in the general context of processor-executable code or instructions stored on memory that may comprise one or more computer readable storage media (e.g., tangible non-transitory computer-readable storage media such as memory <NUM>). As should be readily understood, the terms "computer-readable storage media" or "non-transitory computer-readable media" include the media for storing of data, code and program instructions, such as memory <NUM>, and do not include portions of the media for storing transitory propagated or modulated data communication signals.

While implementations have been disclosed and described as having functions implemented on particular computing devices, server devices, and/or wireless devices operating in a network, one or more of the described functions for the devices may be moved between the devices and implemented on a different one of the devices than shown in the figures, or on different types of equipment.

While the functionality disclosed herein has been described by illustrative example using descriptions of the various components and devices of embodiments by referring to functional blocks and processors or processing units, controllers, and memory including instructions and code, the functions and processes of the embodiments may be implemented and performed using any type of processor, circuitry or combinations of processors and/or circuitry and code. This may include, at least in part, one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. Use of the term processor or processing unit in this disclosure is meant to include all such implementations.

Claim 1:
A dialog-based system configured for providing an interface to application functionality of a device (<NUM>) that allows a user to interact with and receive services from an application server (<NUM>), wherein the device (<NUM>) comprises a home appliance, the system comprising:
one or more processors (<NUM>); and,
memory (<NUM>) in communication with the one or more processors, the memory comprising code (<NUM>, <NUM>, <NUM>), that when executed, causes the one or more processor to control the system to:
control an interactive dialog with a user;
determine marked-up text from an input text (<NUM>) of the user wherein an entity extractor (<NUM>) extracts one or more entities from the input text (<NUM>) of the user to form the marked-up text, wherein the one or more entities comprise one or more of dates, times, and locations;
determine, by a controller implemented by the execution of the code on the one or more processors (<NUM>), a mask over available actions from the marked-up text (<NUM>), wherein the mask indicates which actions of a set of actions are allowed and which are disallowed based on which entity types were extracted from the input text (<NUM>);
provide, by the controller, the mask and a set of features to a recurrent neural network (<NUM>), wherein the set of features include the input text (<NUM>) of the user, an indication of the types of entities extracted from the input text, an indication of a previous action if a previous action took place and at least one feature associated with the previous action if the previous action took place (<NUM>, <NUM>);
receive, by the controller, an indication of a selected action from the recurrent neural network (<NUM>);
update an internal state of the controller based on the indication of a selected action (<NUM>); and,
initiate, by the controller, the selected action (<NUM>, <NUM>, <NUM>) to interface with an application back-end to control the home appliance.