Artificially intelligent interaction agent

A system includes a memory having instructions therein and at least one processor configured to execute the instructions to: begin control of a user-interaction session; determine a first user state; use a reinforcement learning agent to select a first motivational action; communicate the first motivational action to a user device; determine a second user state; generate a reward based at least in part on a tiered reinforcement learning reward categorization of the second user state; communicate the reward and the second user state to the reinforcement learning agent; update the reinforcement learning agent; and determine, based at least in part on whether the second user state corresponds to a goal of the user-interaction session, to wind up control of the user-interaction session.

BACKGROUND

The present disclosure relates generally to the field of cognitive computing and, more particularly, to interaction design and artificial intelligence.

According to Wikipedia, interaction design has been characterized as “the practice of designing interactive digital products, environments, systems, and services,” and is often abbreviated as IxD. According to the Interaction Design Association, “Interaction Design (IxD) defines the structure and behavior of interactive systems” and “Interaction Designers strive to create meaningful relationships between people and the products and services that they use, from computers to mobile devices to appliances and beyond.” One commentator has remarked that: “Interaction design is goal-driven design; users interact with an interface to accomplish a goal or better yet, a set of goals, whether it's buying something, getting somewhere, contacting someone, and so forth. The ultimate goal of interaction design is to design for those goals.”

Meanwhile, various abilities of machines to acquire and apply knowledge and skills have been categorized as artificial intelligence (“AI”). Machine learning has been considered to be a form of AI. Machine learning has employed algorithms and statistical models that have enabled computer systems use to perform tasks without using explicit instructions, relying on patterns and inferences instead. Supervised learning has been an area of machine learning concerned with employing mathematical models based on sample data (i.e., “training data”) that has been generated as a result of environmental responses to known stimuli (i.e., “input data”). Reinforcement learning has been an area of machine learning concerned with developing and using machines, typically referred to as “agents,” to operate in or on an environment so as to maximize some notion of cumulative reward. Unlike supervised learning, reinforcement learning has typically not relied on labelled input/output pairs or explicit corrections of sub-optimal actions. Instead, reinforcement learning has typically focused on finding balances between explorations of uncharted territory and exploitations of current knowledge.

SUMMARY

A method is disclosed. The method includes beginning control of a user-interaction session. The method also includes receiving a first communication from a user device. The method also includes determining a first user state based at least in part on the first communication from the user device. The method also includes communicating the first user state to a reinforcement learning agent. The method also includes using the reinforcement learning agent to select a first motivational action based at least in part on the first user state. The method also includes communicating the first motivational action to the user device. The method also includes receiving a second communication from the user device. The method also includes determining a second user state based at least in part on the second communication from the user device, the second user state comprising an attribute conveying a tiered reinforcement learning reward categorization. The method also includes generating a reward based at least in part on the tiered reinforcement learning reward categorization. The method also includes communicating the reward to the reinforcement learning agent. The method also includes communicating the second user state to the reinforcement learning agent. The method also includes updating the reinforcement learning agent. The method also includes determining, based at least in part on whether the second user state corresponds to a goal of the user-interaction session, to wind up control of the user-interaction session.

A system is also disclosed. The system includes a memory having instructions therein and at least one processor in communication with the memory. The at least one processor is configured to execute the instructions to begin control of a user-interaction session. The at least one processor is also configured to execute the instructions to receive a first communication from a user device. The at least one processor is also configured to execute the instructions to determine a first user state based at least in part on the first communication from the user device. The at least one processor is also configured to execute the instructions to communicate the first user state to a reinforcement learning agent. The at least one processor is also configured to execute the instructions to use the reinforcement learning agent to select a first motivational action based at least in part on the first user state. The at least one processor is also configured to execute the instructions to communicate the first motivational action to the user device. The at least one processor is also configured to execute the instructions to receive a second communication from the user device. The at least one processor is also configured to execute the instructions to determine a second user state based at least in part on the second communication from the user device, the second user state comprising an attribute conveying a tiered reinforcement learning reward categorization. The at least one processor is also configured to execute the instructions to generate a reward based at least in part on the tiered reinforcement learning reward categorization. The at least one processor is also configured to execute the instructions to communicate the reward to the reinforcement learning agent. The at least one processor is also configured to execute the instructions to communicate the second user state to the reinforcement learning agent. The at least one processor is also configured to execute the instructions to update the reinforcement learning agent. The at least one processor is also configured to execute the instructions to determine, based at least in part on whether the second user state corresponds to a goal of the user-interaction session, to wind up control of the user-interaction session.

A computer program product is also disclosed. The computer program product includes a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by at least one processor to cause the at least one processor to begin control of a user-interaction session. The program instructions are also executable by the at least one processor to cause the at least one processor to receive a first communication from a user device. The program instructions are also executable by the at least one processor to cause the at least one processor to determine a first user state based at least in part on the first communication from the user device. The program instructions are also executable by the at least one processor to cause the at least one processor to communicate the first user state to a reinforcement learning agent. The program instructions are also executable by the at least one processor to cause the at least one processor to use the reinforcement learning agent to select a first motivational action based at least in part on the first user state. The program instructions are also executable by the at least one processor to cause the at least one processor to communicate the first motivational action to the user device. The program instructions are also executable by the at least one processor to cause the at least one processor to receive a second communication from the user device. The program instructions are also executable by the at least one processor to cause the at least one processor to determine a second user state based at least in part on the second communication from the user device, the second user state comprising an attribute conveying a tiered reinforcement learning reward categorization. The program instructions are also executable by the at least one processor to cause the at least one processor to generate a reward based at least in part on the tiered reinforcement learning reward categorization. The program instructions are also executable by the at least one processor to cause the at least one processor to communicate the reward to the reinforcement learning agent. The program instructions are also executable by the at least one processor to cause the at least one processor to communicate the second user state to the reinforcement learning agent. The program instructions are also executable by the at least one processor to cause the at least one processor to update the reinforcement learning agent. The program instructions are also executable by the at least one processor to cause the at least one processor to determine, based at least in part on whether the second user state corresponds to a goal of the user-interaction session, to wind up control of the user-interaction session.

DETAILED DESCRIPTION

As used within the written disclosure and in the claims, the terms “including” and “comprising” (and inflections thereof) are used in an open-ended fashion, and thus should be interpreted to mean “including, but not limited to.” Unless otherwise indicated, as used throughout this document, “or” does not require mutual exclusivity, and the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

A “module” or “unit” (and inflections thereof) as referenced herein comprises one or more hardware or electrical components such as electrical circuitry, processors, and memory that may be specially configured to perform a particular function. The memory may comprise volatile memory or non-volatile memory that stores data such as, but not limited to, computer executable instructions, machine code, and other various forms of data. The module or unit may be configured to use the data to execute one or more instructions to perform one or more tasks. In certain instances, a module or unit may also refer to a particular set of functions, software instructions, or circuitry that is configured to perform a specific task. For example, a module or unit may comprise software components such as, but not limited to, data access objects, service components, user interface components, application programming interface (“API”) components; hardware components such as electrical circuitry, processors, and memory; and/or a combination thereof. As referenced herein, computer executable instructions may be in any form including, but not limited to, machine code, assembly code, and high-level programming code written in any programming language.

Also, as used herein, the term “communicate” (and inflections thereof) means to receive and/or transmit data or information over a communication link. The communication link may include both wired and wireless links, and may comprise a direct link or may comprise multiple links passing through one or more communication networks or network devices such as, but not limited to, routers, firewalls, servers, and switches. The communication networks may comprise any type of wired or wireless network. The networks may include private networks and/or public networks such as the Internet. Additionally, in some embodiments, the term communicate may also encompass internal communication between various components of a system and/or with an external input/output device such as a keyboard or display device.

Also, as used herein, the term “superuser-defined” (and inflections thereof) means provided by a system developer, system administrator, superuser, or other such extrinsic human or machine agent. Inputting, receiving, and/or otherwise suitably incorporating superuser-defined data, a superuser-defined parameter, a superuser-defined relationship, a superuser-defined rule, and/or any other superuser-defined quantity into any subject matter in accordance with aspects of the present disclosure may comprise (at any time prior to use of such quantity in accordance with aspects of the present disclosure) encoding such quantity into a suitable memory, incorporating such quantity into a suitable module or unit, selecting such quantity from a suitable drop-down or other menu, receiving such quantity as a suitable textual or audible input, receiving a suitable communication of such quantity from an external device, and/or any other one or more suitable approaches to incorporating such quantity into the subject matter. Further, as used herein, the term “superuser” refers to any system developer, system administrator, superuser, or other such extrinsic human or machine agent that may cause such a superuser-defined quantity to be inputted, received, and/or otherwise suitably incorporated in accordance with aspects of the present disclosure.

FIG. 1is a block diagram illustration of a network environment100in accordance with aspects of the present disclosure. The network environment100includes a network112, one or more user devices118, one or more remote data-storage modules124, one or more server devices130, and one or more users136. The network112comprises any type of network that enables the one or more server devices130to communicate with each other and with the one or more user devices118as well as with the one or more remote data-storage modules124. For example, the network112may comprise one or more wired and/or wireless networks such as, but not limited to, one or more radio networks (e.g., cellular network or mobile network), one or more local area networks (“LANs”), one or more wide area networks (“WANs”), one or more metropolitan area networks (“MANs”), etc. The network112may also comprise one or more private networks and/or one or more public networks (such as, but not limited to, the Internet).

Each of the one or more user devices118is communicatively coupled to the network112and (through the network112) to the one or more server devices130and the one or more remote data-storage modules124. Each of the one or more user devices118comprises any type of device that allows the one or more users136to audibly, textually, or otherwise suitably interact with the one or more server devices130through the network112. Non-limiting examples of one of the one or more user devices118include a personal computer (desktop or laptop), a mobile device (e.g., personal digital assistant (“PDA”), smart phone, tablet, etc.), and a cognitive voice assistant device (e.g., Amazon's Alexa®, a Google Home® device, etc.). In some embodiments, the one or more user devices118may comprise a corresponding one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 1, but seeFIG. 5).

Each of the one or more remote data-storage modules124is communicatively coupled to the network112and (through the network112) to each other and to the one or more server devices130and the one or more user devices118. The one or more remote data-storage modules124are configured to (alone or in combination) store and provide access to various data that may be generated, modified, and/or used in accordance with aspects of the present disclosure. In some embodiments, the one or more remote data-storage modules124may comprise a corresponding one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 1, but seeFIG. 5).

Each of the one or more server devices130is communicatively coupled to the network112and (through the network112) to each other and to the one or more remote data-storage modules124and the one or more user devices118. Each of the one or more server devices130comprises any type of device that can (alone or in combination with one or more other components of the network environment100) implement an interaction agent module200in accordance with aspects of the present disclosure (the interaction agent module200per se is not explicitly illustrated inFIG. 1, but seeFIG. 2). In some embodiments, the one or more server devices130may comprise a corresponding one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated in FIG.1, but seeFIG. 5). In some embodiments, the one or more server devices130may comprise one or more suitable computers, machines, modules, and/or devices provided by an Internet service provider.

FIG. 2is a block diagram illustration of the interaction agent module (“IAM”)200in accordance with aspects of the present disclosure. The IAM200is configured to interact (through the network112and the one or more user devices118) with the one or more users136in accordance with aspects of the present disclosure (the network112, the user devices118, and the users136are not shown inFIG. 2, but seeFIG. 1). In some embodiments, the IAM200may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5). In the depicted example, the IAM200comprises one or more of the server devices130(the server devices130per se are not explicitly illustrated inFIG. 2, but seeFIG. 1).

The IAM200includes a master control module (“MCM”)220. The MCM220is configured to coordinate operations of other components of the IAM200and to acquire, process, and move data in support of such operations in accordance with aspects of the present disclosure. In some embodiments, the MCM220may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5).

The IAM200also includes a network interface module (“NIM”)236. The NIM236is communicatively coupled to, among other things, the MCM220and the network112(the network112per se is not explicitly illustrated inFIG. 2, but seeFIG. 1). It should be appreciated that the NIM236is also communicatively coupled (through the network112) to any number of the one or more user devices118, any number of the one or more remote data-storage modules124, and any number of the one or more server devices130(the user devices118, the remote data-storage modules124, and the server devices130are not shown inFIG. 2, but seeFIG. 1). The NIM236is configured to communicatively couple and act as an interface between the MCM220, various other components of the IAM200, the network112, and (through the network112) various other components of the network environment100in accordance with aspects of the present disclosure. In some embodiments, the NIM236may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5).

The IAM200also includes a local data-storage module (“LDSM”)252. The LDSM252is communicatively coupled to, among other things, the MCM220and the NIM236. It should be appreciated that the LDSM252is also communicatively coupled (through the NIM236and the network112) to any number of the one or more user devices118, any number of the one or more remote data-storage modules124, and any number of the one or more server devices130(the user devices118, the remote data-storage modules124, and the server devices130are not shown inFIG. 2, but seeFIG. 1). The LDSM252is configured to store and provide access to various data that may be generated, modified, and/or used in accordance with aspects of the present disclosure. In some embodiments, the LDSM252may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5).

The IAM200also includes a Markov decision process module (“MDPM”)260. The MDPM260is configured to implement a finite-state, overridable, tiered-rewarding, effectively continuous-sojourn-time Markov decision process300in accordance with aspects of the present disclosure (the process300per se is not explicitly illustrated inFIG. 2, but seeFIG. 3). In some embodiments, the MDPM260may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5).

The MDPM260includes a reinforcement learning agent module (RLAM”)268. The RLAM268is configured to implement reinforcement learning agent tasks304of the process300in accordance with aspects of the present disclosure (the process300and the reinforcement learning agent tasks304are not shown inFIG. 2, but seeFIG. 3). In some embodiments, the RLAM268may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5).

The MDPM260also includes a user interface module (“UIM”)276. The UIM276is communicatively coupled to, among other things, the MCM220, the NIM236, and the LDSM252. It should be appreciated that the UIM276is also communicatively coupled (through the NIM236and the network112) to any number of the one or more user devices118, any number of the one or more remote data-storage modules124, and any number of the one or more server devices130(the user devices118, the remote data-storage modules124, and the server devices130are not shown inFIG. 2, but seeFIG. 1). The UIM276is configured to implement user interface tasks308of the process300in accordance with aspects of the present disclosure (the process300and the user interface tasks308are not shown inFIG. 2, but seeFIG. 3). In some embodiments, the UIM276may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5).

The MDPM260also includes an override control module (“OCM”)284. The OCM284is communicatively interposed between the RLAM268and the UIM276. The OCM284is configured to implement override control tasks312of the process300in accordance with aspects of the present disclosure (the process300and the override control tasks312are not shown inFIG. 2, but seeFIG. 3). In some embodiments, the OCM284may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5).

The MDPM260also includes a decision epoch control module (“DECM”)292. The DECM292is communicatively coupled to the RLAM268, the UIM276, and the OCM284. The DECM292is configured to coordinate operations of the RLAM268, the UIM276, and the OCM284and to acquire, process, and move data in support of such operations in accordance with aspects of the present disclosure. In some embodiments, the DECM292may comprise one or more of a data processing system like the data processing system600(the data processing system600per se is not explicitly illustrated inFIG. 2, but seeFIG. 5).

FIG. 3is a process flow diagram illustration of the finite-state, overridable, tiered-rewarding, effectively continuous-sojourn-time Markov decision process300in accordance with aspects of the present disclosure. The process300is configured to control a current goal-oriented user-interaction session. In some embodiments, the goal of the user-interaction session (“session goal”) may be to get the user to make a purchase. In some embodiments, the session goal may be to get the user to make a charitable donation, respond to a survey, open a financial account, or enter into a lease agreement. In other embodiments, the session goal may be any other suitable user behavior or response. In some instances, one or more steps of the process300may be performed by one or more of the server devices130(the server devices130per se are not explicitly illustrated inFIG. 3, but seeFIG. 1), the IAM200(the IAM200per se is not explicitly illustrated inFIG. 3, but seeFIG. 2), and/or one or more of the other systems, components, methods, and/or processes described herein. For clarity of exposition, the following description(s) of the process300may refer to one or more of such systems, components, methods, and/or processes. Nevertheless, it should be appreciated that the process300and/or any one or more of its particular steps may be performed by any suitable system(s), component(s), method(s), and/or process(es). It should also be appreciated that the process300may be performed concurrently or substantially concurrently with any other method(s) and/or process(es) described herein.

The process300includes the reinforcement learning agent tasks304. The reinforcement learning agent tasks304are configured to provide a reinforcement learning agent that: (1) receives communications of governed user states (as environmental state inputs) from the override control tasks312; (2) receives communications of governed rewards (as reward inputs) from the override control tasks312; (3) selects a respective motivational action (from a superuser-defined motivational action space) corresponding to each governed user state (except at a final decision epoch364, described further below), based on the governed user states and the governed rewards; (4) communicates the selected motivational actions (as action outputs) to the override control tasks312; and (5) retains what the reinforcement learning agent learns from the user-interaction session (as baseline learning for the next user-interaction session). In some embodiments, the reinforcement learning agent may comprise one or more model-free reinforcement learning engines. Non-limiting examples of suitable model-free reinforcement learning engines include a deep Q network (“DQN”), a double deep Q network (“DDQN”), a dueling double deep Q network (“Dueling DDQN”), and asynchronous advantage actor-critic agents (“A3C”). In some embodiments, the reinforcement learning agent may comprise any other one or more suitable model-free reinforcement learning engines and/or any one or more suitable model-based machine learning engines.

The superuser-defined motivational action space comprises a finite number, k (not explicitly illustrated), of particular sets, A, of superuser-defined motivational actions. Each set, A, of superuser-defined motivational actions comprises an identification of each of a finite number of superuser-defined motivational actions available for influencing a finite number, k (not explicitly illustrated), of respective particular superuser-defined user states, S (described further below in connection with the user interface tasks308), within a superuser-defined finite user state space. Non-limiting examples of possible motivational actions include: (1) communicating zero or more motivational, promotional, solicitous, and/or inquisitive in-app messages, emails, graphical displays, drop-down lists, audio messages, and/or videos to the user; and/or (2) connecting the user to a human or machine help agent.

The process300also includes the user interface tasks308. The user interface tasks308are configured to: (1) receive communications of superuser control commands (described further below in connection with the override control tasks312) from a respective user device (e.g., one of the user devices118, shown inFIG. 1); (2) communicate the superuser control commands to the override control tasks312; (3) receive communications regarding current user states from a respective user device (e.g., one of the user devices118, shown inFIG. 1); (4) discern the respective user states and assign their respective runtime variable classification attributes (discussed further below) from the communications regarding the current user states (e.g., by using one or more suitable lookup tables, translation tables, or other reference resources to determine which of the communications correspond to which of the particular superuser-defined user states, S); (5) communicate the discerned user states to the override control tasks312; (6) generate a respective reward corresponding to each discerned user state (except at an initial decision epoch316, described further below); (7) communicate the rewards to the override control tasks312; (8) receive communications of governed motivational actions from the override control tasks312; and (9) communicate the governed motivational actions to the respective user device.

As noted above in connection with the reinforcement learning agent tasks304, the superuser-defined finite user state space comprises the finite number, k (not explicitly illustrated), of the respective particular superuser-defined user states, S. Non-limiting examples of possible superuser-defined user states include: (1) one or more mouse clicks that may be made by the user during the user-interaction session; (2) one or more navigation paths that may be traversed by the user during the user-interaction session; (3) one or more screen images that may be uploaded by or otherwise associated with the user during the user-interaction session; (4) one or more data profiles that may be associated with the user during the user-interaction session; and (5) current usage data that may be associated with the user during the user-interaction session.

Each superuser-defined user state, S, also comprises a runtime variable classification attribute. Each of these classification attributes conveys a superuser-defined tiered reinforcement learning reward categorization of the respective superuser-defined user state, S, as either a “goal state,” a “milestone state,” a “simply positive state,” a “simply negative state,” or an “abort state,” such that: (1) the goal state categorization corresponds to the superuser-defined user state, S, for which a superuser has predetermined the session goal will be achieved; (2) each respective milestone state categorization (if any) corresponds to a respective superuser-defined user state, S, at which the session goal will not be achieved, but, nevertheless, some suitably significant advancement toward the session goal will be reached (there may be zero or more milestone states); (3) each respective simply positive state categorization (if any) corresponds to a respective superuser-defined user state, S, other than a goal state or a milestone state, at which the session goal will not be achieved, but, nevertheless, some relatively less significant advancement toward the session goal will be reached (there may be zero or more simply positive states); (4) each respective negative state categorization (if any) corresponds to a respective superuser-defined user state, S, that is neither a goal state, nor a milestone state, nor a simply positive state, nor an abort state, at which some divergence away from the session goal has occurred (there may be zero or more negative states); and (5) each respective abort state categorization (if any) corresponds to a respective superuser-defined user state, S, that is not a goal state yet is nevertheless a state at which the process300will still wind up control of the user-interaction session (there may be zero or more abort states). Non-limiting examples of possible goal states include: (1) a user mouse click corresponding to a credit card payment for a magazine subscription (when the session goal is to get the user to buy a magazine subscription); and (2) a user mouse click corresponding to final submission of survey responses (when the session goal is to get the user to respond to the survey). Non-limiting examples of possible milestone states include: (1) a user entry or drop-down list selection of a desired genre (when the session goal is to get the user to buy a magazine subscription); (2) a user's submission of the user's demographic information (when the session goal is to get the user to respond to a survey); and (3) a user's acceptance of a promotional video. A non-limiting example of a possible simply positive state is a user request to peruse available magazine genres (when the session goal is to get the user to buy a magazine subscription). A non-limiting example of a negative state is a user's rejection of an offer to sell a T-shirt to the user at an undiscounted price. It should be appreciated that an abort state may be any state (other than a goal state) at which a user commands termination of the user-interaction session, at which the process300has reached a predetermined maximum number of decision epochs, or at which the process300will wind up control of the user-interaction session for any other reason.

The user interface tasks308are further configured to generate the respective rewards corresponding to the one or more respective user states that may follow an initial user state, Si(i.e., at each second or subsequent decision epoch at which the process300reaches a current user state corresponding to any one of the superuser-defined user states, S) based on the current user state and the categorization of the corresponding superuser-defined user state, S. More particularly, the user interface tasks308are configured to generate (at each second or subsequent decision epoch) each respective reward, such that: (1) when the current user state corresponds to the particular superuser-defined user state, S, that is categorized as the goal state, the reward will have a suitably relatively high-weighted, positive, superuser-defined value; (2) when the current user state corresponds to one of the superuser-defined user states, S, that is categorized as a simply positive state, the reward will have a suitably relatively low-weighted, positive, superuser-defined value; (3) when the current user state corresponds to one of the superuser-defined user states, S, that is categorized as a milestone state, the reward will have a suitably relatively medium-weighted, positive, superuser-defined value; (4) when the current user state corresponds to one of the superuser-defined user states, S, that is categorized as a negative state, the reward will have a suitably relatively low-weighted, negative, superuser-defined value; and (5) when the current user state corresponds to one of the superuser-defined user states, S, that is categorized as an abort state, the reward will have a suitably relatively high-weighted, negative, superuser-defined value.

The process300also includes the override control tasks312. The override control tasks312are configured to facilitate “baseline training” (i.e., superuser-defined training of the reinforcement learning agent prior to the user-interaction session) and to facilitate contemporaneous superuser override control of the user-interaction session. More particularly, the override control tasks312are configured to: (1) receive communications of the superuser control commands from the user interface tasks308; (2) receive communications of the discerned user states from the user interface tasks308; (3) generate the respective governed user states; (4) communicate the governed user states to the reinforcement learning agent tasks304; (5) receive communications of the rewards from the user interface tasks308; (6) generate the respective governed rewards; (7) communicate the governed rewards to the reinforcement learning agent tasks304; (8) receive communications of the selected motivational actions from the reinforcement learning agent tasks304; (9) generate the respective governed motivational actions; and (10) communicate the governed motivational actions to the user interface tasks308.

Further, the override control tasks312are configured to generate the outputs of the override control tasks312(namely, the governed user states, the governed rewards, and the governed motivational actions) based on the inputs to the override control tasks312(namely, the discerned user states, the rewards, and the selected motivational actions), based on the superuser control commands and based on superuser-defined data and/or one or more superuser-defined rules, such that: (1) when the superuser control commands call for baseline training using one or more superuser-defined “fake users,” the override control tasks312will generate the outputs of the override control tasks312using the fake users rather than by using the respective discerned user states, the respective rewards, and/or the respective selected motivational actions (i.e., the override control tasks312will effectively overwrite or override the discerned user states, the rewards, and/or the selected motivational actions with the respective superuser-defined fake user data); (2) when the superuser control commands call for baseline training using one or more superuser-defined “simulated users,” the override control tasks312will generate the outputs of the override control tasks312using the simulated users rather than by using the respective discerned user states, the respective rewards, and/or the respective selected motivational actions (i.e., the override control tasks312will effectively overwrite or override the discerned user states, the rewards, and/or the selected motivational actions with the respective superuser-defined simulated user data); (3) when the superuser control commands call for one or more contemporaneous superuser overrides, the override control tasks312will generate the outputs of the override control tasks312using the contemporaneous superuser overrides rather than by using the respective discerned user states, the respective rewards, and/or the respective selected motivational actions (i.e., the override control tasks312will effectively overwrite or override the discerned user states, the rewards, and/or the selected motivational actions with the respective contemporaneous superuser override data); and (4) when the superuser control commands do not call for any baseline training or contemporaneous superuser overrides, the override control tasks312will simply pass the inputs to the override control tasks312through as the respective outputs of the override control tasks312(i.e., the override control tasks312will generate the outputs of the override control tasks312by effectively copying the discerned user states, the rewards, and the selected motivational actions into the governed user states, the governed rewards, and the governed motivational actions, respectively).

It should be appreciated that one or more of the fake users may be based on one or more preexisting policies, rules, or other stratagems for user-interaction that have been conceived independently of the reinforcement learning agent. In some instances, a fake user may include a superuser-defined state paired with a superuser-defined action that together conform to such a preexisting policy, rule, or other stratagem for user-interaction. Such a fake user may be configured to, when the superuser control commands cause the override control tasks312to use the fake user: (1) when the responsive selected motivational action (from the reinforcement learning agent tasks304) agrees with the superuser-defined action for the fake user, generate a suitably large positive reward (which the override control tasks312will in turn output as the governed reward); and (2) when the responsive selected motivational action does not agree with the superuser-defined action for the fake user, generate a suitably large negative reward (which the override control tasks312will in turn output as the governed reward). For example, when a preexisting business rule comprises “send a notification to the user when the user adds an item to the user's shopping cart,” a fake user may be configured to: (1) provide a very high positive reward when the user interface learning tasks308communicate “item added to shopping cart” as the discerned user state and (in response) the reinforcement learning agent tasks304communicate “notify the user” as the selected motivational action; and (2) provide a very high magnitude negative reward when the user interface tasks308communicate “item added to shopping cart” as the discerned user state but (in response) the reinforcement learning agent tasks304do not communicate “notify the user” as the selected motivational action.

Meanwhile, one or more of the simulated users may be based on statistical data regarding historical behaviors of interactions with one or more humans independently of the reinforcement learning agent. For example, statistical analyses of user profiles may be used to identify typical user features or characteristics. Further, such user feature data may be extracted and categorized to create representative user personas. Thereafter, a representative state transition matrix may be constructed for each user persona, and the simulated users may be generated based on the respective representative state transition matrices.

Referring still toFIG. 3, in operation the process300includes the initial decision epoch316, zero or more intermediate decision epochs340, and the final decision epoch364. At the initial decision epoch316, the process300begins control of the user-interaction session, at the initial user state, Si, and the reinforcement learning agent tasks304, the user interface tasks308, and the override control tasks312cooperate accordingly. In some instances, optional superuser control commands and data, SUi, may be used during the initial decision epoch316. Nevertheless, it should be appreciated that the reinforcement learning agent may begin the initial decision epoch316with learning retained from one or more previous and/or concurrent user-interaction sessions.

At the intermediate decision epochs340, the process300further controls the user-interaction session, at various respective intermediate user states, Sm, and the reinforcement learning agent tasks304, the user interface tasks308, and the override control tasks312cooperate accordingly. In some instances, various optional superuser control commands and data, SUm, may be used during any of the intermediate decision epochs340. Depending on user reactions and/or superuser inputs, any intermediate user state, Sm, may be a milestone state, a simply positive state, or a negative state (but not a goal state). However, it should be appreciated that when the goal state is the next user state following the initial user state, Si, the process300does not include any intermediate decision epochs340(i.e., in such instances the number of intermediate decision epochs340is zero).

At the final decision epoch364, the process300winds up control of the user-interaction session, at a final user state, Sf, and the reinforcement learning agent tasks304, the user interface tasks308, and the override control tasks312cooperate accordingly. In some instances, optional superuser control commands and data, SUf, may be used during the final decision epoch364. Depending on user reactions and/or superuser inputs, the final user state, Sf, may be a goal state, a milestone state, a simply positive state, or a negative state. It should be appreciated that the winding up control of the user-interaction session may include the reinforcement learning agent saving and retaining what the reinforcement learning agent has learned (as baseline learning for the next user-interaction session).

FIG. 4A-4Dare a flowchart illustration of the process300. At step408, the process300(e.g., using the DECM292) sets a decision epoch counter to 1 and sets a “previous user state” variable to a suitable nonce value. From step408, the process goes to (and continues at) step412.

At step412, the process300(e.g., using the override control tasks312) determines whether communications (e.g., from one of the user devices118) (hereinafter referred to as the “current superuser device”) indicate that superuser control over the user-interaction session is commanded. If superuser control is commanded, then the process300goes to (and continues at) step416; otherwise, the process300goes to (and continues at) step472.

At step416, the process300(e.g., using the override control tasks312) determines whether communications from the current superuser device indicate that baseline training is commanded. If baseline training is commanded, then the process300goes to (and continues at) step424; otherwise, the process300goes to (and continues at) step420.

At step420, the process300(e.g., using the override control tasks312) sets the governed user state to the contemporaneous superuser-defined user state. From step420, the process goes to (and continues at) step436.

At step424, the process300(e.g., using the override control tasks312) determines whether communications from the current superuser device indicate that using a fake user for the baseline training is commanded. If using a fake user is commanded, then the process300goes to (and continues at) step432; otherwise, the process300goes to (and continues at) step428.

At step428, the process300(e.g., using the override control tasks312) sets the governed user state to the superuser-defined simulated user state. From step428, the process goes to (and continues at) step436.

At step432, the process300(e.g., using the override control tasks312) sets the governed user state to the superuser-defined fake user state. From step432, the process goes to (and continues at) step436.

At step436, the process300communicates the governed user state to the reinforcement learning agent and the reinforcement learning agent receives the governed user state (e.g., using the override control tasks312and the reinforcement learning agent tasks304, respectively). From step436, the process goes to (and continues at) step440.

At step440, the process300(e.g., using the DECM292) determines whether the decision epoch counter is greater than 1. If the decision epoch counter is greater than 1, then the process300goes to (and continues at) step444; otherwise, the process300goes to (and continues at) step520.

At step444, the process300(e.g., using the override control tasks312) determines whether communications from the current superuser device indicate that baseline training is commanded. If baseline training is commanded, then the process300goes to (and continues at) step452; otherwise, the process300goes to (and continues at) step448.

At step448, the process300(e.g., using the override control tasks312) sets the governed reward to the contemporaneous superuser-defined reward. From step448, the process goes to (and continues at) step464.

At step452, the process300(e.g., using the override control tasks312) determines whether communications from the current superuser device indicate that using a fake user for the baseline training is commanded. If using a fake user is commanded, then the process300goes to (and continues at) step460; otherwise, the process300goes to (and continues at) step456.

At step456, the process300(e.g., using the override control tasks312) sets the governed reward to the superuser-defined simulated reward. From step456, the process goes to (and continues at) step464.

At step460, the process300(e.g., using the override control tasks312) sets the governed reward to the superuser-defined fake reward. From step460, the process goes to (and continues at) step464.

At step464, the process300communicates the governed reward to the reinforcement learning agent and the reinforcement learning agent receives the governed reward (e.g., using the override control tasks312and the reinforcement learning agent tasks304, respectively). From step464, the process goes to (and continues at) step468.

At step468, the process300(e.g., using the reinforcement learning agent tasks304) updates the reinforcement learning agent (e.g., causes the reinforcement learning agent to execute a cycle or iteration of machine learning computations in search of an optimum policy for selecting motivational actions for driving the user toward the session goal). From step468, the process300goes to (and continues at) step520.

At step472, the process300(e.g., using the user interface tasks308) receives communications (e.g., from one of the user devices118) (hereinafter referred to as the “current user device”) regarding the current user state. From step472, the process300goes to (and continues at) step476.

At step476, the process300(e.g., using the user interface tasks308) discerns the user state from the communications regarding the current user state. From step476, the process300goes to (and continues at) step480.

At step480, the process300(e.g., using the DECM292) determines whether the discerned user state equals the previous user state (i.e., the process300determines whether the user state has changed). If the discerned user state equals the previous user state, then the process300goes to (and continues at) step484; otherwise, the process300goes to (and continues at) step496.

At step484, the process300(e.g., using the DECM292) determines whether a superuser-defined user-response time limit has expired. If the user-response time limit has expired, then the process300goes to (and continues at) step488; otherwise, the process300goes to (and continues at) step492.

At step488, the process300(e.g., using the DECM292) resets a user-response timer. From step488, the process300goes to (and continues at) step496.

At step492, the process300(e.g., using the DECM292) suitably decrements the user-response timer. From step492, the process300goes to (and continues at) step472.

At step496, the process300(e.g., using the override control tasks312) sets the governed user state to the discerned user state. From step496, the process goes to (and continues at) step500.

At step500, the process300(e.g., using the DECM292) determines whether the decision epoch counter is greater than 1. If the decision epoch counter is greater than1, then the process300goes to (and continues at) step504; otherwise, the process300goes to (and continues at) step520.

At step504, the process300(e.g., using the user interface tasks308) generates the reward. From step504, the process300goes to (and continues at) step508.

At step508, the process300(e.g., using the override control tasks312) sets the governed reward to the generated reward. From step508, the process goes to (and continues at) step512.

At step512, the process300communicates the governed reward to the reinforcement learning agent and the reinforcement learning agent receives the governed reward (e.g., using the override control tasks312and the reinforcement learning agent tasks304, respectively). From step512, the process goes to (and continues at) step516.

At step516, the process300(e.g., using the reinforcement learning agent tasks304) updates the reinforcement learning agent. From step516, the process300goes to (and continues at) step520.

At step520, the process300(e.g., using the DECM292) determines whether the discerned user state is the goal state. If the discerned user state is the goal state, then the process300goes to (and winds up at) step576; otherwise, the process300goes to (and continues at) step524.

At step524, the process300(e.g., using the DECM292) determines whether the decision epoch counter equals a superuser-defined maximum number. If the decision epoch counter equals the maximum number, then the process300goes to (and winds up at) step576; otherwise, the process300goes to (and continues at) step526.

At step526, the process300(e.g., using the user interface tasks308) determines whether the user has commanded termination of the user-interaction session. If the user has commanded termination of the user-interaction session, then the process300goes to (and winds up at) step576; otherwise, the process300goes to (and continues at) step528.

At step528, the process300(e.g., using the reinforcement learning agent tasks304, which provide the reinforcement learning agent) selects a motivational action. From step528, the process300goes to (and continues at) step532.

At step532, the process300(e.g., using the override control tasks312) sets the governed motivational action to the selected motivational action. From step532, the process goes to (and continues at) step536.

At step536, the process300(e.g., using the override control tasks312) determines whether communications (e.g., from one of the user devices118) (hereinafter referred to as the “current superuser device”) indicate that superuser control over the user-interaction session is commanded. If superuser control is commanded, then the process300goes to (and continues at) step540; otherwise, the process300goes to (and continues at) step560.

At step540, the process300(e.g., using the override control tasks312) determines whether communications from the current superuser device indicate that baseline training is commanded. If baseline training is commanded, then the process300goes to (and continues at) step548; otherwise, the process300goes to (and continues at) step544.

At step544, the process300(e.g., using the override control tasks312) sets the governed motivational action to the contemporaneous superuser-defined motivational action. From step544, the process goes to (and continues at) step560.

At step548, the process300(e.g., using the override control tasks312) determines whether communications from the current superuser device indicate that using a fake user for the baseline training is commanded. If using a fake user is commanded, then the process300goes to (and continues at) step556; otherwise, the process300goes to (and continues at) step552.

At step552, the process300(e.g., using the override control tasks312) sets the governed motivational action to the superuser-defined simulated motivational action. From step552, the process goes to (and continues at) step560.

At step556, the process300(e.g., using the override control tasks312) sets the governed motivational action to the superuser-defined fake motivational action. From step556, the process goes to (and continues at) step560.

At step560, the process300(e.g., using the user interface tasks308) communicates the governed motivational action to the current user device. From step560, the process goes to (and continues at) step564.

At step564, the process300(e.g., using the current user device) conveys the governed motivational action to the respective user. From step564, the process goes to (and continues at) step568.

At step568, the process300(e.g., using the DECM292) increments the decision epoch counter. From step568, the process300goes to (and continues at) step572.

At step572, the process300(e.g., using the DECM292) sets the previous user state to the discerned user state. From step572, the process goes to (and continues at) step412.

At step576, the process300winds up control of the user-interaction session.

FIG. 5is a block diagram illustration of a hardware architecture of a data processing system600in accordance with aspects of the present disclosure. In some embodiments, one or more of the systems and/or components described herein (e.g., the network112, the one or more user devices118, the one or more remote data-storage modules124, the one or more server devices130, etc.) may be implemented using a corresponding one or more of the data processing system600. Moreover, the data processing system600may be configured to store and execute one or more instructions of one or more of the methods and/or any other processes described herein.

The data processing system600employs a hub architecture including north bridge and memory controller hub (“NB/MCH”)606and south bridge and input/output (“I/O”) controller hub (“SB/ICH”)610. Processor(s)602, main memory604, and graphics processor608are connected to NB/MCH606. Graphics processor608may be connected to NB/MCH606through an accelerated graphics port (“AGP”). A computer bus, such as bus632or bus634, may be implemented using any type of communication fabric or architecture that provides for a transfer of data between different components or devices attached to the fabric or architecture.

Network adapter616connects to SB/ICH610. Audio adapter630, keyboard and mouse adapter622, modem624, read-only memory (“ROM”)626, hard disk drive (“HDD”)612, compact disk read-only memory (“CD-ROM”) drive614, universal serial bus (“USB”) ports and other communication ports618, and peripheral component interconnect/peripheral component interconnect express (“PCI/PCIe”) devices620connect to SB/ICH610through bus632and bus634. PCI/PCIe devices may include, for example, Ethernet adapters, add-in cards, and personal computing (“PC”) cards for notebook computers. PCI uses a card bus controller, while PCIe does not. ROM626may comprise, for example, a flash basic input/output system (“BIOS”). Modem624or network adapter616may be used to transmit and receive data over a network.

HDD612and CD-ROM drive614connect to SB/ICH610through bus634. HDD612and CD-ROM drive614may use, for example, an integrated drive electronics (“IDE”) or serial advanced technology attachment (“SATA”) interface. In some embodiments, the HDD612may be replaced by other forms of data storage devices including, but not limited to, solid-state drives (“SSDs”). A super I/O (“SIO”) device628may be connected to SB/ICH610. SIO device628may comprise a chip on the motherboard that is configured to assist in performing less demanding controller functions for the SB/ICH610such as controlling a printer port, controlling a fan, and/or controlling the small light emitting diodes (“LEDS”) of the data processing system600.

The data processing system600may include a single processor602or may include a plurality of processors602. Additionally, processor(s)602may have multiple cores. In some embodiments, data processing system600may employ a large number of processors602that include hundreds or thousands of processor cores. In some embodiments, the processors602may be configured to perform a set of coordinated computations in parallel.

An operating system is executed on the data processing system600using the processor(s)602. The operating system coordinates and provides control of various components within the data processing system600. Various applications and services may run in conjunction with the operating system. Instructions for the operating system, applications, and other data are located on storage devices, such as one or more of the HDD612, and may be loaded into main memory604for execution by processor(s)602. In some embodiments, additional instructions or data may be stored on one or more external devices. The processes described herein for the illustrative embodiments may be performed by processor(s)602using computer usable program code, which may be located in a memory such as, for example, main memory604, ROM626, or in one or more peripheral devices.