Patent Description:
In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

Some reinforcement learning systems select the action to be performed by the agent in response to receiving a given observation in accordance with an output of a neural network.

Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. <NPL>, discloses a machine learning system which learns to optimize a sequence of "imagined" internal simulations over predictive models of the world.

This specification generally describes a reinforcement learning system that selects actions to be performed by a reinforcement learning agent interacting with an environment. A definition of the subject matter is provided by claim <NUM>. In order for the agent to interact with the environment, the system receives data characterizing the current state of the environment and selects an action to be performed by the agent in response to the received data. Data characterizing a state of the environment will be referred to in this specification as an observation.

In a comparative example which does not fall under the scope of any of the claims, the environment is a simulated environment and the agent is implemented as one or more computer programs interacting with the simulated environment. For example, the simulated environment may be a video game and the agent may be a simulated user playing the video game. As another example not falling under the scope of any of the claims, the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent is a simulated vehicle navigating through the motion simulation. In these examples which do not fall under the scope of any of the claims, actions may be control inputs to control the simulated user or simulated vehicle. More generally the environment may be one in which a robotic control task is performed.

In implementations, the environment is a real-world environment and the agent is a mechanical robot interacting with the environment to accomplish a specific task. In a comparative example, the agent may be an autonomous or semi-autonomous vehicle navigating through the environment. The actions are control inputs to control the robot.

In one aspect, this disclosure proposes a neural network system for task learning used to select actions to be performed by a mechanical agent interacting with the real-world environment to perform a task in an attempt to achieve a specified result. The system includes a controller neural network module to receive state data, for example a state vector, and context data, and to output action data. The state data characterizes a real or imagined state of the environment. The context data defines a context for planning actions and/or proposed actions. The action data defines a real or imagined action to be performed on the environment or on an imagined version of the environment.

The system includes a model neural network module (also called here "an imagination") to receive the state and action data and to output consequent state data. The consequent state data defines a state consequent upon an action defined by the action data. The model neural network module also outputs reward data defining a modelled reward.

The system includes a manager network module to receive the state data and the context data and to output route data. The route data defines whether the system is to execute an action or to imagine. "Imagine" means generating consequent state data for one or more consequent states (that is, states of the environment which are predictions of a result from the agent taking specific actions proposed by the controller) and reward data describing rewards associated with the consequent states.

The system also includes a memory to store the context data. The context data is derived from at least the state data or the consequent state data, action data for a real and/or imagined action, and from the reward data. For example the context data may be an embedding of these data. The context data may also be derived from previous context data. The context data may further be derived from auxiliary data such as a number of actions taken, a number of imagination rollouts performed, and the like. The memory may comprise a sequential state generation neural network such as a LSTM (Long Short Term Memory) neural network.

When the route data defines that the system is to imagine, state data for an imagined state (denoted later in this document as ŝj,k, where j is a step index which indicates the most recent time for which the neural network system has received state data, sj, and k indicates a number of iterations (imagination steps) which the neural network system has used to produce ŝj,k using sj) is provided to the controller neural network module to generate imagined action data. The imagined action data and the state data is provided to the model neural network module. The model neural network module then generates imagined consequent state data. Context data is derived using the consequent state data for storage in the memory.

The state data may comprise state data for a current action step, for example an action having a step index j. Thus in one example the state data may be data for state j (initially the imagined state may be reset to state j). As a comparative example, one-step imagination may be employed and the predicted consequent state from the model may be used to update the imagined state to the next step. In the present system, the state data provided to said controller neural network module comprises imagined state data for an imagined state k action steps ahead of a current action step. Optionally the manager network module may build an imagination tree by additionally determining one or more previous imagined states from which imagination is to proceed.

A method of training the system involves separately training the model neural network module, and the manager network module, and jointly training the controller neural network module and memory. The training may be based upon a loss function include a term representing a computational cost of imagining using the system.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The system can be used for continuous control applications where there is no finite search tree to consider. The system, in particular the manager module, can learn to decide whether the agent should keep planning by imagining actions or if it is ready to act, and optionally can also decide from which state to imagine. Both these abilities contribute to achieving good performance efficiently. Experimentally, we observed a clear advantage, with a uniform increase in rewards achieved by the agent from taking a fixed number of actions, with an increasing number of imagination steps.

The system can be used on almost any model which can be run forwards and is differentiable so that a policy gradient can be computed. This includes real-world control problems, such as controlling complex, non-linear dynamical systems. One class of comparative examples for which this system has been found to be successful is those in which the agent is an autonomous or semi-autonomous vehicle navigating through the environment. In comparative examples the real-world environment may be a manufacturing plant or service facility, the observations may relate to operation of the plant or facility, for example to resource usage such as power consumption, and the agent may control actions or operations in the plant/facility, for example to reduce resource usage.

The system can better distinguish between similar observed states by using the model to roll out forwards to distinguish between the effects of actions. It can also improve handling of examples (states) which are different to those encountered during its training. More particularly the ability to learn faster may reduce the amount of training data, and hence memory requirements, needed to achieve a given level of performance. As a corollary, the described systems may also require reduced computing resources to achieve a given level of performance, since this may be achieved with less training than with previous techniques.

Implementations of the system can also strike a balance between the computational cost of imagining and external task performance.

This specification generally describes a reinforcement learning system implemented as computer programs on one or more computers in one or more locations that selects actions to be performed by a reinforcement learning agent interacting with an environment by using a neural network. This specification also describes how such a system can adjust the parameters of the neural network.

The system has an advantage that a mechanical agent can improve its interaction with the real-world environment. It can enable for example the accomplishment of a specific task or improvement of navigation through or interaction with the environment.

In order to interact with the environment, the agent receives data characterizing the current state of the environment and performs an action from an action space, i.e., a discrete action space or continuous action space, in response to the received data. Data characterizing a state of the environment will be referred to in this specification as an observation.

The observation can be data captured by one or more sensors of the agent as it interacts with the environment, e.g., a camera, a LIDAR sensor, a temperature sensor, and so forth.

<FIG> illustrates an example of a neural network system <NUM> as proposed in the present disclosure. The neural network system <NUM> is for controlling an agent <NUM> which is acts on (or within) an environment <NUM>. The neural network system <NUM> receives, at any time j a dataset sj which is an observation of the system, and is referred to as "state data". The neural network system <NUM> generates an action aj and transmits it as a command to the agent <NUM>, which acts on the environment <NUM> and thereby obtains a reward rj which is also communicated to the neural network system <NUM>.

The neural network system <NUM> includes four major components: a manager network module <NUM> (a "manager"), a controller neural network module <NUM> (a "controller"), an imagination <NUM> (also referred to here as a "model neural network module", since it functions as a model of the environment <NUM>) and a memory <NUM>.

The neural network system <NUM> determines aj by performing an iterative process (illustrated in <FIG>, and explained in more detail below) having a number of steps. In each step, the manager <NUM> determines whether to act (i.e. generate action data to define an action by the agent <NUM>, and transmit the action data to the agent <NUM> so that the action is executed), or "imagine" (by which is meant proposing an action and evaluating consequences of the action). Depending on this determination, the manager <NUM> outputs route data, which defines whether the system is to execute the action, or to imagine. If the manager <NUM> determines that the system should act, the controller <NUM> produces the action data, and transmits it to the agent <NUM>, so that it is executed in the environment <NUM>. If the manager <NUM> determines that the system should imagine, the controller <NUM> produces action data defining an action which is evaluated by the model-based imagination <NUM>. In both cases, data resulting from each step are aggregated by the memory <NUM> and used to influence future actions. Thus, the collective activity of the components of the neural network system <NUM> supports various strategies for constructing, evaluating, and executing a plan.

The iterations of the iterative process are labelled by an integer index i, which is assumed below to run from <NUM>. The sequence of imagination steps the neural network system <NUM> performs before an action, are indexed by integer index k. During the iterative process, two types of data are generated: external data and internal data. The external data includes the observed states of the environment, sj, executed actions, aj, and obtained rewards, rj. The internal data includes: imagined states of the environment generated by the imagination <NUM>, which are denoted by ŝj,k (that is, the k-th state which is imagined when the neural network system <NUM> determines how to generate action data in response to the state sj); actions which are proposed by the controller <NUM>, which are denoted by âj,k; and rewards predicted by the imagination <NUM>, which are denoted by r̂j,k. The internal data further comprises the route data which depends upon manager's decision about whether to act or imagine (and, optionally, how to imagine), and which is denoted pj,k. The internal data further includes data defining the number of actions and imaginations which have been performed, and all other auxiliary information from each step. We denote the external and internal data for a single iteration i as di, and the history of all external and internal data up to, and including, the present iteration as, hi = (d<NUM>,. The set of all imagined states since the previous executed action is thus {ŝj,<NUM> ,. , ŝj,k}, where ŝj,<NUM>, is initialized as the current state sj of the environment <NUM>.

We now define the operation of the components of the neural network system <NUM> more precisely.

The manager <NUM> is a discrete policy which maps a history h obtained from the memory <NUM> to the route data p. We can denote the space of all possible histories as <IMG>, and so that h ∈ <IMG>. Similarly, we can denote the space of all possible route data as <IMG>, so that p ∈ <IMG>. Thus, the manager <NUM> performs the function πM : <IMG> → <IMG>. The route data p determines whether the agent will execute an action in the environment, or imagine the consequences of a proposed action.

As described below with reference to <FIG>, in the case of imagining, the route data may also define which state (which may be the real state sj or a previously imagined state) should be the starting point of the imagination (i.e. the state in relation to which the consequences of a proposed action are predicted) to imagine from. Thus, the route data may be pj,k ∈ {act, ŝj,<NUM> ,. , ŝj,k } , where "act" is the signal to generate action data to execute an action in the environment <NUM>, and if pj,k does not take this value it instead takes a value ŝj,l which is one of {ŝj,<NUM> ,. , ŝj,k} and which is a signal to propose and evaluate an action from imagined state, ŝj,l. As described below (with reference to <FIG>) there are various ways in which ŝj,l can be determined by the manager <NUM>. In one example, the manager <NUM> may be implemented as a multi-layer perceptron (MLP). In another example (particularly when the state data s is a two-dimensional array of values) the manager <NUM> may be convolutional network (i.e. a network including at least one convolutional layer).

The controller <NUM> is a contextualized action policy which maps a state s (which is member of the set S of all possible states of the environment <NUM>) and a history h, to an action, a which is a member of the set of all possible actions, denoted A. Thus, the manager <NUM> performs the function πC: S × <IMG> → A. The state s which is provided as input to the controller <NUM> is in accordance with the route data p output by the manager <NUM>. If the route data p indicates that an action is to be executed, the state provided to the controller <NUM> is sj. However, if the route data p indicates that an imagining process should be carried out, the input to the controller <NUM> is ŝj,l. In one example, the controller <NUM> may be implemented as a MLP.

The imagination <NUM> is an adaptive model of the world. It maps a state s (where s ∈ S), and an action a (where a ∈ A), to a consequent state, s' ∈ S, and a scalar reward, r which is a member of a set of possible rewards R. Thus, the imagination <NUM> performs the function I: S × A → S × R. Examples of possible forms of the imagination <NUM> are given below.

The memory <NUM> is also an adaptive system, such as a long-short-term-memory (LSTM). It recurrently aggregates the external and internal data di generated from each iteration i (where di is a member of the set D of all possible data), to update the history, i.e. hi = µ(di, hi-<NUM>), where µ is a trained adaptive function.

The method <NUM> performed by the neural network system <NUM> for a given value of j is illustrated in <FIG>.

In step <NUM>, the neural network system <NUM> receives an actual data state sj. At this time the integer value k is equal to zero.

In step <NUM>, the manager <NUM> receives the data state sj and also data from the memory <NUM>. The manager <NUM> determines whether to execute an action, or to imagine, and generates corresponding route data.

If the result of the determination was to imagine, the neural network system <NUM> passes to step <NUM>, in which the controller <NUM> uses state data and the context data to generate an action âj,k.

In step <NUM>, the imagination <NUM> receives the same state data and the action âj,k, and generates from them a consequent state, s' ∈ S, and a scalar reward, r̂j,k.

In step <NUM>, the memory <NUM> is updated, and used to generate a new output hi. The variable k is set to k+<NUM>. The method then returns to step <NUM>. This loop of steps <NUM>-<NUM> may be carried out any number of times.

However, if in step <NUM>, the route data indicates that an action is to be executed, the method <NUM> passes to step <NUM> in which the controller generates the action aj. In step <NUM> the action is transmitted to the agent <NUM> which acts on the system <NUM>, and obtains a reward rj. In step <NUM>, the memory <NUM> is updated to include the external and internal data and the method then terminates. It is repeated later for the next value of j, and this process continues until a termination condition is reached.

In summary, the method of constructing a plan involves the neural network system <NUM> choosing to propose actions and imagine consequences of the actions, and thereby build up a record of possible sequences of actions' expected quality. If a sequence of actions predicted to yield a high reward is identified, the manager <NUM> can then choose to act and the controller <NUM> can produce the appropriate actions.

Note that in variations of the embodiment, the method <NUM> may be varied, e.g. such that the controller <NUM> proposes an action at the same time as (or before) the manager <NUM> decides whether to act, so that the route data is used to determine whether the previously generated action is executed or used by the imagination <NUM>.

There are various possibilities for how to choose the state data s on which the controller <NUM> and imagination <NUM> operate in steps <NUM> and <NUM>. A first possibility (the "one-step" possibility), which is a comparative example, is for the controller <NUM> and the imagination <NUM> always to operate based on sj. Note that as the controller may be such as to generate a proposed action as a sample from a distribution, so that in different iterations k, the action âj,k is different. This strategy is illustrated in row (a) of <FIG>. In <FIG>, the circles indicate states of the environment. Lines extending down from a circle indicate an action proposed for that state by the controller <NUM>. A circle at the lower end of such a line indicates the imagined consequent state. Such a circle is shown with a dark center (as a "dark circle") until it is stored in the memory <NUM>. Once this happens, the circle is shown without a dark centre (a "light circle"). Similarly, a proposed action is shown as a dashed line until it is stored in the memory <NUM>, and then it is shown as a solid line.

Thus, for the "one step" possibility (the row (a)), the k=<NUM> iteration begins with a single state sj (indicated as a light circle) which the manager ("Ma" in <FIG>) indicates should be the basis for imagination. An action for this state is proposed by the controller <NUM> ("C"), to give a circle plus a downwardly extending dashed line. The imagination <NUM> ("I") then proposes a state, indicated as a dark circle below the dashed line. The state and line are then saved to the memory <NUM> ("Me"), so that the line is shown as a solid line, and the state is shown as a light circle.

In the "one step" possibility, the k=<NUM> and k=<NUM> iterations of row (a) each also begin from the state sj used as the starting point in the iteration k=<NUM>: a downwardly extending dashed line is added representing a new action proposed by the controller <NUM>; then a state (dark circle) is added to the lower end of the dashed line, indicating a state predicted by the imagination <NUM> if the action is implemented; and then the state and action are stored in the memory (the imaged state is now shown as a light circle, and the new action is shown as a solid line).

A second possibility (the "n step" strategy) is for the state data s on which the controller <NUM> and imagination <NUM> operate in steps <NUM> and <NUM> to be chosen as sj in iteration k=<NUM>, and subsequent iterations to be set as ŝj,k-<NUM>, i.e. the state output by the imagination <NUM> at the preceding iteration. The "n step" planning strategy is illustrated in row (b) of <FIG>. The k=<NUM> iteration is the same as in row (a). The k=<NUM> iteration is different, in that the circle representing the state imagined in iteration k=<NUM>, is the starting point for the addition of a downwardly extending dashed line, and then a circle. Similarly, in the k=<NUM> iteration is different, in that the circle representing the state imagined in iteration k=<NUM>, is the starting point for the addition of a downwardly extending dashed line, and then a circle. More generally, the state imagined in the n-th iteration will be one which is sj after it has been subject to n actions. Note that after k iterations, the imagination <NUM> has generated a consequent state ŝj,k which is k steps after j, so that if step <NUM> is performed at this point the controller has available to it a plan which extends for k steps into the future.

A third possibility (referred to as the "tree" strategy) is for the manager <NUM> to specify, in each iteration, which of the previously considered states is to be used as the starting point for the controller <NUM> in step <NUM> of the k-th iteration, and the imagination <NUM> in step <NUM> of the k-th iteration. The tree strategy is illustrated in row (c) of <FIG>. Note that in the k=<NUM> iteration, there is only one possibility sj for the state data to input to the controller <NUM> and imagination <NUM>, so the manager <NUM> has no option but to choose it. Thus, the k=<NUM> iteration is the same as the "one step" and "n step" strategies. In the k=<NUM> iteration the manager <NUM> determines that the state generated in the k=<NUM> iteration is to be input to the controller <NUM> and imagination <NUM>, so the k=<NUM> iteration happens to be the same as for the n step strategy. However, in the k=<NUM> iteration the manager <NUM> determines that the state generated in the k=<NUM> iteration is again to be input to the controller <NUM> and imagination <NUM>, so in row (c) of <FIG> this state is shown at the end of the k=<NUM> iteration as having two states beneath it, connected to it by lines. More generally, in the tree strategy, there is an "imagination tree" (in contrast, for example, to the array of states produced at the right hand end of row (b), which is just a chain of states), because imagined actions can be proposed from any previously imagined state.

Experiments using the neural network system <NUM> were performed in relation to a continuous control task, in which the parameters of the environment were defined by real numbers. The memory, <NUM>, was an long short-term memory (LSTM) which performed a function µ. In this implementation, the memory <NUM> was used, at each iteration i, to produce a context ci which was used, in place of the full history, as the input to the manager <NUM> and the controller <NUM>. That is, the memory <NUM> is arranged to output a function µ of arguments which comprise a portion of the internal and/or external data, and subset of the data stored in the memory, such as a function of a portion of the internal and/or external data for the last iteration and an output of the memory <NUM> in the last iteration. This is a generalization of the function of the memory <NUM> explained above. Specifically, for imaging, the memory <NUM> was arranged to output ci as a function µ(pj,k, sj, ŝj,pk, âj,k, ŝj,k+<NUM>, řj,k, j, k, ci-<NUM>). For acting, the memory <NUM> was arranged to output a function µ(pj,k, sj, ŝj,<NUM>, aj, sj+<NUM>, rj, j, k, ci-<NUM>). The manager <NUM> and controller <NUM> were multi-layer perceptrons (MLP). The manager took sj and ci-<NUM> as inputs, and outputted pj,k. The controller <NUM> took ŝj,pk and ci-<NUM> as inputs, and outputted âj,k or aj, for imagining or acting, respectively. The imagination <NUM> (i.e. imagination-based model of the environment) was an interaction network (see <NPL>) which is known to be able to learn to predict gravitational system dynamics accurately. For acting, it took as inputs ŝj,k and aj,k and returned ŝj,k+<NUM> for imagining, and for acting it took as inputs sj and aj and returned sj+<NUM> for acting.

In an experiment, the neural network system <NUM> was trained by jointly optimizing a cost function comprising two loss terms: a external term (termed performance loss) and an internal term (called resource loss) term. The performance loss term reflects a cost, in the environment chosen, of executing an action in the environment <NUM>. The resource loss term reflects the cost of using the imagination in a particular time step. It may be fixed, or vary with the number of actions taken so far, expressing the constraint that imagining early is more or less expensive than imagining on-the-fly. The training consisted of optimizing, by gradient descent, the parameters of the neural network system to jointly minimize the performance loss term and the resource loss term. Where gradients of the terms were not directly available, approximation methods were used to estimate them.

The training method is shown in <FIG>. In step <NUM> the imagination <NUM> was taught to make next-step predictions of the state in a supervised fashion, with error gradients computed by backpropagation. The training data is collected from the observations the agent makes when acting in the environment <NUM>. The policy of the imagination <NUM> in this case was stochastic, so an entropy reward was used during training to encourage exploration. In step <NUM>, the manager <NUM> was trained. In step <NUM>, the controller <NUM> and the memory <NUM> were jointly trained. Note that steps <NUM>, <NUM> and <NUM> are independent, so they can be performed in any order or simultaneously.

In our experiments for other tasks (e.g. tasks for which the environment does not have continuous parameters) the structure of the imagination <NUM> was chosen differently, and the training was not as shown in <FIG>.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. The computer storage medium is not, however, a propagated signal.

Claim 1:
A neural network system for task learning, wherein the neural network system is used to select actions to be performed by a mechanical agent interacting with the real-world environment to perform a task in an attempt to achieve a specified result, the system comprising:
a controller neural network module configured to receive state data and context data and to output action data, wherein said state data characterizes a real or imagined state of the environment, wherein said context data defines a context for planning actions and of proposed actions, and wherein said action data defines a real or imagined action to be performed on said environment or on an imagined version of said environment, the state data characterizing the real state of the environment being observations of the environment captured by one or more sensors of the mechanical agent;
a model neural network module configured to receive said state data and said action data and to output consequent state data defining a state consequent upon an action defined by said action data, and reward data defining a modelled reward;
a manager module configured to receive said state data and said context data and to output route data, wherein said route data defines whether the system is to execute an action or to imagine; and
a memory to store said context data, wherein said context data is derived from at least (i) said state data or said consequent state data, (ii) said action data and (iii) said reward data; and
wherein:
the manager module is configured, following storage of context data or updating of said stored context data, to receive the stored context data and to output consequent route data which defines whether the system is to execute an action or to imagine, and
when said consequent route data defines that the system is to imagine,
(i) the controller neural network module is configured to receive imagined state data, and to output imagined action data; (ii) the model neural network module is configured to receive said imagined state data and said imagined action data, and output imagined consequent state data and imagined reward data; and
(iii) the memory is configured to update the stored context data based on the imagined consequent state data, the imagined action data and the imagined reward data;
when said consequent route data defines that the system is to execute an action, the system is configured to transmit control data generated by the controller to the mechanical agent.