Patent Description:
Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model. Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

<NPL>, describes a system in which an embedding of each observation is added to a memory, and in which a policy network applies attention over the memory to produce an action.

This specification describes an action selection system implemented as computer programs on one or more computers in one or more locations for controlling an agent interacting with an environment to accomplish a goal.

Throughout this specification, an "embedding" of an entity (e.g., an observation of an environment) can refer to a representation of the entity as an ordered collection of numerical values, e.g., a vector or matrix of numerical values. An embedding of an entity can be generated, e.g., as the output of a neural network that processes data characterizing the entity.

According to a first aspect there is provided a method according to the appended independent claim <NUM>.

In implementations the planning embedding does not include a representation of a current observation characterizing a current state of the environment.

The method includes processing the planning embeddings using a planning neural network to generate an implicit plan for accomplishing the goal. The implicit plan may thus comprise an embedding encoding information about the previous interactions of the agent with the environment, and the goal. As described later, it may also depend on a representation of the current observation. It may implicitly characterize actions that can be performed by the agent to accomplish the goal. The planning neural network may be any neural network which is configured to process the planning embeddings, a goal embedding, and in implementations the representation of the current observation. However in implementations the planning neural network may include one or more self-attention layers, as described later.

The method further comprises selecting the action to be performed by the agent at the time step using the implicit plan.

In implementations the method iteratively updates the planning embeddings using attention over the planning embeddings, e.g. using an attention sub-network. Multiple iterations of the same attention, e.g. self-attention, function may be applied over the planning embeddings. The implicit plan may be generated based on the planning embeddings and the current observation. Generating the implicit plan may comprise appending a representation of the current embedding to each planning embedding and processing the combined embedding using one or more neural network layers e.g. self-attention layers, e.g. using the attention sub-network. In implementations these neural network layers do not process the representation of the current observation.

In broad terms using attention involves applying an attention mechanism, e.g. a self-attention mechanism which relates the planning embeddings to one another to determine the implicit plan. Details of attention mechanisms vary, but in general an attention mechanism may map a learned query vector and a learned set of key-value vector pairs to an output. The output may be computed as a weighted sum of the values, with weights dependent on a similarity of the query and key. In a self-attention mechanism of this type the input to the attention mechanism may be a set of planning embeddings and the output may the a transformed version of the same set of planning embeddings. Merely as one example, a dot-product attention mechanism is described in arXiv: <NUM> (which also describes an example of multi-head attention). In implementations using (self-)attention facilitates determining relationships amongst past states.

In some implementations using attention over the planning embeddings involves processing the planning embeddings using a residual neural network block (i.e. one that includes a residual or skip connection). The residual neural network block may be configured to apply a sequence of operations to the planning embeddings including a layer normalization operation (see, e.g., arXiv: <NUM>:<NUM>), an attention operation, and a linear projection operation.

The method may involve training the planning neural network jointly with an action selection using any reinforcement learning technique e.g. by backpropagating gradients of a reinforcement learning objective function. The reinforcement learning objective function may be any suitable objective function e.g. a temporal difference objective function or a policy gradient objective function, e.g. using an actor-critic objective function, dependent upon the rewards received by the agent from the environment in response to the actions.

The system described in this specification can enable an agent to use information learned about an environment to generate "implicit plans" for solving tasks (i.e., accomplishing goals) in the environment. An implicit plan refers to data (e.g., numerical data represented as an ordered collection of numerical values, e.g., a vector or matrix of numerical values) that implicitly characterizes actions that can be performed by the agent to accomplish a task. By selecting actions to be performed by the agent using implicit plans, the system described in this specification can enable the agent to accomplish tasks and explore the environment more efficiently (e.g., by accomplishing tasks and exploring the environment over fewer time steps). That is, the described techniques allow a mix of exploration and goal-directed behavior and at the same time enable the agent to learn to plan over long timescales, such that once trained the agent is able to generalize beyond its training experience. Thus in particular, the system described in this specification can enable an agent to exploit its previously gained knowledge of tasks and environments to efficiently perform new (i.e., previously unseen) tasks in new environments. In one example, the agent may be a consumer robot that performs household tasks (e.g., cleaning tasks), and the system described in this specification may enable the agent to efficiently perform new tasks as the agent is placed in new environments (e.g., different rooms in a house).

The system described in this specification can generate an implicit plan for solving a task by generating planning embeddings based on past interactions of the agent with the environment, and iteratively updating the planning embeddings using attention operations. Iteratively updating the planning embeddings using attention operations allows information to be shared amongst the planning embeddings and thereby facilitates more effective planning which can enable the agent to accomplish tasks and explore the environment more efficiently, e.g., over fewer time steps.

Other advantages will become apparent from the description, the drawings, and the claims.

<FIG> is a block diagram of an example action selection system <NUM>. The action selection system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The system <NUM> selects actions <NUM> to be performed by an agent <NUM> interacting with an environment <NUM> at each of multiple time steps to accomplish a goal. At each time step, the system <NUM> receives data characterizing the current state of the environment <NUM>, e.g., an image of the environment <NUM>, and selects an action <NUM> to be performed by the agent <NUM> in response to the received data. Data characterizing a state of the environment <NUM> will be referred to in this specification as an observation <NUM>. At each time step, the state of the environment <NUM> at the time step (as characterized by the observation <NUM>) depends on the state of the environment <NUM> at the previous time step and the action <NUM> performed by the agent <NUM> at the previous time step.

At each time step, the system <NUM> may receive a reward <NUM> based on the current state of the environment <NUM> and the action <NUM> of the agent <NUM> at the time step. Generally, the reward <NUM> may be represented a numerical value. The reward <NUM> can be based on any event in or aspect of the environment <NUM>. For example, the reward <NUM> may indicate whether the agent <NUM> has accomplished a goal (e.g., navigating to a target location in the environment <NUM>) or the progress of the agent <NUM> towards accomplishing a goal.

In some implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment. For example, the agent may be a robot interacting with the environment to accomplish a goal, e.g., to locate an object of interest in the environment, to move an object of interest to a specified location in the environment, to physically manipulate an object of interest in the environment in a specified way, or to navigate to a specified destination in the environment; or the agent may be an autonomous or semi-autonomous land, air, or sea vehicle navigating through the environment to a specified destination in the environment. Then the actions may be actions taken by the mechanical agent in the real-world environment to accomplish the goal, and may include control signals to control the mechanical agent.

In these implementations, the observations may include, for example, one or more of images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator.

For example in the case of a robot the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, for example gravity-compensated torque feedback, and global or relative pose of an item held by the robot.

In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in <NUM>, <NUM> or <NUM> dimensions, and may be absolute and/or relative observations.

The observations may also include, for example, data obtained by one of more sensor devices which sense a real-world environment; for example, sensed electronic signals such as motor current or a temperature signal; and/or image or video data for example from a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

In the case of an electronic agent the observations may include data from one or more sensors monitoring part of a plant or service facility such as current, voltage, power, temperature and other sensors and/or electronic signals representing the functioning of electronic and/or mechanical items of equipment.

The actions may be control inputs to control a robot, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land or air or sea vehicle, e.g., torques to the control surface or other control elements of the vehicle or higher-level control commands.

In other words, the actions can include for example, position, velocity, or force/torque/acceleration data for one or more joints of a robot or parts of another mechanical agent. Actions may additionally or alternatively include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land, air, or sea vehicle the actions may include actions to control navigation e.g. steering, and movement e.g., braking and/or acceleration of the vehicle.

In some other applications the agent may control actions in a real-world environment including items of equipment, for example in a data center or grid mains power or water distribution system, or in a manufacturing plant or service facility. The observations may then relate to operation of the plant or facility. For example the observations may include observations of power or water usage by equipment, or observations of power generation or distribution control, or observations of usage of a resource or of waste production. The agent may control actions in the environment to accomplish the goal of increased efficiency, for example by reducing resource usage, and/or reduce the environmental impact of operations in the environment, for example by reducing waste. The actions may include actions controlling or imposing operating conditions on items of equipment of the plant/facility, and/or actions that result in changes to settings in the operation of the plant/facility e.g. to adjust or turn on/off components of the plant/facility.

In some further applications, the environment is a real-world environment and the agent manages distribution of tasks across computing resources e.g. on a mobile device and/or in a data center. In these implementations, the actions may include assigning tasks to particular computing resources, and the goal to be achieved can include minimizing the time required to complete a set of tasks using specified computing resources.

In any of the above implementations, the observation at any given time step may include data from a previous time step that may be beneficial in characterizing the environment, e.g., the action performed at the previous time step, the reward received at the previous time step, and so on.

The system <NUM> selects the action to be performed by the agent <NUM> at each time step using an external memory <NUM>, a planning neural network <NUM>, and an action selection neural network <NUM>, as will be described in more detail next.

The memory <NUM> stores a respective "experience tuple" corresponding to each of multiple previous time steps (e.g., the memory <NUM> can store a respective experience tuple for each time step before the current time step). The memory <NUM> can be implemented, e.g., as a logical data storage area or physical data storage device.

An experience tuple for a time step refers to data that characterizes the interaction of the agent <NUM> with the environment <NUM> at the previous time step. For example, an experience tuple for a previous time step can include respective embeddings (representations) of: (i) the observation at the previous time step, (ii) the action performed by the agent at the previous time step, and (iii) the subsequent observation that resulted from the action performed by the agent at the previous time step.

The system <NUM> can generate an embedding of an observation (e.g., that is included in an experience tuple) by providing the observation to an embedding neural network that is configured to process the observation to generate a corresponding embedding. The system <NUM> can generate an embedding of an action (e.g., that is included in an experience tuple) by associating the action with a one-hot embedding that uniquely identifies the action from a set of possible actions.

In some implementations, the system <NUM> clears the memory <NUM> (i.e., by deleting or overwriting the contents of the memory <NUM>) each time a clearing criterion is satisfied. For example, the clearing criterion can be satisfied if the agent accomplishes the goal in the environment, if the agent is placed in a new environment, or if the memory is full (e.g., because an experience tuple is stored in each available slot in the memory).

To select the action to be performed at a time step, the system <NUM> generates a respective "planning" embedding <NUM> corresponding to each of multiple experience tuples stored in the memory <NUM>. The system <NUM> generates the planning embedding <NUM> for an experience tuple, e.g., by concatenating an embedding of a "goal" observation to the experience tuple, where the goal observation represents a state of the environment when the goal of the agent has been accomplished. For example, if the goal of the agent is to navigate to a specified location in the environment, then the goal observation may be an observation representing the state of the environment when the agent is located at the specified location. In some other implementations, the system <NUM> can identify the planning embedding <NUM> associated with an experience tuple as being a copy of the experience tuple (e.g., such that the planning embeddings <NUM> and the experience tuples stored in the memory <NUM> are the same).

The system <NUM> can generate a respective planning embedding <NUM> corresponding to each experience tuple stored in the memory <NUM>. Alternatively, the system <NUM> can generate planning embeddings <NUM> for only a proper subset of the experience tuples stored in the memory <NUM>, e.g., for only the experience tuples corresponding to a predetermined number L of most recent time steps. (L can be any appropriate positive integer value, e.g., L = <NUM>).

The planning neural network <NUM> is configured to process: (i) the planning embeddings <NUM> representing previous interactions of the agent with the environment, and (ii) a current observation <NUM> representing the current state of the environment, to generate an "implicit plan" <NUM> for accomplishing the goal of the agent. The implicit plan <NUM> is an embedding that can encode information about the current state of the environment (from the current observation <NUM>), the history of the interaction of the agent with the environment (from the planning embeddings <NUM>), and optionally, the goal to be accomplished by the agent (also from the planning embeddings <NUM>).

The planning neural network <NUM> can have any appropriate neural network architecture that enables it to perform its described functions. As part of generating the implicit plan <NUM> from the planning embeddings <NUM>, the planning neural network <NUM> can enrich the planning embeddings by updating the planning embedding using self-attention operations. An example architecture of the planning neural network <NUM> is described in more detail with reference to <FIG>.

The action selection neural network <NUM> is configured to process an input that includes the implicit plan <NUM> generated by the planning neural network <NUM> to generate an action selection output <NUM>. Optionally, the action selection neural network <NUM> can process other data in addition to the implicit plan <NUM>, e.g., the action selection neural network <NUM> can also process respective embeddings of one or more of: the current observation, the action performed at the previous time step, or the reward received at the previous time step. The action selection output <NUM> can include a respective score for each action in a set of possible actions that can be performed by the agent.

The system <NUM> selects the action <NUM> to be performed by the agent <NUM> at the time step using the action selection output <NUM> generated by the action selection neural network <NUM> at the time step. For example, the system <NUM> can select the action having the highest score, according to the action selection output <NUM>, as the action to be performed by the agent at the time step. In some implementations, the system <NUM> selects the action to be performed by the agent in accordance with an exploration strategy. For example, the system <NUM> can use an ε-greedy exploration strategy. In this example, the system <NUM> can select the action having a highest score (according to the action selection output <NUM>) with probability <NUM> - ε, and select an action randomly with probability ε, where ε is a number between <NUM> and <NUM>.

The action selection neural network <NUM> can have any appropriate neural network architecture that enables it to perform its described functions. For example, the action selection neural network can include any appropriate neural network layers (e.g., convolutional layers, fully connected layers, attention layers, etc.) connected in any appropriate configuration (e.g., as a linear sequence of layers). In one example, the action selection neural network <NUM> can include: an input layer that is configured to receive the implicit plan <NUM>, a linear sequence of multiple fully-connected layers, and an output layer that includes a respective neuron corresponding to each action in the set of possible actions that can be performed by the agent.

After the system <NUM> selects the action <NUM> to be performed by the agent <NUM> at the time step, the agent <NUM> interacts with the environment <NUM> by performing the action <NUM>, and the system <NUM> can receive a reward <NUM> based on the interaction. The system <NUM> can generate an experience tuple characterizing the interaction of the agent with the environment at the time step, and store the experience tuple in the memory <NUM>.

A training engine <NUM> can use the observations <NUM> and corresponding rewards <NUM> resulting from the interactions of the agent <NUM> with the environment <NUM> to train the action selection system <NUM> using reinforcement learning techniques. The training engine <NUM> trains the action selection system <NUM> by iteratively adjusting the parameters of the action selection neural network <NUM> and the planning neural network <NUM>. The training engine <NUM> can adjust the parameters of the action selection system <NUM> by iteratively backpropagating gradients of a reinforcement learning objective function through the action selection system <NUM>. By training the action selection system <NUM>, the training engine <NUM> can cause the action selection system <NUM> to select actions that increase a cumulative measure of reward (e.g., a long-term time-discounted cumulative reward) received by the action selection system <NUM> and cause the agent to accomplish its goal more effectively (e.g., over fewer time steps).

<FIG> shows an example architecture of a planning neural network <NUM> that is included in the action selection system <NUM> described with reference to <FIG>.

The planning neural network <NUM> is configured to process: (i) a set of planning embeddings <NUM> representing previous interactions of the agent with the environment, and (ii) a current observation <NUM> representing the current state of the environment, to generate an implicit plan <NUM> for accomplishing the goal of the agent.

The planning neural network <NUM> includes an attention sub-network <NUM> and a fusion sub-network <NUM>, which will each be described in more detail next.

The attention sub-network <NUM> is configured to iteratively (i.e., at each of one or more iterations) update the planning embeddings <NUM> to generate updated planning embeddings <NUM>. More specifically, the attention sub-network <NUM> iteratively updates the planning embeddings <NUM> by processing the planning embeddings <NUM> using a sequence of one or more "attention blocks. " Each attention block is a collection of one or more neural network layers that is configured to receive a set of current planning embeddings, to update the current planning embeddings by applying attention operations to the current planning embeddings, and to output the updated planning embeddings. The first attention block can receive the initial planning embeddings <NUM>, each subsequent attention block can receive the planning embeddings output by the preceding attention block, and the final attention block can output the updated planning embeddings <NUM> (i.e., that define the output of the attention sub-network <NUM>).

Each attention block updates the planning embeddings by applying attention operations to the planning embeddings, in particular, by updating each planning embedding using self-attention over the planning embeddings. To update a given planning embedding using self-attention over the planning embeddings, the attention block can determine a respective "attention weight" between the given planning embedding and each planning embedding in the set of planning embeddings. The attention block can then update the given planning embedding using: (i) the attention weights, and (ii) the planning embeddings.

For example, if the planning embeddings <NUM> are denoted by <MAT>, where N is the number of planning embeddings, then to update planning embedding pi, an attention block can determine attention weights <MAT> where ai,j denotes the attention weight between pi and pj, as: <MAT> <MAT> where Wq and Wk are learned parameter matrices, softmax(-) denotes a soft-max normalization operation, and c is a constant. Using the attention weights, the attention block can update planning embedding pi as: <MAT> where Wv is a learned parameter matrix. (Wqpi can be referred to as the "query embedding" for planning embedding pi, Wkpj can be referred to as the "key embedding" for planning embedding pj, and Wvpj can be referred to as the "value embedding" for planning embedding pj). The parameter matrices Wq (the "query embedding matrix"), Wk (the "key embedding matrix"), and Wv (the "value embedding matrix") are trainable parameters of the attention block. Generally, each attention block in the attention sub-network <NUM> can use query, key, and value embedding matrices with different parameter values to update the planning embeddings.

Optionally, each attention block can have multiple "heads" that each generate a respective updated planning embedding corresponding to each input planning embedding, i.e., such that each input planning embedding is associated with multiple updated planning embeddings. For example, each head may generate updated planning embeddings in accordance with different values of the parameter matrices Wq, Wk, and Wv that are described with reference to equations (<NUM>)-(<NUM>). An attention block with multiple heads can implement a "gating" operation to combine the updated planning embeddings generated by the heads for an input planning embedding, i.e., to generate a single updated planning embedding corresponding to each input planning embedding. For example, the attention block can process the input planning embeddings using one or more neural network layers (e.g., fully connected neural network layers) to generate a respective gating value for each head. The attention block can then combine the updated planning embeddings corresponding to an input planning embedding in accordance with the gating values. For example, the attention block can generate the updated planning embedding for an input planning embedding pi as: <MAT> where k indexes the heads, αk is the gating value for head k, and <MAT> is the updated planning embedding generated by head k for input planning embedding pi. The attention operations described with reference to equations (<NUM>)-(<NUM>) can be referred as "multi-head key-query-value attention operations.

By updating the planning embeddings <NUM> using self-attention operations, the planning neural network <NUM> uses learned operations to share information amongst the planning embeddings <NUM> and thereby enrich each planning embedding with information content from the other planning embeddings. Enriching the information content of the planning embeddings <NUM> can enable the planning neural network <NUM> to generate more informative implicit plans <NUM> that enable the agent to accomplish goals in the environment more efficiently, e.g., over fewer time steps.

In some implementations, as described with reference to <FIG>, the action selection system <NUM> generates planning embeddings <NUM> corresponding to only a proper subset of the experience tuples stored in the memory, e.g., for only the experience tuples corresponding to the L most recent time steps. To enable the planning system <NUM> to incorporate information from all the stored experience tuples (i.e., in addition to only the L most recent experience tuples), the action selection system <NUM> can generate a respective "static" embedding corresponding to each experience tuple stored in the memory. The action selection system <NUM> can generate a static embedding corresponding to an experience tuple, e.g., by concatenating an embedding of a goal observation to the experience tuple. After generating the static embeddings for the experience tuples stored in the memory, the action selection system <NUM> can then provide the static embeddings to the planning system <NUM> in addition to the planning embeddings <NUM>.

Each attention block of the attention sub-network <NUM> can update the planning embeddings using cross-attention over the static embeddings, in addition to using self-attention over the planning embeddings themselves (as described above). For example, each attention block can first update the planning embeddings using cross-attention over the static embeddings, and then update the planning embeddings using self-attention over the planning embeddings. Generally, the attention blocks of the attention sub-network <NUM> do not update the static embeddings, i.e., such that the static embeddings remain fixed even as the planning embeddings are updated by each attention block of the attention sub-network <NUM>.

To update a given planning embedding using cross-attention over the static embeddings, an attention block can determine a respective attention weight between the given planning embedding and each static embedding. The attention block can then update the given planning embedding using: (i) the attention weights, and (ii) the static embeddings. For example, if the planning embeddings <NUM> are denoted by <MAT> and the static embeddings are denoted by <MAT>, then to update planning embedding pi, an attention block can determine attention weights <MAT> where ai,j denotes the attention weight between pi and sj, as: <MAT> <MAT> where Wq and Wk are learned parameter matrices, softmax(-) denotes a soft-max normalization operation, and c is a constant. Using the attention weights, the attention block can update planning embedding pi as: <MAT> where Wv is a learned parameter matrix. Optionally, the attention block can have multiple heads that generate multiple updated planning embeddings corresponding to each input planning embedding using cross-attention over the static embeddings. The attention block can combine the multiple updated planning embeddings corresponding to each input planning embedding to generate a single updated planning embedding corresponding to each input planning embedding, as described above.

By updating the planning embeddings <NUM> using cross-attention over static embeddings for every experience tuple in the external memory, the planning neural network <NUM> can efficiently capture information from all of the previous interactions of the agent with environment and thereby generate a more informative implicit plan. The action selection system can use the more informative implicit plan to select actions that enable the agent to accomplish goals in the environment more effectively, e.g., over fewer time steps. By refraining from updating the static embeddings, e.g., using attention operations, the planning neural network can significantly reduce consumption of computational resources (e.g., memory and computing power).

Each attention block can implement any other appropriate neural network operations in addition to the attention operations described above to update the current planning embeddings. For example, each attention block can be a residual block that processes current planning embeddings Bi to generate updated planning embeddings Bi+<NUM> as follows: <MAT> where LayerNorm(·) denotes a layer normalization operation, MHA(·) denotes a multi-head attention operation (including self-attention over the planning embeddings Bi, and optionally, cross attention over the static embeddings), and f(·) denotes a linear projection operation.

The fusion sub-network <NUM> is configured to process: (i) the updated planning embeddings <NUM>, and (ii) the current observation <NUM>, to generate the implicit plan <NUM> for accomplishing the goal. Generally, the fusion sub-network <NUM> can have any appropriate neural network architecture that enables it to perform its described function, including any appropriate neural network layers (e.g., convolutional layers or fully-connected layers) connected in any appropriate configuration (e.g., as a linear sequence of layers).

For example, to generate the implicit plan <NUM>, the fusion sub-network <NUM> can generate an embedding of the current observation <NUM>, e.g., by processing the current observation <NUM> using an embedding neural network. The fusion sub-network <NUM> can then append (concatenate) the embedding of the current observation <NUM> to each updated planning embedding <NUM> to generate a respective "combined" embedding corresponding to each updated planning embedding <NUM>. The fusion sub-network <NUM> can process each combined embedding using one or more neural network layers (e.g., fully-connected layers) to generate a respective "transformed" embedding corresponding to each updated planning embedding <NUM>. The fusion sub-network <NUM> can generate the implicit plan by applying a pooling operation to the transformed embeddings. The pooling operation can be any appropriate operation that, when applied to the transformed embeddings, generates an implicit plan having a dimensionality that is independent of the number of transformed embeddings. For example, the pooling operation can be a feature-wise max pooling operation, i.e., where the implicit plan is defined as having the same dimensionality as each transformed embedding, and each entry of the implicit plan is defined as the maximum of the corresponding entries of the transformed embeddings.

The parameters of the planning neural network <NUM>, including the parameters of the attention sub-network <NUM> (including its constituent attention blocks) and the fusion sub-network <NUM>, are jointly trained along with the parameters of the action selection neural network <NUM> by a training engine using reinforcement learning techniques (as described with reference to <FIG>). In particular, the gradients of a reinforcement learning objective function are backpropagated through the action selection neural network and into the fusion sub-network and the attention sub-network of the planning neural network. These gradients are used to adapt the parameters of the planning neural network to enable the generation of implicit plans encoding information that, when processed by the action selection neural network, result in the selection of actions that allow the agent to effectively accomplish goals in the environment.

<FIG> is a flow diagram of an example process <NUM> for selecting actions to be performed by an agent interacting with an environment to accomplish a goal. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system, e.g., the action selection system <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>.

The system generates a respective planning embedding corresponding to each of multiple experience tuples in an external memory (<NUM>). Each experience tuple characterizes interaction of the agent with the environment at a respective previous time step.

The system processes the planning embeddings using a planning neural network to generate an implicit plan for accomplishing the goal (<NUM>).

The system selects the action to be performed by the agent at the time step using the implicit plan (<NUM>).

<FIG> a schematic illustration of an example of the system of <FIG>, in which like elements to those previously described are indicated by like reference numerals.

Claim 1:
A method performed by one or more data processing apparatus (<NUM>) for selecting actions (<NUM>) to be performed by an agent (<NUM>) interacting with an environment (<NUM>) to accomplish a goal, the method comprising:
generating a respective planning embedding (<NUM>) corresponding to each of multiple experience tuples in an external memory (<NUM>), wherein each experience tuple characterizes interaction of the agent with the environment at a respective previous time step, wherein generating the respective planning embedding comprises appending a representation of a goal state of the environment to the experience tuple, and wherein the goal state represents a state of the environment when the goal of the agent has been accomplished;
processing the planning embeddings using a planning neural network (<NUM>) to generate an implicit plan (<NUM>) for accomplishing the goal; and
selecting the action to be performed by the agent at the time step using the implicit plan; and wherein
i) the agent is a mechanical agent and the environment is a real-world environment, and the actions include control signals for actions of the mechanical agent in the real-world environment to accomplish the goal, or
ii) the environment is a real-world environment including items of equipment in a plant or facility and the agent controls actions in the environment, the actions controlling or imposing operating conditions on the items of equipment of the plant/facility, or
iii) the environment is a real-world environment and the agent manages distribution of tasks across computing resources in the environment, and the actions include assigning tasks to particular computing resources.