Unsupervised control using learned rewards

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting actions to be performed by an agent that interacts with an environment. In one aspect, a system comprises: an action selection subsystem that selects actions to be performed by the agent using an action selection policy generated using an action selection neural network; a reward subsystem that is configured to: receive an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment; generate a reward using an embedded representation of the observation characterizing the current state of the environment and an embedded representation of the observation characterizing the goal state of the environment; and a training subsystem that is configured to train the action selection neural network based on the rewards generated by the reward subsystem using reinforcement learning techniques.

BACKGROUND

This specification relates to reinforcement learning.

In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.

Some reinforcement learning systems select the action to be performed by the agent in response to receiving a given observation in accordance with an output of a neural network.

Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer.

SUMMARY

This specification describes a system implemented as computer programs on one or more computers in one or more locations that controls an agent interacting with an environment.

According to a first aspect there is provided a system for selecting actions to be performed by an agent that interacts with an environment by performing actions from a predetermined set of actions.

The system includes an action selection subsystem that is configured to, at each of multiple of time steps: receive an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment; process the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using an action selection neural network to generate an action selection policy; and select an action from the predetermined set of actions to be performed by the agent using the action selection policy.

The system includes a reward subsystem that is configured to, at each of multiple time steps: receive an observation characterizing a current state of the environment and an observation characterizing a goal state of the environment; process the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using an embedding neural network, where the embedding neural network is configured to process an observation to generate an embedded representation of the observation; and generate a reward using the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the observation characterizing the goal state of the environment.

The system includes a training subsystem that is configured to train the action selection neural network based on the rewards generated by the reward subsystem using reinforcement learning techniques.

In some implementations, the reward subsystem is further configured to: obtain one or more decoy observations, where each decoy observation characterizes a past state of the environment; for each decoy observation, generate an embedded representation of the decoy observation by processing the decoy observation using the embedding neural network; and determine a similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using: (i) the embedded representation of the observation characterizing the current state of the environment, (ii) the embedded representation of the observation characterizing the goal state of the environment, and (iii) the embedded representations of the decoy observations.

In some implementations, determining the similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment includes: for each decoy observation, determining a similarity metric between the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the decoy observation; determining a similarity metric between the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the observation characterizing the goal state of the environment; and determining the similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment using the determined similarity metrics.

In some implementations, the similarity metric is an inner product.

In some implementations, generating the reward includes generating the reward based on the determined similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment.

In some implementations, generating the reward includes determining the reward based on a similarity between the embedded representation of the observation characterizing the current state of the environment and the embedded representation of the observation characterizing the goal state of the environment.

In some implementations, the training subsystem is further configured to: determine a gradient of a loss function that depends on the determined similarity between the observation characterizing the current state of the environment and the observation characterizing the goal state of the environment; and adjust current values of embedding neural network parameters using the gradient.

In some implementations, the reward subsystem is configured to generate a predetermined reward if a current time step is not a goal time step.

In some implementations, the system further includes a goal buffer including multiple past observations, and the training subsystem is configured to determine the observation characterizing the goal state of the environment by sampling from the goal buffer.

In some implementations, the training subsystem is configured to update the goal buffer by, at each of multiple time steps, replacing a randomly selected observation from the goal buffer by a current observation with a predetermined probability.

In some implementations, the training subsystem is configured to update the goal buffer by, at each of multiple time steps: randomly selecting a candidate observation from the goal buffer; and determining whether to replace the candidate observation by a current observation based on: (i) an aggregate similarity measure between the current observation and the observations included in the goal buffer, and (ii) an aggregate similarity measure between the candidate observation and the observations included in the goal buffer.

In some implementations, the action selection neural network is a Q neural network and the action selection policy includes a respective Q value for each action in the predetermined set of actions.

In some implementations, selecting an action includes determining an action with a highest Q value.

In some implementations, selecting an action includes selecting the action in accordance with an exploration policy.

In some implementations, the action selection neural network is further configured to process a periodic representation of the current time step.

In some implementations, the training subsystem is configured to train the action selection neural network and the embedding neural network using experience replay.

In some implementations, training the action selection neural network and the embedding neural network using experience replay includes training the action selection neural network and the embedding neural network using hindsight experience replay.

In some implementations, the embedding neural network and the action selection neural network share one or more parameter values.

According to a second aspect there is provided a method performed by one or more data processing apparatus for performing operations including the operations of the first aspect.

According to a third aspect there is provided one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations including the operations of the first aspect.

The system described in this specification learns to generate rewards that are used to train an agent to “master” an environment by performing actions that cause the state of the environment to transition into specified “goal” states.

By making use of the learned rewards, the system can train the action selection network even in the absence of an explicit external reward signal, or when the external reward signal is very sparse.

Moreover, using the learned rewards can enable the system to reduce the consumption of computational resources (e.g., memory and computing power) required to train the agent to master the environment relative to using, e.g., hand-crafted rewards. In particular, using the learned rewards can enable the system to train the agent to master the environment over fewer training iterations than would otherwise be necessary. Hand-crafted rewards refer to rewards generated by a reward function that is explicitly defined by a person, rather than being learned using machine learning techniques.

Hand-crafting rewards may be difficult, expensive, and time consuming. For example, in a robotics environment, manually specifying a reward that depends on a robotic agent putting a ball inside a cup may require using image processing techniques to localize the cup and determine that the cup is positioned at an angle such that it could hold the ball. Even after localizing the cup and identifying its orientation, determining whether the ball is inside the cup may require, e.g., obtaining measurements from a weight sensor placed in the cup for this purpose. The system described in this specification obviates certain difficulties associated with hand-crafting rewards by automatically learning to generate rewards.

Many tasks that the agent can perform in the environment (e.g., navigation tasks) implicitly rely on the agent having mastered the environment. By training the agent to master the environment, the system described in this specification can enable the agent to learn how to perform tasks in the environment more quickly, e.g., over fewer training iterations. Therefore, the system described in this specification can reduce the consumption of computational resources required to train the agent to perform tasks in the environment.

DETAILED DESCRIPTION

This specification describes an action selection system that controls an agent interacting with an environment by, at each of multiple time steps, processing data characterizing the current state of the environment at the time step (i.e., an “observation”) to select an action to be performed by the agent.

The action selection system can learn to “master” the environment by selecting actions that cause the state of the environment to transition into specified “goal” states. In particular, the action selection system can jointly learn: (i) an action selection policy, and (ii) a reward function. The action selection policy selects actions to be performed by the agent that cause the state of the environment to transition into a specified goal state. The reward function generates rewards that characterize the similarity of the current state of the environment to the goal state of the environment. The action selection system can learn the action selection policy based on the rewards generated by the reward function using reinforcement learning techniques.

At each time step, the state of the environment at the time step depends on the state of the environment at the previous time step and the action performed by the agent at the previous time step.

In some implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle navigating through the environment.

In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent as it interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator.

For example in the case of a robot the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot.

In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and/or relative observations.

The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and/or image or video data for example from a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

In the case of an electronic agent the observations may include data from one or more sensors monitoring part of a plant or service facility such as current, voltage, power, temperature and other sensors and/or electronic signals representing the functioning of electronic and/or mechanical items of equipment.

In these implementations, the actions may be control inputs to control the robot, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements of the vehicle or higher-level control commands.

In other words, the actions can include for example, position, velocity, or force/torque/acceleration data for one or more joints of a robot or parts of another mechanical agent. Action data may additionally or alternatively include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land or air or sea vehicle the actions may include actions to control navigation e.g. steering, and movement e.g., braking and/or acceleration of the vehicle.

In some implementations the environment is a simulated environment and the agent is implemented as one or more computers interacting with the simulated environment.

The simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle.

In another example, the simulated environment may be a video game and the agent may be a simulated user playing the video game. Generally, in the case of a simulated environment, the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions.

In some other applications the agent may control actions in a real-world environment including items of equipment, for example in a data center, in a power/water distribution system, or in a manufacturing plant or service facility. The observations may then relate to operation of the plant or facility. For example the observations may include observations of power or water usage by equipment, or observations of power generation or distribution control, or observations of usage of a resource or of waste production. The actions may include actions controlling or imposing operating conditions on items of equipment of the plant/facility, and/or actions that result in changes to settings in the operation of the plant/facility e.g. to adjust or turn on/off components of the plant/facility.

These features and other features are described in more detail below.

FIG. 1shows an example action selection system100. The action selection system100is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

The system100controls an agent102interacting with an environment104by selecting actions106to be performed by the agent102that cause the state of the environment to transition into a goal state specified by a goal observation108.

The system100includes an action selection neural network110, an embedding neural network112, a goal buffer114, a training engine116, and a set of model parameters118of the action selection network110and the embedding network112.

At each of multiple time steps, the action selection network110is configured to process: (i) a current observation120characterizing the current state of the environment104, and (ii) a goal observation108characterizing a goal state of the environment104, to generate an action selection output122(“action selection policy”). The system100uses the action selection output122to select the action106to be performed by the agent at the current time step. A few examples of using the action selection output122to select the action106to be performed by the agent are described next.

In one example, the action selection output122may include a respective numerical probability value for each action in a set of possible actions that can be performed by the agent. The system100can select the action to be performed by the agent, e.g., by sampling an action in accordance with the probability values for the actions, or by selecting the action with the highest probability value.

In another example, the action selection output122may directly define the action to be performed by the agent, e.g., by defining the values of torques that should be applied to the joints of a robotic agent.

In another example, the action selection output122may include a respective Q-value for each action in the set of possible actions that can be performed by the agent. The system100can process the Q-values (e.g., using a soft-max function) to generate a respective probability value for each possible action, which can be used to select the action to be performed by the agent (as described earlier). The system100could also select the action with the highest Q-value as the action to be performed by the agent.

The Q value for an action is an estimate of a “return” that would result from the agent performing the action in response to the current observation120and thereafter selecting future actions performed by the agent102in accordance with current values of the action selection network parameters.

A return refers to a cumulative measure of “rewards”124received by the agent, for example, a time-discounted sum of rewards. The agent can receive a respective reward124at each time step, where the reward124is specified by a scalar numerical value and characterizes, e.g., a progress of the agent towards reaching the goal state. The system100determines the rewards124received by the agent102using the embedding network112, as will be described in more detail below.

In some cases, the system100can select the action to be performed by the agent in accordance with an exploration policy. For example, the exploration policy may be an ϵ-greedy exploration policy, where the system100selects the action to be performed by the agent in accordance with the action selection output122with probability 1-ϵ, and randomly selects the action with probability ϵ. In this example, ϵ is a scalar value between 0 and 1.

The action selection network110can have any of a variety of neural network architectures. For example, the architecture of the action selection network110may include a sequence of one or more convolutional layers, followed by a recurrent layer (e.g., a long short-term memory (LSTM) layer) and a linear output layer. The action selection network may process both the current observation120and the goal observation108using the convolutional layers to generate respective intermediate outputs, and thereafter concatenate the intermediate outputs before providing them to the recurrent layer and the linear output layer. The output layer may include a respective neuron corresponding to each possible action that can be performed by the agent, and the activation of each neuron may specify the Q value of the action corresponding to the neuron.

The embedding neural network is configured to process an observation characterizing a state of the environment (e.g., a current observation120or a goal observation108) to generate an embedding of the observation. An embedding of an observation refers to a representation of the observation as an ordered collection of numerical values, e.g., a vector or matrix of numerical values.

The embedding network112can have any of a variety of neural network architectures. For example, the embedding network112may include a sequence of one or more convolutional layers followed by a fully-connected output layer. In some implementations, the convolutional layers of the embedding network112may share the same parameter values as the convolutional layers of the action selection network110(as described earlier). The embedding network112may normalize the embedding generated by the output layer, e.g., by dividing each component of the embedding by the L2-norm of the embedding.

At some or all of the time steps, the system100determines the reward124received by the agent102at the time step using the embedding network112(i.e., the embedding network defines the “reward function” used by the system100). In particular, to determine the reward124received by the agent at a time step, the system100processes the current observation120and the goal observation108using the embedding network112to generate an embedding126of the current observation and an embedding128of the goal observation. Thereafter, the system100determines the reward124received by the agent at the time step based on the respective embeddings126and128of the current observation120and the goal observation108.

The system100may generate a higher reward if the embedding of the current observation120is more similar (e.g., according to an appropriate numerical similarity measure) to the embedding of the goal observation108. An example process for determining the reward124based on the respective embeddings of the current and goal observations is described in more detail with reference toFIG. 2.

During training, the system100may select a goal observation108, and cause the agent102to interact with the environment104to attempt to achieve the goal state specified by the goal observation108for one or more time steps, referred to as an “episode”. The system100may determine the reward received by the agent to be zero (i.e., by default) at each time step until the final time step of the episode. At the final time step of the episode, the system100may determine the reward124received by the agent using the embedding network112, as described above.

The system100can determine the number of time steps in an episode in any of a variety of ways. For example, the system100can determine each episode to have a fixed number of time steps, e.g., 100 time steps. As another example, the system100can adaptively determine the number of time steps in each episode by setting a maximum number of time steps, but terminating the episode early if a similarity measure between the respective embeddings of the current and goal observations satisfies a predetermined threshold.

The system100may select the goal observation108at the start of each episode from a goal buffer114that stores previous observations of the environment. For example, the system100may randomly sample the goal observation108from the goal buffer114at the start of each episode.

The system100may continually update the goal buffer114by adding new observations (potentially overwriting previous observations) as the agent102interacts with the environment. An example process for updating the goal buffer114is described in more detail with reference toFIG. 3.

The training engine116is configured to jointly train the model parameters118of the action selection network110and the embedding network112based on the interactions of the agent with the environment. In particular, the training engine116can train the action selection network110to increase the return (i.e., cumulative measure of reward) received by the agent using reinforcement learning techniques. The system100can train the embedding neural network to increase a similarity measure between the embeddings of: (i) the observation of the environment at the final time step of an episode, and (ii) the goal observation of the environment corresponding to the episode.

In some implementations, the training engine116can train the system100based on rewards124generated using the embedding network112for a first number of training iterations, before replacing the rewards124generated using the embedding network112with external rewards. An external reward refers to a reward that is provided to the system by an external source.

In some implementations, the system can combine the rewards124generated using the embedding network112with external rewards, e.g., by generating an overall reward at each time step by adding the reward124and the external reward.

In some implementations, the action selection network110may share parameter value with an additional action selection network that is configured to generate additional action selection outputs that are directed to causing the agent to perform an additional task. The training engine116can train the action selection network110using the rewards124generated using the embedding network112, and can train the additional action selection network using, e.g., external rewards. The shared parameters of the two action selection networks would be trained based on both the internal rewards124and the external rewards.

FIG. 2is a flow diagram of an example process200for jointly training the action selection network and the embedding network. For convenience, the process200will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system, e.g., the action selection system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process200.

Steps202to210of the process200correspond to a given episode. For convenience, each of the steps202to210are described with reference to a “current” time step.

If the current time step is the first time step of an episode, the system selects a goal observation for the episode from a goal buffer (202). The goal buffer stores multiple previous observations of the environment. The system may select the goal observation, e.g., by randomly sampling an observation from the goal buffer. The goal buffer may, in some cases, store additional data associated with each observation in the goal buffer, e.g., how recently the observation was encountered, and the system can select the goal based at least in part on this additional data.

While this specification mainly describes the goal observation as being selected from a goal buffer, other implementations are possible. An example process for selecting a goal observation without using a goal buffer is described with reference to: C. Florensa, D. Held, X. Geng, P. Abbeel: “Automated goal generation for reinforcement learning agents”, arXiv:1705.06366v5 (2018).

The system receives an observation characterizing the current state of the environment (204). The current observation may be generated by or derived from sensors of the agent. For example, the current observation may include one or more of: a color image captured by a camera sensor of the agent, a hyperspectral image captured by a hyperspectral sensor of the agent, and geometric data captured by a laser sensor of the agent.

Optionally, the system can use the current observation to update the goal buffer (206). For example, if the goal buffer is not “full” (i.e., if the goal buffer currently stores fewer than a predetermined maximum number of observations), the system can store the current observation in the goal buffer. If the goal buffer is full, the system can store the current observation in the goal buffer with a predetermined probability (e.g., 25%). For example, the system can replace a randomly selected observation from the goal buffer with the current observation with the predetermined probability. Another example process for updating the goal buffer is described with reference toFIG. 3.

The system selects the action to be performed by the agent at the current time step (208). In particular, the system processes the current observation and the goal observation using the action selection network to generate an action selection output, and thereafter selects the action based on the action selection output. In one example, the action selection neural network is a Q neural network and the action selection output defines a respective Q value for each action in a predetermined set of actions. In this example, the system may select the action with a highest Q value.

In some implementations, the action selection network processes other inputs in addition to the current observation and the goal observation. For example, the action selection network may process an input that includes a periodic representation of the current time step, e.g.:

[sin⁢⁢2⁢π⁢⁢tT,cos⁢⁢2⁢π⁢⁢tT](1)
where t indexes the current time step and T is a predetermined number of time steps in the current episode. Processing the periodic representation of the current time step can enable the action selection system to select actions that cause the agent to achieve goals states which may be unmaintainable due to their instability in the environment dynamics.

The system determines the reward for the current time step (210). To determine the reward for the time step, the system may generate an embedding of the current observation and an embedding of the goal observation by processing the current observation and the goal observation using the embedding neural network. The system may then determine the reward using the respective embeddings of the current observation and the goal observation. A few examples of generating the reward follow.

In one example, the system may determine the reward based on a similarity measure between the embedding of the current observation and the embedding of the goal observation, e.g.:
r=max(0,(e(s),e(g)))  (2)
where r is the reward, e(s) is the embedding of the current observation, e(g) is the embedding of the goal observation, and(⋅,⋅) is a similarity measure, e.g., an inner product similarity measure.

In another example, to determine the reward, the system may obtain one or more “decoy” observations that each characterize a respective previous state of the environment, e.g., by randomly sampling them from the goal buffer. The system may then process each of the decoy observations using the embedding network to generate a respective embedding of each decoy observation, and determine the reward based on the embeddings of: (i) the current observation, (ii) the goal observation, and (iii) the decoy observations. In a particular example, the system may determine the reward as:

r=exp⁡(β·𝒮⁡(e⁡(s),e⁡(g)))exp⁡(β·𝒮⁡(e⁡(s),e⁡(g)))+∑k=1K⁢exp⁡(β·𝒮⁡(e⁡(s),e⁡(dk)))(3)
where r is the reward, e(s) is embedding of the current observation, e(g) is the embedding of the goal observation, k indexes the decoy observations, K is the total number of decoy observations, e(dk) is the embedding of the k-th decoy observation, β is an inverse temperature parameter (e.g., set to K+1), and S(⋅,⋅) is a similarity measure, e.g., an inner product similarity measure.

For computational efficiency, the system may generate the embedding of the goal observation at the first time step of the episode and reuse the embedding of the goal observation at each subsequent time step of the episode rather than re-generating it.

In some cases, the system may determine the reward using the embeddings of the goal and current observations for only the final time step of the episode. For each time step preceding the final time step, the system may generate a default (i.e., predetermined) reward, e.g., a reward of zero.

If the current time step is not the final time step of the episode, the system can return to step204and continue the current episode by repeating steps204-210. If the current time step is the final time step of the episode, the system can return to step202and start a new episode.

As the agent interacts with the environment, the system trains the action selection network and the embedding network by repeatedly updating their respective parameter values based on data characterizing the interaction of the agent with the environment (212).

The data characterizing the interaction of the agent with the environment during an episode can be referred to as a “trajectory” and may specify:
(s1:T,a1:T,r1:T,g)  (4)
where s1:Trepresents observations characterizing the states of the environment during the episode, a1:Trepresents the actions performed by the agent during the episode, r1:Trepresents the rewards received during the episode, and g represents the goal observation for the episode.

The system trains the action selection network using reinforcement learning training techniques, based on the rewards received by the agent during an episode, to increase a cumulative measure of rewards received by the agent as a result of interacting with the environment. The system can train the action selection network using any appropriate reinforcement learning training technique, e.g., an actor-critic technique or a Q learning technique, e.g., an n-step Q learning technique. In particular, the system can train the action selection network by iteratively adjusting the parameter values of the action selection network using gradients of a reinforcement learning objective function, e.g., by stochastic gradient descent techniques. The system can use the gradients to adjust the parameter values of the action selection network using any appropriate gradient descent optimization technique, e.g., an Adam or RMSprop gradient descent optimization technique.

The system can train the embedding network to increase a similarity measure between: (i) the embedding of the observation corresponding to the final time step of an episode, and (ii) the embedding of the goal observation for the episode. For example, the system can train the embedding network by iteratively adjusting the parameter values of the embedding network using gradients of a loss function, e.g., using stochastic gradient descent techniques, whereis given by:

In some cases, the system trains the action selection network and the embedding network using experience replay. More specifically, the system stores trajectories (e.g., as described with reference to equation (4)) corresponding to multiple episodes in a data store, and trains the action network and the embedding network on trajectories sampled from the data store. In these implementations, the system may train the action selection network using off-policy reinforcement learning techniques, to account for differences potential differences between: (i) the current values of the action selection network parameters, and (ii) the values of the action selection network parameters when the trajectories were generated.

The system may use a particular variant of experience replay, referred to as hindsight experience replay, during training of the action selection network and the embedding network. In hindsight experience replay, the system can adjust one or more trajectories from the data store by: (i) replacing the goal observation from the trajectory with an observation randomly sampled from the final H steps of the trajectory (where H is a positive integer value), and (ii) replacing the reward from the final time step of the trajectory, e.g. with the maximum possible reward. Using hindsight experience replay may enable the system to train the action selection network and embedding network more effectively, e.g., over fewer training iterations. For example, using hindsight experience replay enable the action selection network to be trained more effectively by artificially generating “successful” trajectories, i.e., where the agent has reached (or approximately reached) the goal state. As another example, using hindsight experience replay may regularize the training of the embedding network by encouraging the embedding network to generate temporally consistent observation embeddings, i.e., where observations that are nearby in time have similar embeddings.

Jointly training the action selection network and the embedding network can enable the system to learn to generate rewards that implicitly measure similarity in the space of “controllable” aspects of the environment instead of in the space of raw observations of the environment. An aspect of the environment is said to be controllable if it is controlled by the actions performed by the agent. For example, for a robotic agent, the position of an actuator of the agent may be a controllable aspect of the environment, while the ambient lighting conditions and the movement of “distractor” objects in the vicinity of the agent may be uncontrollable aspects of the environment.

Using learned rewards that measure similarity in controllable aspects of the environment can enable the system to effectively learn to approximately achieve goal states that are not perfectly reachable, e.g., due to aspects of the environment outside of the control of the agent. For example, the system can train a robotic agent to maneuver an actuator to a goal position independently of variations in the ambient lighting conditions and the movement of distractor objects in the vicinity of the agent. In contrast, using rewards that directly measure similarity between states of the environment in the space of raw observations (e.g., by a pixel-level comparison of images of the environment) may cause the system to master the environment more slowly (or not at all).

FIG. 3is a flow diagram of an example process300for determining whether to update the goal buffer with a current observation at a given time step. Prior to the first step of the process300, the system may sample a Bernoulli random variable, and thereafter determine that the current observation is eligible to update the goal buffer only if the random variable assumes a specified value (e.g., the value 1). For convenience, the process300will be described as being performed by a system of one or more computers located in one or more locations. For example, an action selection system, e.g., the action selection system100ofFIG. 1, appropriately programmed in accordance with this specification, can perform the process300.

The system selects a candidate observation from the goal buffer (302). For example, the system may select the candidate observation by randomly sampling from the goal buffer.

The system determines an “aggregate” similarity measure between: (i) the candidate observation, and (ii) the remaining observations (i.e., other than the candidate observation) in the goal buffer (304). For example, the system may determine the aggregate similarity measure as:

1N-1⁢∑k=1N-1⁢𝒮⁡(c,dk)(6)
where N is the total number of observations in the goal buffer, c is the candidate observation, k indexes the remaining observations in the goal buffer, dkis the k-th remaining observation in the goal buffer, and S(⋅,⋅) is a similarity measure, e.g., an L2similarity measure.

The system determines an aggregate similarity measure between: (i) the current observation, and (ii) the remaining observations (i.e., other than the candidate observation) in the goal buffer (306). For example, the system may determine the aggregate similarity measure as:

1N-1⁢∑k=1N-1⁢𝒮⁡(s,dk)(7)
where s is the current observation, and the remaining variables are defined in the same manner as for equation (6).

The system determines whether to update the goal buffer with the current observation based on the aggregate similarity measures determined at steps304and306(308). For example, if the aggregate similarity measures indicate that the candidate observation is more similar to the remaining observations than the current observation, the system may replace the candidate observation with the current observation. On the other hand, if the aggregate similarity measures indicate that the current observation is more similar to the remaining observations than the candidate observation, the system may replace the candidate observation with the current observation with a predetermined probability that is strictly less than one.

Determining whether to update the goal buffer in this manner can increase the diversity of the observations in the goal buffer, which can enhance the training of the action selection network and the embedding network.